Oct 30 16:22:12.165733 ip-10-0-132-180 systemd[1]: Starting Kubernetes Kubelet... Oct 30 16:22:12.535084 ip-10-0-132-180 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 16:22:12.535084 ip-10-0-132-180 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 16:22:12.535084 ip-10-0-132-180 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 16:22:12.535084 ip-10-0-132-180 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 16:22:12.542281 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542187 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:22:12.542281 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542273 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:22:12.542281 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542276 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:22:12.542281 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542279 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:22:12.542281 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542282 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:22:12.542281 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542285 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:22:12.542281 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542288 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:22:12.542281 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542290 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:22:12.542281 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542293 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:22:12.542577 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542296 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:22:12.542577 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542298 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:22:12.542577 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542301 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:22:12.542577 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542304 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:22:12.542577 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542306 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:22:12.542577 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542309 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:22:12.542577 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542311 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:22:12.542577 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542314 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:22:12.542577 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542316 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:22:12.542577 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542318 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:22:12.542577 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542335 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:22:12.542577 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542338 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:22:12.542577 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542341 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:22:12.542577 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542344 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:22:12.542577 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542347 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:22:12.542577 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542351 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:22:12.542577 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542354 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:22:12.542577 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542357 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:22:12.542577 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542360 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:22:12.543039 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542362 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:22:12.543039 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542365 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:22:12.543039 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542367 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:22:12.543039 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542370 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:22:12.543039 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542372 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:22:12.543039 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542375 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:22:12.543039 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542378 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:22:12.543039 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542380 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:22:12.543039 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542383 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:22:12.543039 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542386 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:22:12.543039 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542388 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:22:12.543039 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542391 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:22:12.543039 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542393 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:22:12.543039 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542396 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:22:12.543039 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542399 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:22:12.543039 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542402 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:22:12.543039 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542404 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:22:12.543039 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542407 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:22:12.543039 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542409 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:22:12.543039 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542413 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:22:12.543546 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542415 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:22:12.543546 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542418 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:22:12.543546 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542421 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:22:12.543546 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542423 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:22:12.543546 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542427 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:22:12.543546 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542431 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:22:12.543546 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542434 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:22:12.543546 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542436 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:22:12.543546 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542439 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:22:12.543546 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542442 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:22:12.543546 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542444 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:22:12.543546 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542447 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:22:12.543546 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542449 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:22:12.543546 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542452 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:22:12.543546 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542454 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:22:12.543546 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542457 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:22:12.543546 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542459 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:22:12.543546 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542462 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:22:12.543546 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542466 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:22:12.544007 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542470 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:22:12.544007 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542473 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:22:12.544007 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542476 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:22:12.544007 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542478 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:22:12.544007 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542481 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:22:12.544007 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542484 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:22:12.544007 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542486 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:22:12.544007 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542489 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:22:12.544007 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542492 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:22:12.544007 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542494 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:22:12.544007 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542497 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:22:12.544007 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542499 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:22:12.544007 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542502 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:22:12.544007 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542504 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:22:12.544007 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542507 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:22:12.544007 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542509 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:22:12.544007 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542512 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:22:12.544007 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542515 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:22:12.544007 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542517 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:22:12.544007 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542520 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:22:12.544492 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542522 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:22:12.544492 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542524 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:22:12.544492 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542527 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:22:12.544492 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542529 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:22:12.544492 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542532 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:22:12.544492 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542534 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:22:12.544492 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542536 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:22:12.544492 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542539 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:22:12.544492 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542541 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:22:12.544492 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542544 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:22:12.544492 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542546 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:22:12.544492 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542549 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:22:12.544492 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542551 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:22:12.544492 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542554 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:22:12.544492 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542557 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:22:12.544492 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542559 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:22:12.544492 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542562 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:22:12.544492 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542564 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:22:12.544492 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.542567 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:22:12.544972 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543343 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:22:12.544972 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543350 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:22:12.544972 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543353 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:22:12.544972 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543357 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:22:12.544972 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543359 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:22:12.544972 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543362 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:22:12.544972 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543365 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:22:12.544972 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543368 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:22:12.544972 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543370 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:22:12.544972 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543373 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:22:12.544972 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543376 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:22:12.544972 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543379 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:22:12.544972 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543382 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:22:12.544972 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543384 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:22:12.544972 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543386 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:22:12.544972 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543389 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:22:12.544972 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543391 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:22:12.544972 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543394 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:22:12.544972 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543398 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:22:12.545444 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543402 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:22:12.545444 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543404 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:22:12.545444 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543407 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:22:12.545444 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543410 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:22:12.545444 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543413 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:22:12.545444 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543415 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:22:12.545444 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543418 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:22:12.545444 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543420 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:22:12.545444 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543423 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:22:12.545444 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543426 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:22:12.545444 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543428 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:22:12.545444 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543431 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:22:12.545444 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543435 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:22:12.545444 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543438 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:22:12.545444 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543440 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:22:12.545444 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543443 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:22:12.545444 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543445 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:22:12.545444 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543448 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:22:12.545444 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543450 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:22:12.545444 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543453 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:22:12.545993 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543455 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:22:12.545993 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543457 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:22:12.545993 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543460 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:22:12.545993 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543462 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:22:12.545993 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543465 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:22:12.545993 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543468 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:22:12.545993 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543470 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:22:12.545993 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543473 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:22:12.545993 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543475 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:22:12.545993 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543477 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:22:12.545993 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543480 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:22:12.545993 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543483 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:22:12.545993 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543485 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:22:12.545993 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543488 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:22:12.545993 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543490 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:22:12.545993 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543493 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:22:12.545993 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543495 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:22:12.545993 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543497 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:22:12.545993 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543500 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:22:12.545993 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543502 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:22:12.546480 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543505 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:22:12.546480 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543507 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:22:12.546480 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543510 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:22:12.546480 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543512 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:22:12.546480 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543516 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:22:12.546480 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543518 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:22:12.546480 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543521 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:22:12.546480 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543523 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:22:12.546480 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543526 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:22:12.546480 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543529 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:22:12.546480 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543531 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:22:12.546480 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543534 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:22:12.546480 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543536 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:22:12.546480 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543538 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:22:12.546480 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543541 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:22:12.546480 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543543 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:22:12.546480 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543546 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:22:12.546480 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543549 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:22:12.546480 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543552 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:22:12.546480 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543555 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:22:12.546961 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543558 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:22:12.546961 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543560 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:22:12.546961 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543564 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:22:12.546961 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543568 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:22:12.546961 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543571 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:22:12.546961 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543574 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:22:12.546961 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543577 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:22:12.546961 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543580 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:22:12.546961 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543583 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:22:12.546961 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543586 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:22:12.546961 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543589 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:22:12.546961 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543591 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:22:12.546961 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543594 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:22:12.546961 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543596 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:22:12.546961 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543599 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:22:12.546961 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543602 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:22:12.546961 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543604 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:22:12.546961 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543607 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:22:12.546961 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543609 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:22:12.546961 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543611 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:22:12.547443 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543614 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:22:12.547443 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543617 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:22:12.547443 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543620 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:22:12.547443 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543622 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:22:12.547443 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543624 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:22:12.547443 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543627 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:22:12.547443 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.543629 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:22:12.547443 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544161 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:22:12.547443 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544168 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:22:12.547443 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544173 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:22:12.547443 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544176 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:22:12.547443 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544179 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:22:12.547443 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544182 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:22:12.547443 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544184 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:22:12.547443 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544187 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:22:12.547443 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544189 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:22:12.547443 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544192 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:22:12.547443 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544194 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:22:12.547443 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544197 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:22:12.547443 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544199 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:22:12.547934 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544202 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:22:12.547934 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544205 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:22:12.547934 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544209 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:22:12.547934 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544213 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:22:12.547934 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544216 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:22:12.547934 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544219 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:22:12.547934 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544221 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:22:12.547934 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544224 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:22:12.547934 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544227 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:22:12.547934 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544230 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:22:12.547934 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544232 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:22:12.547934 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544235 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:22:12.547934 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544237 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:22:12.547934 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544239 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:22:12.547934 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544242 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:22:12.547934 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544244 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:22:12.547934 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544248 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:22:12.547934 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544250 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:22:12.547934 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544253 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:22:12.548400 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544255 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:22:12.548400 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544258 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:22:12.548400 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544260 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:22:12.548400 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544263 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:22:12.548400 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544266 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:22:12.548400 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544268 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:22:12.548400 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544271 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:22:12.548400 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544273 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:22:12.548400 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544276 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:22:12.548400 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544278 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:22:12.548400 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544280 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:22:12.548400 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544283 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:22:12.548400 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544285 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:22:12.548400 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544288 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:22:12.548400 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544291 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:22:12.548400 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544293 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:22:12.548400 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544296 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:22:12.548400 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544298 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:22:12.548400 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544300 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:22:12.548908 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544303 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:22:12.548908 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544305 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:22:12.548908 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544308 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:22:12.548908 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544310 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:22:12.548908 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544312 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:22:12.548908 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544315 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:22:12.548908 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544317 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:22:12.548908 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544320 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:22:12.548908 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544322 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:22:12.548908 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544324 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:22:12.548908 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544326 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:22:12.548908 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544329 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:22:12.548908 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544331 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:22:12.548908 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544333 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:22:12.548908 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544336 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:22:12.548908 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544338 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:22:12.548908 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544341 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:22:12.548908 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544344 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:22:12.548908 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544346 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:22:12.549385 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544349 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:22:12.549385 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544351 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:22:12.549385 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544353 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:22:12.549385 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544356 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:22:12.549385 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544358 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:22:12.549385 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544360 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:22:12.549385 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544365 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:22:12.549385 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544368 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:22:12.549385 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544370 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:22:12.549385 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544373 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:22:12.549385 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544375 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:22:12.549385 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544378 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:22:12.549385 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544380 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:22:12.549385 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544382 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:22:12.549385 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544385 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:22:12.549385 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544388 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:22:12.549385 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544390 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:22:12.549385 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544393 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:22:12.549385 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544395 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:22:12.549385 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544398 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:22:12.549865 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544400 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:22:12.549865 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544402 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:22:12.549865 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544405 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:22:12.549865 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544407 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:22:12.549865 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544409 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:22:12.549865 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544412 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:22:12.549865 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544414 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:22:12.549865 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544416 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:22:12.549865 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544419 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:22:12.549865 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544422 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:22:12.549865 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544424 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:22:12.549865 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544427 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:22:12.549865 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544429 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:22:12.549865 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544432 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:22:12.549865 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544434 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:22:12.549865 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.544439 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:22:12.549865 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545243 2575 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 16:22:12.549865 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545258 2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 16:22:12.549865 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545266 2575 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 16:22:12.549865 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545271 2575 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 16:22:12.549865 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545275 2575 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 16:22:12.550478 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545288 2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 16:22:12.550478 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545293 2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 16:22:12.550478 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545297 2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 16:22:12.550478 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545300 2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 16:22:12.550478 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545303 2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 16:22:12.550478 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545306 2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 16:22:12.550478 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545309 2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 16:22:12.550478 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545312 2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 16:22:12.550478 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545315 2575 flags.go:64] FLAG: --cgroup-root="" Oct 30 16:22:12.550478 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545318 2575 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 16:22:12.550478 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545321 2575 flags.go:64] FLAG: --client-ca-file="" Oct 30 16:22:12.550478 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545324 2575 flags.go:64] FLAG: --cloud-provider="external" Oct 30 16:22:12.550478 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545327 2575 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 16:22:12.550478 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545332 2575 flags.go:64] FLAG: --cluster-domain="" Oct 30 16:22:12.550478 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545335 2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 16:22:12.550478 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545338 2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 16:22:12.550478 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545341 2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 16:22:12.550478 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545344 2575 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 16:22:12.550478 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545348 2575 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 16:22:12.550478 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545351 2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 16:22:12.550478 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545354 2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 16:22:12.550478 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545357 2575 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 16:22:12.550478 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545360 2575 flags.go:64] FLAG: --contention-profiling="false" Oct 30 16:22:12.550478 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545369 2575 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 16:22:12.551040 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545372 2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 16:22:12.551040 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545376 2575 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 16:22:12.551040 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545378 2575 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 16:22:12.551040 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545383 2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 16:22:12.551040 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545386 2575 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 16:22:12.551040 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545390 2575 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 16:22:12.551040 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545393 2575 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 16:22:12.551040 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545396 2575 flags.go:64] FLAG: --enable-server="true" Oct 30 16:22:12.551040 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545399 2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 16:22:12.551040 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545406 2575 flags.go:64] FLAG: --event-burst="100" Oct 30 16:22:12.551040 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545409 2575 flags.go:64] FLAG: --event-qps="50" Oct 30 16:22:12.551040 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545412 2575 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 16:22:12.551040 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545415 2575 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 16:22:12.551040 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545418 2575 flags.go:64] FLAG: --eviction-hard="" Oct 30 16:22:12.551040 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545422 2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 16:22:12.551040 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545425 2575 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 16:22:12.551040 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545428 2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 16:22:12.551040 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545431 2575 flags.go:64] FLAG: --eviction-soft="" Oct 30 16:22:12.551040 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545433 2575 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 16:22:12.551040 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545436 2575 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 16:22:12.551040 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545439 2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 16:22:12.551040 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545442 2575 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 16:22:12.551040 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545445 2575 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 16:22:12.551040 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545448 2575 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 16:22:12.551040 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545451 2575 flags.go:64] FLAG: --feature-gates="" Oct 30 16:22:12.551677 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545455 2575 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 16:22:12.551677 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545458 2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 16:22:12.551677 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545461 2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 16:22:12.551677 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545464 2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 16:22:12.551677 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545467 2575 flags.go:64] FLAG: --healthz-port="10248" Oct 30 16:22:12.551677 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545470 2575 flags.go:64] FLAG: --help="false" Oct 30 16:22:12.551677 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545473 2575 flags.go:64] FLAG: --hostname-override="ip-10-0-132-180.ec2.internal" Oct 30 16:22:12.551677 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545477 2575 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 16:22:12.551677 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545480 2575 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 16:22:12.551677 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545483 2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 16:22:12.551677 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545486 2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 16:22:12.551677 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545489 2575 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 16:22:12.551677 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545495 2575 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 16:22:12.551677 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545498 2575 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 16:22:12.551677 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545500 2575 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 16:22:12.551677 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545503 2575 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 16:22:12.551677 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545506 2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 16:22:12.551677 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545510 2575 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 16:22:12.551677 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545512 2575 flags.go:64] FLAG: --kube-reserved="" Oct 30 16:22:12.551677 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545515 2575 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 16:22:12.551677 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545518 2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 16:22:12.551677 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545521 2575 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 16:22:12.551677 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545524 2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 16:22:12.551677 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545526 2575 flags.go:64] FLAG: --lock-file="" Oct 30 16:22:12.552284 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545529 2575 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 16:22:12.552284 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545532 2575 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 16:22:12.552284 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545535 2575 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 16:22:12.552284 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545541 2575 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 16:22:12.552284 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545544 2575 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 16:22:12.552284 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545547 2575 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 16:22:12.552284 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545549 2575 flags.go:64] FLAG: --logging-format="text" Oct 30 16:22:12.552284 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545552 2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 16:22:12.552284 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545556 2575 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 16:22:12.552284 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545558 2575 flags.go:64] FLAG: --manifest-url="" Oct 30 16:22:12.552284 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545561 2575 flags.go:64] FLAG: --manifest-url-header="" Oct 30 16:22:12.552284 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545566 2575 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 16:22:12.552284 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545569 2575 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 16:22:12.552284 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545577 2575 flags.go:64] FLAG: --max-pods="110" Oct 30 16:22:12.552284 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545580 2575 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 16:22:12.552284 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545583 2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 16:22:12.552284 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545586 2575 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 16:22:12.552284 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545589 2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 16:22:12.552284 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545591 2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 16:22:12.552284 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545594 2575 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 16:22:12.552284 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545600 2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 16:22:12.552284 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545607 2575 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 16:22:12.552284 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545610 2575 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 16:22:12.552284 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545613 2575 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 16:22:12.552284 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545616 2575 flags.go:64] FLAG: --pod-cidr="" Oct 30 16:22:12.552900 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545619 2575 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 16:22:12.552900 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545622 2575 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 16:22:12.552900 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545625 2575 flags.go:64] FLAG: --pods-per-core="0" Oct 30 16:22:12.552900 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545628 2575 flags.go:64] FLAG: --port="10250" Oct 30 16:22:12.552900 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545630 2575 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 16:22:12.552900 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545633 2575 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c2f018e142dba3de" Oct 30 16:22:12.552900 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545636 2575 flags.go:64] FLAG: --qos-reserved="" Oct 30 16:22:12.552900 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545639 2575 flags.go:64] FLAG: --read-only-port="10255" Oct 30 16:22:12.552900 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545642 2575 flags.go:64] FLAG: --register-node="true" Oct 30 16:22:12.552900 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545645 2575 flags.go:64] FLAG: --register-with-taints="" Oct 30 16:22:12.552900 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545649 2575 flags.go:64] FLAG: --registry-burst="10" Oct 30 16:22:12.552900 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545651 2575 flags.go:64] FLAG: --registry-qps="5" Oct 30 16:22:12.552900 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545654 2575 flags.go:64] FLAG: --reserved-cpus="" Oct 30 16:22:12.552900 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545657 2575 flags.go:64] FLAG: --reserved-memory="" Oct 30 16:22:12.552900 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545661 2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 16:22:12.552900 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545664 2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 16:22:12.552900 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545666 2575 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 16:22:12.552900 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545669 2575 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 16:22:12.552900 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545672 2575 flags.go:64] FLAG: --runonce="false" Oct 30 16:22:12.552900 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545675 2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 16:22:12.552900 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545678 2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 16:22:12.552900 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545681 2575 flags.go:64] FLAG: --seccomp-default="false" Oct 30 16:22:12.552900 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545684 2575 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 16:22:12.552900 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545687 2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 16:22:12.552900 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545690 2575 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 16:22:12.552900 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545694 2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 16:22:12.553528 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545697 2575 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 16:22:12.553528 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545702 2575 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 16:22:12.553528 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545705 2575 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 16:22:12.553528 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545708 2575 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 16:22:12.553528 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545711 2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 16:22:12.553528 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545714 2575 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 16:22:12.553528 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545717 2575 flags.go:64] FLAG: --system-cgroups="" Oct 30 16:22:12.553528 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545719 2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 16:22:12.553528 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545725 2575 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 16:22:12.553528 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545728 2575 flags.go:64] FLAG: --tls-cert-file="" Oct 30 16:22:12.553528 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545731 2575 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 16:22:12.553528 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545735 2575 flags.go:64] FLAG: --tls-min-version="" Oct 30 16:22:12.553528 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545738 2575 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 16:22:12.553528 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545741 2575 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 16:22:12.553528 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545744 2575 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 16:22:12.553528 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545746 2575 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 16:22:12.553528 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545749 2575 flags.go:64] FLAG: --v="2" Oct 30 16:22:12.553528 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545754 2575 flags.go:64] FLAG: --version="false" Oct 30 16:22:12.553528 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545758 2575 flags.go:64] FLAG: --vmodule="" Oct 30 16:22:12.553528 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545762 2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 16:22:12.553528 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.545765 2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 16:22:12.553528 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.545913 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:22:12.553528 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.545918 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:22:12.553528 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.545921 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:22:12.554107 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.545924 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:22:12.554107 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.545926 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:22:12.554107 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.545929 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:22:12.554107 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.545932 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:22:12.554107 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.545934 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:22:12.554107 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.545937 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:22:12.554107 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.545939 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:22:12.554107 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.545942 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:22:12.554107 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.545945 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:22:12.554107 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.545951 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:22:12.554107 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.545954 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:22:12.554107 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.545956 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:22:12.554107 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.545959 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:22:12.554107 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.545961 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:22:12.554107 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.545964 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:22:12.554107 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.545967 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:22:12.554107 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.545970 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:22:12.554107 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.545972 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:22:12.554107 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.545975 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:22:12.554559 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.545977 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:22:12.554559 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.545980 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:22:12.554559 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.545982 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:22:12.554559 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.545984 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:22:12.554559 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.545987 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:22:12.554559 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.545989 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:22:12.554559 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.545992 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:22:12.554559 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.545994 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:22:12.554559 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.545997 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:22:12.554559 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.545999 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:22:12.554559 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546002 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:22:12.554559 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546004 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:22:12.554559 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546007 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:22:12.554559 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546010 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:22:12.554559 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546012 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:22:12.554559 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546014 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:22:12.554559 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546017 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:22:12.554559 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546021 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:22:12.554559 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546025 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:22:12.554559 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546028 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:22:12.555157 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546031 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:22:12.555157 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546034 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:22:12.555157 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546039 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:22:12.555157 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546041 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:22:12.555157 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546044 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:22:12.555157 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546046 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:22:12.555157 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546049 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:22:12.555157 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546052 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:22:12.555157 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546054 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:22:12.555157 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546061 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:22:12.555157 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546064 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:22:12.555157 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546067 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:22:12.555157 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546069 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:22:12.555157 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546071 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:22:12.555157 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546074 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:22:12.555157 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546076 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:22:12.555157 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546079 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:22:12.555157 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546081 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:22:12.555157 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546084 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:22:12.555600 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546086 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:22:12.555600 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546088 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:22:12.555600 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546091 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:22:12.555600 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546094 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:22:12.555600 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546096 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:22:12.555600 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546098 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:22:12.555600 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546101 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:22:12.555600 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546103 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:22:12.555600 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546106 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:22:12.555600 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546108 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:22:12.555600 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546110 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:22:12.555600 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546113 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:22:12.555600 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546115 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:22:12.555600 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546118 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:22:12.555600 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546120 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:22:12.555600 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546126 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:22:12.555600 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546128 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:22:12.555600 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546131 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:22:12.555600 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546134 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:22:12.556050 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546136 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:22:12.556050 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546139 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:22:12.556050 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546141 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:22:12.556050 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546144 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:22:12.556050 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546147 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:22:12.556050 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546149 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:22:12.556050 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546151 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:22:12.556050 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546154 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:22:12.556050 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546156 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:22:12.556050 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546159 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:22:12.556050 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546161 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:22:12.556050 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546164 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:22:12.556050 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546166 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:22:12.556050 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546169 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:22:12.556050 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546173 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:22:12.556050 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546176 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:22:12.556050 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546179 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:22:12.556050 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546181 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:22:12.556050 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546184 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:22:12.556050 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546187 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:22:12.556516 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546189 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:22:12.556516 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546192 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:22:12.556516 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546194 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:22:12.556516 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546197 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:22:12.556516 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546199 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:22:12.556516 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546202 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:22:12.556516 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.546229 2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 16:22:12.556516 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546276 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:22:12.556516 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546282 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:22:12.556516 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546287 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:22:12.556516 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546290 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:22:12.556516 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546292 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:22:12.556516 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546295 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:22:12.556516 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546298 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:22:12.556516 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546300 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:22:12.556865 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546303 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:22:12.556865 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546306 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:22:12.556865 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546309 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:22:12.556865 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546311 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:22:12.556865 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546313 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:22:12.556865 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546316 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:22:12.556865 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546319 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:22:12.556865 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546321 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:22:12.556865 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546323 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:22:12.556865 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546326 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:22:12.556865 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546328 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:22:12.556865 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546331 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:22:12.556865 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546333 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:22:12.556865 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546336 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:22:12.556865 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546338 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:22:12.556865 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546341 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:22:12.556865 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546343 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:22:12.556865 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546346 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:22:12.556865 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546348 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:22:12.556865 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546350 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:22:12.557339 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546352 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:22:12.557339 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546355 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:22:12.557339 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546358 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:22:12.557339 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546362 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:22:12.557339 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546365 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:22:12.557339 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546369 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:22:12.557339 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546374 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:22:12.557339 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546377 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:22:12.557339 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546380 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:22:12.557339 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546383 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:22:12.557339 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546385 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:22:12.557339 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546388 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:22:12.557339 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546390 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:22:12.557339 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546393 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:22:12.557339 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546395 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:22:12.557339 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546397 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:22:12.557339 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546400 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:22:12.557339 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546403 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:22:12.557339 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546405 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:22:12.557779 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546407 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:22:12.557779 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546410 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:22:12.557779 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546412 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:22:12.557779 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546414 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:22:12.557779 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546417 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:22:12.557779 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546419 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:22:12.557779 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546422 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:22:12.557779 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546424 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:22:12.557779 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546426 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:22:12.557779 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546430 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:22:12.557779 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546432 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:22:12.557779 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546435 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:22:12.557779 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546437 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:22:12.557779 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546439 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:22:12.557779 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546442 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:22:12.557779 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546444 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:22:12.557779 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546447 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:22:12.557779 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546449 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:22:12.557779 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546454 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:22:12.558240 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546458 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:22:12.558240 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546460 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:22:12.558240 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546463 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:22:12.558240 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546466 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:22:12.558240 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546469 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:22:12.558240 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546471 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:22:12.558240 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546474 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:22:12.558240 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546476 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:22:12.558240 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546479 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:22:12.558240 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546481 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:22:12.558240 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546484 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:22:12.558240 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546486 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:22:12.558240 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546489 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:22:12.558240 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546491 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:22:12.558240 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546494 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:22:12.558240 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546497 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:22:12.558240 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546499 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:22:12.558240 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546501 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:22:12.558240 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546504 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:22:12.558240 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546506 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:22:12.558706 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546510 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:22:12.558706 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546514 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:22:12.558706 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546517 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:22:12.558706 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546520 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:22:12.558706 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546522 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:22:12.558706 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546524 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:22:12.558706 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546527 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:22:12.558706 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546529 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:22:12.558706 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546532 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:22:12.558706 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546534 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:22:12.558706 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546537 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:22:12.558706 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546540 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:22:12.558706 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546543 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:22:12.558706 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546546 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:22:12.558706 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546548 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:22:12.558706 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546551 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:22:12.558706 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546554 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:22:12.558706 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546556 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:22:12.558706 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546559 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:22:12.559243 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.546563 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:22:12.559243 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.554001 2575 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 16:22:12.559243 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.554017 2575 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 16:22:12.559243 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.554766 2575 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 16:22:12.559243 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.554773 2575 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 16:22:12.559243 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.554853 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:22:12.559243 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.554859 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:22:12.559243 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.554862 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:22:12.559243 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.554865 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:22:12.559243 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.554868 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:22:12.559243 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.554871 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:22:12.559243 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.554874 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:22:12.559243 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.554876 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:22:12.559243 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.554879 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:22:12.559243 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.554881 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:22:12.559243 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.554884 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:22:12.559243 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.554886 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:22:12.559243 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.554890 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:22:12.559243 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.554893 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:22:12.559688 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.554896 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:22:12.559688 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.554900 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:22:12.559688 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.554904 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:22:12.559688 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.554907 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:22:12.559688 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.554909 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:22:12.559688 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.554912 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:22:12.559688 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.554914 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:22:12.559688 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.554917 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:22:12.559688 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.554919 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:22:12.559688 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.554922 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:22:12.559688 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.554924 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:22:12.559688 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.554927 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:22:12.559688 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.554929 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:22:12.559688 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.554932 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:22:12.559688 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.554935 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:22:12.559688 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.554944 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:22:12.559688 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.554947 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:22:12.559688 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.554950 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:22:12.559688 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.554952 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:22:12.560182 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.554955 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:22:12.560182 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.554957 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:22:12.560182 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.554960 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:22:12.560182 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.554962 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:22:12.560182 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.554965 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:22:12.560182 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.554967 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:22:12.560182 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.554969 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:22:12.560182 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.554972 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:22:12.560182 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.554974 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:22:12.560182 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.554977 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:22:12.560182 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.554979 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:22:12.560182 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.554983 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:22:12.560182 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.554987 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:22:12.560182 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.554990 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:22:12.560182 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.554994 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:22:12.560182 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.554996 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:22:12.560182 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.554999 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:22:12.560182 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555001 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:22:12.560182 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555004 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:22:12.560621 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555006 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:22:12.560621 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555009 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:22:12.560621 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555011 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:22:12.560621 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555014 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:22:12.560621 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555016 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:22:12.560621 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555019 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:22:12.560621 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555021 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:22:12.560621 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555024 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:22:12.560621 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555026 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:22:12.560621 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555029 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:22:12.560621 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555037 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:22:12.560621 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555040 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:22:12.560621 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555042 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:22:12.560621 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555045 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:22:12.560621 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555047 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:22:12.560621 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555050 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:22:12.560621 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555052 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:22:12.560621 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555055 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:22:12.560621 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555057 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:22:12.560621 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555059 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:22:12.561115 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555062 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:22:12.561115 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555064 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:22:12.561115 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555066 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:22:12.561115 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555069 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:22:12.561115 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555071 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:22:12.561115 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555074 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:22:12.561115 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555076 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:22:12.561115 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555078 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:22:12.561115 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555081 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:22:12.561115 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555083 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:22:12.561115 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555085 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:22:12.561115 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555088 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:22:12.561115 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555090 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:22:12.561115 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555093 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:22:12.561115 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555095 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:22:12.561115 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555098 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:22:12.561115 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555101 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:22:12.561115 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555103 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:22:12.561115 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555106 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:22:12.561570 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555108 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:22:12.561570 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555111 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:22:12.561570 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555114 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:22:12.561570 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555116 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:22:12.561570 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555124 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:22:12.561570 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555127 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:22:12.561570 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555129 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:22:12.561570 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555132 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:22:12.561570 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555134 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:22:12.561570 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555136 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:22:12.561570 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555139 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:22:12.561570 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555141 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:22:12.561570 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555144 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:22:12.561570 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555146 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:22:12.561570 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555149 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:22:12.561570 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.555179 2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 16:22:12.561965 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555360 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:22:12.561965 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555364 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:22:12.561965 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555367 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:22:12.561965 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555369 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:22:12.561965 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555372 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:22:12.561965 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555374 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:22:12.561965 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555377 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:22:12.561965 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555379 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:22:12.561965 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555382 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:22:12.561965 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555384 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:22:12.561965 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555386 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:22:12.561965 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555389 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:22:12.561965 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555391 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:22:12.561965 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555393 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:22:12.561965 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555396 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:22:12.561965 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555398 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:22:12.561965 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555401 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:22:12.561965 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555404 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:22:12.561965 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555406 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:22:12.561965 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555409 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:22:12.562447 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555411 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:22:12.562447 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555419 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:22:12.562447 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555421 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:22:12.562447 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555424 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:22:12.562447 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555426 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:22:12.562447 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555429 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:22:12.562447 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555431 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:22:12.562447 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555434 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:22:12.562447 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555436 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:22:12.562447 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555438 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:22:12.562447 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555441 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:22:12.562447 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555443 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:22:12.562447 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555445 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:22:12.562447 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555449 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:22:12.562447 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555453 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:22:12.562447 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555455 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:22:12.562447 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555458 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:22:12.562447 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555461 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:22:12.562447 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555463 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:22:12.563215 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555465 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:22:12.563215 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555468 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:22:12.563215 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555470 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:22:12.563215 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555473 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:22:12.563215 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555475 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:22:12.563215 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555478 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:22:12.563215 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555480 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:22:12.563215 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555483 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:22:12.563215 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555485 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:22:12.563215 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555488 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:22:12.563215 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555490 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:22:12.563215 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555493 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:22:12.563215 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555495 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:22:12.563215 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555498 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:22:12.563215 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555500 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:22:12.563215 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555508 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:22:12.563215 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555510 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:22:12.563215 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555513 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:22:12.563215 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555515 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:22:12.563215 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555518 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:22:12.564160 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555520 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:22:12.564160 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555522 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:22:12.564160 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555525 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:22:12.564160 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555527 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:22:12.564160 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555530 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:22:12.564160 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555532 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:22:12.564160 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555534 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:22:12.564160 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555537 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:22:12.564160 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555539 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:22:12.564160 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555542 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:22:12.564160 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555544 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:22:12.564160 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555546 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:22:12.564160 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555549 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:22:12.564160 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555551 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:22:12.564160 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555553 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:22:12.564160 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555556 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:22:12.564160 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555564 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:22:12.564160 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555567 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:22:12.564160 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555569 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:22:12.564832 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555572 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:22:12.564832 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555574 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:22:12.564832 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555576 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:22:12.564832 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555579 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:22:12.564832 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555581 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:22:12.564832 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555583 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:22:12.564832 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555586 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:22:12.564832 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555588 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:22:12.564832 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555591 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:22:12.564832 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555598 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:22:12.564832 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555601 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:22:12.564832 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555603 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:22:12.564832 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555606 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:22:12.564832 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555608 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:22:12.564832 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555611 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:22:12.564832 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555613 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:22:12.564832 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555616 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:22:12.564832 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555619 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:22:12.564832 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555621 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:22:12.564832 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555623 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:22:12.565344 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555626 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:22:12.565344 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555628 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:22:12.565344 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555630 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:22:12.565344 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555633 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:22:12.565344 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555635 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:22:12.565344 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555637 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:22:12.565344 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555640 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:22:12.565344 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555644 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:22:12.565344 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.555667 2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 16:22:12.565344 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555704 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:22:12.565344 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555709 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:22:12.565344 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555712 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:22:12.565344 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555715 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:22:12.565344 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555717 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:22:12.565344 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555720 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:22:12.565900 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555722 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:22:12.565900 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555724 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:22:12.565900 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555727 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:22:12.565900 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555729 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:22:12.565900 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555731 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:22:12.565900 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555734 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:22:12.565900 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555736 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:22:12.565900 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555745 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:22:12.565900 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555748 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:22:12.565900 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555750 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:22:12.565900 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555753 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:22:12.565900 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555755 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:22:12.565900 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555758 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:22:12.565900 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555760 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:22:12.565900 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555763 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:22:12.565900 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555765 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:22:12.565900 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555767 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:22:12.565900 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555770 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:22:12.565900 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555772 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:22:12.565900 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555775 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:22:12.566504 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555778 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:22:12.566504 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555796 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:22:12.566504 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555800 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:22:12.566504 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555802 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:22:12.566504 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555805 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:22:12.566504 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555807 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:22:12.566504 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555812 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:22:12.566504 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555816 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:22:12.566504 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555819 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:22:12.566504 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555822 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:22:12.566504 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555825 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:22:12.566504 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555827 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:22:12.566504 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555830 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:22:12.566504 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555832 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:22:12.566504 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555835 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:22:12.566504 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555838 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:22:12.566504 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555840 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:22:12.566504 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555842 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:22:12.566504 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555845 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:22:12.567067 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555847 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:22:12.567067 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555855 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:22:12.567067 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555858 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:22:12.567067 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555861 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:22:12.567067 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555863 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:22:12.567067 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555866 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:22:12.567067 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555869 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:22:12.567067 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555871 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:22:12.567067 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555874 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:22:12.567067 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555876 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:22:12.567067 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555879 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:22:12.567067 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555881 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:22:12.567067 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555884 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:22:12.567067 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555886 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:22:12.567067 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555888 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:22:12.567067 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555891 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:22:12.567067 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555893 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:22:12.567067 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555896 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:22:12.567067 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555898 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:22:12.567522 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555902 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:22:12.567522 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555905 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:22:12.567522 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555907 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:22:12.567522 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555911 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:22:12.567522 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555914 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:22:12.567522 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555917 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:22:12.567522 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555920 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:22:12.567522 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555922 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:22:12.567522 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555924 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:22:12.567522 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555927 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:22:12.567522 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555930 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:22:12.567522 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555932 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:22:12.567522 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555935 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:22:12.567522 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555937 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:22:12.567522 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555939 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:22:12.567522 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555947 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:22:12.567522 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555950 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:22:12.567522 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555953 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:22:12.567522 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555955 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:22:12.567522 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555957 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:22:12.568027 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555960 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:22:12.568027 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555962 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:22:12.568027 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555965 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:22:12.568027 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555967 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:22:12.568027 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555969 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:22:12.568027 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555972 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:22:12.568027 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555974 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:22:12.568027 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555976 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:22:12.568027 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555979 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:22:12.568027 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555981 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:22:12.568027 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555984 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:22:12.568027 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555986 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:22:12.568027 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555990 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:22:12.568027 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555993 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:22:12.568027 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555996 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:22:12.568027 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.555998 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:22:12.568027 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.556001 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:22:12.568027 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.556003 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:22:12.568027 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.556006 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:22:12.568027 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.556008 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:22:12.568492 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.556011 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:22:12.568492 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:12.556013 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:22:12.568492 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.556088 2575 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 16:22:12.568492 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.558959 2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 16:22:12.568492 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.559796 2575 server.go:1014] "Starting client certificate rotation" Oct 30 16:22:12.568492 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.559897 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 16:22:12.568492 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.559932 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 16:22:12.568492 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.564530 2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 16:22:12.572534 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.572513 2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 16:22:12.585319 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.585290 2575 log.go:25] "Validated CRI v1 runtime API" Oct 30 16:22:12.588846 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.588831 2575 log.go:25] "Validated CRI v1 image API" Oct 30 16:22:12.589633 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.589609 2575 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 16:22:12.594961 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.594939 2575 fs.go:135] Filesystem UUIDs: map[32fc8df6-ceea-42e8-92d2-2990ec3d6eb1:/dev/nvme0n1p4 3f22268a-26c0-4a7c-8700-7676b0d0b5bf:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Oct 30 16:22:12.595044 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.594959 2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 16:22:12.600686 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.600578 2575 manager.go:217] Machine: {Timestamp:2026-10-30 16:22:12.599137516 +0000 UTC m=+0.327550149 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3103269 MemoryCapacity:32812023808 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec273da32cfe0dda36f4161300eb9e18 SystemUUID:ec273da3-2cfe-0dda-36f4-161300eb9e18 BootID:b5e11be0-d6ef-4d05-8d25-f802e219ec89 Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406011904 Type:vfs Inodes:4005374 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d9:ac:ba:d1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d9:ac:ba:d1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:16:f6:36:8a:2d:7a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812023808 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 16:22:12.600686 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.600685 2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 16:22:12.600848 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.600760 2575 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 16:22:12.603305 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.603284 2575 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 16:22:12.603473 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.603307 2575 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-180.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 16:22:12.603552 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.603481 2575 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 16:22:12.603552 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.603493 2575 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 16:22:12.603552 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.603542 2575 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 16:22:12.603668 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.603567 2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 16:22:12.604274 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.604262 2575 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 16:22:12.607873 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.607859 2575 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 16:22:12.608013 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.608002 2575 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 16:22:12.608561 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.608550 2575 kubelet.go:493] "Attempting to sync node with API server" Oct 30 16:22:12.608619 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.608568 2575 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 16:22:12.608619 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.608581 2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 16:22:12.608619 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.608594 2575 kubelet.go:405] "Adding apiserver pod source" Oct 30 16:22:12.608736 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.608637 2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 16:22:12.609273 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.609257 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 16:22:12.611210 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.611193 2575 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 16:22:12.616860 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.616845 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 16:22:12.617651 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.617639 2575 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 16:22:12.617698 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.617669 2575 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 16:22:12.617836 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.617826 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 16:22:12.617867 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.617842 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 16:22:12.617867 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.617848 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 16:22:12.617867 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.617853 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 16:22:12.617867 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.617859 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 16:22:12.617867 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.617864 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 16:22:12.618290 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.618282 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 16:22:12.618318 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.618290 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 16:22:12.618318 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.618297 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 16:22:12.618318 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.618303 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 16:22:12.618318 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.618311 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 16:22:12.618430 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.618320 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 16:22:12.618988 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.618979 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 16:22:12.619018 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.618990 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 16:22:12.622522 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.622509 2575 server.go:1263] "Started kubelet" Oct 30 16:22:12.622655 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.622609 2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 16:22:12.622713 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.622676 2575 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 16:22:12.622750 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.622600 2575 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 16:22:12.623250 ip-10-0-132-180 systemd[1]: Started Kubernetes Kubelet. Oct 30 16:22:12.625211 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.625197 2575 server.go:317] "Adding debug handlers to kubelet server" Oct 30 16:22:12.627692 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.627675 2575 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 16:22:12.628090 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:12.628074 2575 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 16:22:12.631006 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.630988 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 16:22:12.631510 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.631494 2575 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 16:22:12.631703 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:12.631682 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-180.ec2.internal\" not found" Oct 30 16:22:12.632206 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.632186 2575 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 16:22:12.632280 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.632212 2575 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 16:22:12.632280 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.632195 2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 16:22:12.632371 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.632321 2575 reconstruct.go:97] "Volume reconstruction finished" Oct 30 16:22:12.632371 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.632330 2575 reconciler.go:29] "Reconciler: start to sync state" Oct 30 16:22:12.632728 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.632713 2575 factory.go:55] Registering systemd factory Oct 30 16:22:12.632728 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.632730 2575 factory.go:223] Registration of the systemd container factory successfully Oct 30 16:22:12.632952 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.632937 2575 factory.go:153] Registering CRI-O factory Oct 30 16:22:12.633018 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.632959 2575 factory.go:223] Registration of the crio container factory successfully Oct 30 16:22:12.633018 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.633013 2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 16:22:12.633083 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.633030 2575 factory.go:103] Registering Raw factory Oct 30 16:22:12.633083 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.633044 2575 manager.go:1199] Started watching for new ooms in manager Oct 30 16:22:12.633918 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.633903 2575 manager.go:319] Starting recovery of all containers Oct 30 16:22:12.647129 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.647111 2575 manager.go:324] Recovery completed Oct 30 16:22:12.651140 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.651127 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 16:22:12.653047 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.653018 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-180.ec2.internal" event="NodeHasSufficientMemory" Oct 30 16:22:12.653117 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.653057 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-180.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 16:22:12.653117 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.653066 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-180.ec2.internal" event="NodeHasSufficientPID" Oct 30 16:22:12.653548 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.653536 2575 cpu_manager.go:226] "Starting" policy="none" Oct 30 16:22:12.653548 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.653545 2575 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 16:22:12.653694 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.653563 2575 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 16:22:12.653694 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.653545 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 16:22:12.654214 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.654197 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-180.ec2.internal" event="NodeHasSufficientMemory" Oct 30 16:22:12.654288 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.654222 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-180.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 16:22:12.654288 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.654252 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-180.ec2.internal" event="NodeHasSufficientPID" Oct 30 16:22:12.655101 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.655077 2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-180.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 16:22:12.655846 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.655831 2575 policy_none.go:50] "Start" Oct 30 16:22:12.655887 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.655851 2575 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 16:22:12.655887 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.655871 2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 16:22:12.655964 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:12.655069 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-180.ec2.internal.18e35b2eab3360d3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-180.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-180.ec2.internal,},FirstTimestamp:2026-10-30 16:22:12.622483667 +0000 UTC m=+0.350896298,LastTimestamp:2026-10-30 16:22:12.622483667 +0000 UTC m=+0.350896298,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-180.ec2.internal,}" Oct 30 16:22:12.657866 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.657855 2575 policy_none.go:44] "Start" Oct 30 16:22:12.664301 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:12.664283 2575 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-180.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 16:22:12.668053 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.668037 2575 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 16:22:12.668096 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.668087 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4qgd4" Oct 30 16:22:12.669868 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:12.669839 2575 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 16:22:12.669868 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:12.669839 2575 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-180.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 16:22:12.669868 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:12.669844 2575 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 16:22:12.679202 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.679185 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4qgd4" Oct 30 16:22:12.686308 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:12.686250 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-180.ec2.internal.18e35b2ead05b4a4 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-180.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-180.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-180.ec2.internal,},FirstTimestamp:2026-10-30 16:22:12.6530449 +0000 UTC m=+0.381457531,LastTimestamp:2026-10-30 16:22:12.6530449 +0000 UTC m=+0.381457531,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-180.ec2.internal,}" Oct 30 16:22:12.693396 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.693355 2575 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 16:22:12.695877 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.695864 2575 manager.go:341] "Starting Device Plugin manager" Oct 30 16:22:12.695948 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:12.695896 2575 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 16:22:12.695948 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.695907 2575 server.go:87] "Starting device plugin registration server" Oct 30 16:22:12.696146 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.696133 2575 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 16:22:12.696200 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.696147 2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 16:22:12.696368 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.696239 2575 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 16:22:12.696462 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.696439 2575 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 16:22:12.696462 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.696448 2575 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 16:22:12.696772 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:12.696755 2575 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 16:22:12.696828 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:12.696821 2575 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-180.ec2.internal\" not found" Oct 30 16:22:12.763288 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.763253 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 16:22:12.764615 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.764595 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 16:22:12.764615 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.764618 2575 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 16:22:12.764777 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.764636 2575 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 16:22:12.764777 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:12.764666 2575 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 16:22:12.779485 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.779462 2575 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 16:22:12.797073 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.797034 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 16:22:12.797899 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.797885 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-180.ec2.internal" event="NodeHasSufficientMemory" Oct 30 16:22:12.797970 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.797914 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-180.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 16:22:12.797970 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.797924 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-180.ec2.internal" event="NodeHasSufficientPID" Oct 30 16:22:12.797970 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.797948 2575 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-132-180.ec2.internal" Oct 30 16:22:12.815684 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.815667 2575 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-132-180.ec2.internal" Oct 30 16:22:12.815751 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:12.815689 2575 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-180.ec2.internal\": node \"ip-10-0-132-180.ec2.internal\" not found" Oct 30 16:22:12.856601 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:12.856581 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-180.ec2.internal\" not found" Oct 30 16:22:12.865719 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.865698 2575 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-180.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-180.ec2.internal"] Oct 30 16:22:12.865833 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.865753 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 16:22:12.866677 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.866662 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-180.ec2.internal" event="NodeHasSufficientMemory" Oct 30 16:22:12.866817 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.866695 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-180.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 16:22:12.866817 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.866709 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-180.ec2.internal" event="NodeHasSufficientPID" Oct 30 16:22:12.868995 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.868979 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 16:22:12.869153 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.869139 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-180.ec2.internal" Oct 30 16:22:12.869197 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.869168 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 16:22:12.869818 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.869775 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-180.ec2.internal" event="NodeHasSufficientMemory" Oct 30 16:22:12.869818 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.869775 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-180.ec2.internal" event="NodeHasSufficientMemory" Oct 30 16:22:12.869975 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.869847 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-180.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 16:22:12.869975 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.869864 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-180.ec2.internal" event="NodeHasSufficientPID" Oct 30 16:22:12.869975 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.869821 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-180.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 16:22:12.869975 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.869921 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-180.ec2.internal" event="NodeHasSufficientPID" Oct 30 16:22:12.871947 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.871929 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-180.ec2.internal" Oct 30 16:22:12.872031 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.871954 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 16:22:12.872628 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.872613 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-180.ec2.internal" event="NodeHasSufficientMemory" Oct 30 16:22:12.872698 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.872639 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-180.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 16:22:12.872698 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.872653 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-180.ec2.internal" event="NodeHasSufficientPID" Oct 30 16:22:12.901433 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:12.901415 2575 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-180.ec2.internal\" not found" node="ip-10-0-132-180.ec2.internal" Oct 30 16:22:12.904708 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:12.904691 2575 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-180.ec2.internal\" not found" node="ip-10-0-132-180.ec2.internal" Oct 30 16:22:12.933705 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.933685 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/83767d666be2011c3ea225fbea981665-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-180.ec2.internal\" (UID: \"83767d666be2011c3ea225fbea981665\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-180.ec2.internal" Oct 30 16:22:12.933776 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.933709 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9a63059ab80503558edad3ed2b226c76-config\") pod \"kube-apiserver-proxy-ip-10-0-132-180.ec2.internal\" (UID: \"9a63059ab80503558edad3ed2b226c76\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-180.ec2.internal" Oct 30 16:22:12.933776 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:12.933725 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/83767d666be2011c3ea225fbea981665-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-180.ec2.internal\" (UID: \"83767d666be2011c3ea225fbea981665\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-180.ec2.internal" Oct 30 16:22:12.957356 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:12.957337 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-180.ec2.internal\" not found" Oct 30 16:22:13.033947 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:13.033923 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/83767d666be2011c3ea225fbea981665-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-180.ec2.internal\" (UID: \"83767d666be2011c3ea225fbea981665\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-180.ec2.internal" Oct 30 16:22:13.034067 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:13.033952 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/83767d666be2011c3ea225fbea981665-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-180.ec2.internal\" (UID: \"83767d666be2011c3ea225fbea981665\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-180.ec2.internal" Oct 30 16:22:13.034067 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:13.033966 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9a63059ab80503558edad3ed2b226c76-config\") pod \"kube-apiserver-proxy-ip-10-0-132-180.ec2.internal\" (UID: \"9a63059ab80503558edad3ed2b226c76\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-180.ec2.internal" Oct 30 16:22:13.034067 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:13.034020 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9a63059ab80503558edad3ed2b226c76-config\") pod \"kube-apiserver-proxy-ip-10-0-132-180.ec2.internal\" (UID: \"9a63059ab80503558edad3ed2b226c76\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-180.ec2.internal" Oct 30 16:22:13.034067 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:13.034027 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/83767d666be2011c3ea225fbea981665-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-180.ec2.internal\" (UID: \"83767d666be2011c3ea225fbea981665\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-180.ec2.internal" Oct 30 16:22:13.034067 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:13.034040 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/83767d666be2011c3ea225fbea981665-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-180.ec2.internal\" (UID: \"83767d666be2011c3ea225fbea981665\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-180.ec2.internal" Oct 30 16:22:13.057554 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:13.057498 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-180.ec2.internal\" not found" Oct 30 16:22:13.157921 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:13.157895 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-180.ec2.internal\" not found" Oct 30 16:22:13.205080 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:13.205059 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-180.ec2.internal" Oct 30 16:22:13.208655 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:13.208635 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-180.ec2.internal" Oct 30 16:22:13.258986 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:13.258957 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-180.ec2.internal\" not found" Oct 30 16:22:13.359555 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:13.359492 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-180.ec2.internal\" not found" Oct 30 16:22:13.460005 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:13.459975 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-180.ec2.internal\" not found" Oct 30 16:22:13.559531 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:13.559496 2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 16:22:13.559989 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:13.559650 2575 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 16:22:13.560621 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:13.560592 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-180.ec2.internal\" not found" Oct 30 16:22:13.661290 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:13.661255 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-180.ec2.internal\" not found" Oct 30 16:22:13.683322 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:13.683285 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 16:17:13 +0000 UTC" deadline="2028-05-29 17:19:20.517084079 +0000 UTC" Oct 30 16:22:13.683322 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:13.683318 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13848h57m6.833769225s" Oct 30 16:22:13.725090 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:13.725063 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83767d666be2011c3ea225fbea981665.slice/crio-c45d537cf983095d6f7f6e59329996acbabab075cf91370b12a541c56665f537 WatchSource:0}: Error finding container c45d537cf983095d6f7f6e59329996acbabab075cf91370b12a541c56665f537: Status 404 returned error can't find the container with id c45d537cf983095d6f7f6e59329996acbabab075cf91370b12a541c56665f537 Oct 30 16:22:13.728025 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:13.728006 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 16:22:13.730319 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:13.729866 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-180.ec2.internal" event="NodeHasSufficientMemory" Oct 30 16:22:13.730319 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:13.729899 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-180.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 16:22:13.730319 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:13.729913 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-180.ec2.internal" event="NodeHasSufficientPID" Oct 30 16:22:13.730664 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:13.730642 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a63059ab80503558edad3ed2b226c76.slice/crio-ec42a59d26b309a2abc07c1f01503b90314473de1dca75a3ae30db9221dcb9dc WatchSource:0}: Error finding container ec42a59d26b309a2abc07c1f01503b90314473de1dca75a3ae30db9221dcb9dc: Status 404 returned error can't find the container with id ec42a59d26b309a2abc07c1f01503b90314473de1dca75a3ae30db9221dcb9dc Oct 30 16:22:13.731932 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:13.731918 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 16:22:13.761706 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:13.761685 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-180.ec2.internal\" not found" Oct 30 16:22:13.768035 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:13.767993 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-180.ec2.internal" event={"ID":"9a63059ab80503558edad3ed2b226c76","Type":"ContainerStarted","Data":"ec42a59d26b309a2abc07c1f01503b90314473de1dca75a3ae30db9221dcb9dc"} Oct 30 16:22:13.769046 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:13.769024 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-180.ec2.internal" event={"ID":"83767d666be2011c3ea225fbea981665","Type":"ContainerStarted","Data":"c45d537cf983095d6f7f6e59329996acbabab075cf91370b12a541c56665f537"} Oct 30 16:22:13.862730 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:13.862655 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-180.ec2.internal\" not found" Oct 30 16:22:13.963153 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:13.963124 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-180.ec2.internal\" not found" Oct 30 16:22:14.002231 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.002205 2575 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 16:22:14.032335 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.032306 2575 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-180.ec2.internal" Oct 30 16:22:14.053025 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.053001 2575 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 16:22:14.054706 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.054690 2575 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-180.ec2.internal" Oct 30 16:22:14.071836 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.071818 2575 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 16:22:14.195304 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.195228 2575 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 16:22:14.195499 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.195482 2575 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 16:22:14.610385 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.610359 2575 apiserver.go:52] "Watching apiserver" Oct 30 16:22:14.632301 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.632273 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 16:22:14.635210 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.635182 2575 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 16:22:14.635575 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.635551 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-pg4kq","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z96s5","openshift-image-registry/node-ca-jlr9c","openshift-cluster-node-tuning-operator/tuned-xbbxh","openshift-multus/multus-9nfhv","openshift-multus/multus-additional-cni-plugins-5jkb7","kube-system/kube-apiserver-proxy-ip-10-0-132-180.ec2.internal","openshift-network-diagnostics/network-check-target-426gj","openshift-network-operator/iptables-alerter-z7jnf","openshift-ovn-kubernetes/ovnkube-node-56v9s","kube-system/global-pull-secret-syncer-2lrlr","openshift-dns/node-resolver-r5ppb","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-180.ec2.internal","openshift-multus/network-metrics-daemon-mbqwf"] Oct 30 16:22:14.638498 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.638477 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:14.640916 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.640894 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2lrlr" Oct 30 16:22:14.641081 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:14.640965 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2lrlr" podUID="dc77c50f-abe0-4d9c-a88b-7a186c17dc3e" Oct 30 16:22:14.643591 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.643570 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jlr9c" Oct 30 16:22:14.646277 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.646252 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:14.649205 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.649183 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9nfhv" Oct 30 16:22:14.652449 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.652430 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5jkb7" Oct 30 16:22:14.653503 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.653293 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 16:22:14.653503 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.653387 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 16:22:14.653503 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.653399 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 16:22:14.653503 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.653455 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9kf5z\"" Oct 30 16:22:14.653737 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.653716 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 16:22:14.654101 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.654082 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 16:22:14.656206 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.656184 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-z7jnf" Oct 30 16:22:14.657310 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.657287 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 16:22:14.657406 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.657385 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 16:22:14.657518 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.657503 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 16:22:14.657610 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.657597 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 16:22:14.659489 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.658828 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mbqwf" Oct 30 16:22:14.659489 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:14.658963 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mbqwf" podUID="b804fc37-0b07-4edf-8720-b31b5698b935" Oct 30 16:22:14.659489 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.659405 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tdd2t\"" Oct 30 16:22:14.661651 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.661633 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z96s5" Oct 30 16:22:14.664388 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.664367 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pg4kq" Oct 30 16:22:14.667432 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.667394 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-426gj" Oct 30 16:22:14.667530 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:14.667503 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-426gj" podUID="78b9342d-5361-4a8c-a771-0c1aea26a48d" Oct 30 16:22:14.670796 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.670464 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-r5ppb" Oct 30 16:22:14.677056 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.677033 2575 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 16:22:14.691553 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.691527 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-c4k8l" Oct 30 16:22:14.705019 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.704998 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-c4k8l" Oct 30 16:22:14.734208 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.734188 2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 16:22:14.743378 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.743342 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-host-run-netns\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:14.743471 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.743381 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-run-ovn\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:14.743471 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.743411 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a773f3a-800c-4c84-b18f-7a48cdd92692-var-lib-kubelet\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:14.743471 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.743438 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc77c50f-abe0-4d9c-a88b-7a186c17dc3e-original-pull-secret\") pod \"global-pull-secret-syncer-2lrlr\" (UID: \"dc77c50f-abe0-4d9c-a88b-7a186c17dc3e\") " pod="kube-system/global-pull-secret-syncer-2lrlr" Oct 30 16:22:14.743471 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.743465 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e02b2b2-c05c-4821-adfe-02f2a6664267-system-cni-dir\") pod \"multus-additional-cni-plugins-5jkb7\" (UID: \"8e02b2b2-c05c-4821-adfe-02f2a6664267\") " pod="openshift-multus/multus-additional-cni-plugins-5jkb7" Oct 30 16:22:14.743684 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.743511 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7c09760d-2160-41d4-b991-f6324ad5aff7-konnectivity-ca\") pod \"konnectivity-agent-pg4kq\" (UID: \"7c09760d-2160-41d4-b991-f6324ad5aff7\") " pod="kube-system/konnectivity-agent-pg4kq" Oct 30 16:22:14.743684 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.743569 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-multus-cni-dir\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:14.743684 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.743604 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zftsq\" (UniqueName: \"kubernetes.io/projected/c407642e-2f4a-4c47-906e-56137857d162-kube-api-access-zftsq\") pod \"iptables-alerter-z7jnf\" (UID: \"c407642e-2f4a-4c47-906e-56137857d162\") " pod="openshift-network-operator/iptables-alerter-z7jnf" Oct 30 16:22:14.743684 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.743632 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m78c4\" (UniqueName: \"kubernetes.io/projected/8e02b2b2-c05c-4821-adfe-02f2a6664267-kube-api-access-m78c4\") pod \"multus-additional-cni-plugins-5jkb7\" (UID: \"8e02b2b2-c05c-4821-adfe-02f2a6664267\") " pod="openshift-multus/multus-additional-cni-plugins-5jkb7" Oct 30 16:22:14.743684 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.743668 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/394d816a-1e33-433f-abb1-586089620763-sys-fs\") pod \"aws-ebs-csi-driver-node-z96s5\" (UID: \"394d816a-1e33-433f-abb1-586089620763\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z96s5" Oct 30 16:22:14.743951 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.743699 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s64j8\" (UniqueName: \"kubernetes.io/projected/0e649944-a09d-44b1-91d2-fda00b30292d-kube-api-access-s64j8\") pod \"node-resolver-r5ppb\" (UID: \"0e649944-a09d-44b1-91d2-fda00b30292d\") " pod="openshift-dns/node-resolver-r5ppb" Oct 30 16:22:14.743951 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.743729 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-ovn-node-metrics-cert\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:14.743951 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.743754 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3a773f3a-800c-4c84-b18f-7a48cdd92692-sys\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:14.743951 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.743779 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-cnibin\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:14.743951 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.743822 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-host-var-lib-cni-multus\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:14.743951 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.743862 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/394d816a-1e33-433f-abb1-586089620763-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z96s5\" (UID: \"394d816a-1e33-433f-abb1-586089620763\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z96s5" Oct 30 16:22:14.743951 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.743901 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7c09760d-2160-41d4-b991-f6324ad5aff7-agent-certs\") pod \"konnectivity-agent-pg4kq\" (UID: \"7c09760d-2160-41d4-b991-f6324ad5aff7\") " pod="kube-system/konnectivity-agent-pg4kq" Oct 30 16:22:14.743951 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.743941 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:14.744321 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.743988 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5v7j\" (UniqueName: \"kubernetes.io/projected/3a773f3a-800c-4c84-b18f-7a48cdd92692-kube-api-access-m5v7j\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:14.744321 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.744016 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-host-run-k8s-cni-cncf-io\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:14.744321 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.744041 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-host-run-netns\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:14.744321 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.744065 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dlwk\" (UniqueName: \"kubernetes.io/projected/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-kube-api-access-6dlwk\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:14.744321 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.744083 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8e02b2b2-c05c-4821-adfe-02f2a6664267-cni-binary-copy\") pod \"multus-additional-cni-plugins-5jkb7\" (UID: \"8e02b2b2-c05c-4821-adfe-02f2a6664267\") " pod="openshift-multus/multus-additional-cni-plugins-5jkb7" Oct 30 16:22:14.744321 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.744120 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-etc-openvswitch\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:14.744321 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.744159 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxrxb\" (UniqueName: \"kubernetes.io/projected/17bb82ba-5262-47f7-abbc-99f1329500fa-kube-api-access-vxrxb\") pod \"node-ca-jlr9c\" (UID: \"17bb82ba-5262-47f7-abbc-99f1329500fa\") " pod="openshift-image-registry/node-ca-jlr9c" Oct 30 16:22:14.744321 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.744177 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3a773f3a-800c-4c84-b18f-7a48cdd92692-etc-tuned\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:14.744321 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.744203 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-cni-binary-copy\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:14.744321 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.744218 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-multus-socket-dir-parent\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:14.744321 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.744253 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/394d816a-1e33-433f-abb1-586089620763-device-dir\") pod \"aws-ebs-csi-driver-node-z96s5\" (UID: \"394d816a-1e33-433f-abb1-586089620763\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z96s5" Oct 30 16:22:14.744321 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.744273 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3a773f3a-800c-4c84-b18f-7a48cdd92692-etc-modprobe-d\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:14.744321 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.744289 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3a773f3a-800c-4c84-b18f-7a48cdd92692-etc-sysconfig\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:14.744980 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.744322 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b804fc37-0b07-4edf-8720-b31b5698b935-metrics-certs\") pod \"network-metrics-daemon-mbqwf\" (UID: \"b804fc37-0b07-4edf-8720-b31b5698b935\") " pod="openshift-multus/network-metrics-daemon-mbqwf" Oct 30 16:22:14.744980 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.744353 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8e02b2b2-c05c-4821-adfe-02f2a6664267-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5jkb7\" (UID: \"8e02b2b2-c05c-4821-adfe-02f2a6664267\") " pod="openshift-multus/multus-additional-cni-plugins-5jkb7" Oct 30 16:22:14.744980 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.744378 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-run-systemd\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:14.744980 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.744408 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-var-lib-openvswitch\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:14.744980 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.744439 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/17bb82ba-5262-47f7-abbc-99f1329500fa-serviceca\") pod \"node-ca-jlr9c\" (UID: \"17bb82ba-5262-47f7-abbc-99f1329500fa\") " pod="openshift-image-registry/node-ca-jlr9c" Oct 30 16:22:14.744980 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.744470 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a773f3a-800c-4c84-b18f-7a48cdd92692-tmp\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:14.744980 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.744490 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-host-var-lib-cni-bin\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:14.744980 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.744528 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3a773f3a-800c-4c84-b18f-7a48cdd92692-lib-modules\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:14.744980 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.744550 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb9mk\" (UniqueName: \"kubernetes.io/projected/394d816a-1e33-433f-abb1-586089620763-kube-api-access-nb9mk\") pod \"aws-ebs-csi-driver-node-z96s5\" (UID: \"394d816a-1e33-433f-abb1-586089620763\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z96s5" Oct 30 16:22:14.744980 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.744565 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vrcg\" (UniqueName: \"kubernetes.io/projected/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-kube-api-access-8vrcg\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:14.744980 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.744586 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3a773f3a-800c-4c84-b18f-7a48cdd92692-etc-sysctl-d\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:14.744980 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.744617 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-system-cni-dir\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:14.744980 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.744633 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8e02b2b2-c05c-4821-adfe-02f2a6664267-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5jkb7\" (UID: \"8e02b2b2-c05c-4821-adfe-02f2a6664267\") " pod="openshift-multus/multus-additional-cni-plugins-5jkb7" Oct 30 16:22:14.744980 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.744647 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/394d816a-1e33-433f-abb1-586089620763-registration-dir\") pod \"aws-ebs-csi-driver-node-z96s5\" (UID: \"394d816a-1e33-433f-abb1-586089620763\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z96s5" Oct 30 16:22:14.744980 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.744662 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/394d816a-1e33-433f-abb1-586089620763-etc-selinux\") pod \"aws-ebs-csi-driver-node-z96s5\" (UID: \"394d816a-1e33-433f-abb1-586089620763\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z96s5" Oct 30 16:22:14.744980 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.744682 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srfcg\" (UniqueName: \"kubernetes.io/projected/78b9342d-5361-4a8c-a771-0c1aea26a48d-kube-api-access-srfcg\") pod \"network-check-target-426gj\" (UID: \"78b9342d-5361-4a8c-a771-0c1aea26a48d\") " pod="openshift-network-diagnostics/network-check-target-426gj" Oct 30 16:22:14.745798 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.744703 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-node-log\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:14.745798 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.744725 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3a773f3a-800c-4c84-b18f-7a48cdd92692-etc-systemd\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:14.745798 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.744777 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3a773f3a-800c-4c84-b18f-7a48cdd92692-host\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:14.745798 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.744803 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-host-var-lib-kubelet\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:14.745798 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.744840 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-host-run-multus-certs\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:14.745798 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.744880 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8vll\" (UniqueName: \"kubernetes.io/projected/b804fc37-0b07-4edf-8720-b31b5698b935-kube-api-access-m8vll\") pod \"network-metrics-daemon-mbqwf\" (UID: \"b804fc37-0b07-4edf-8720-b31b5698b935\") " pod="openshift-multus/network-metrics-daemon-mbqwf" Oct 30 16:22:14.745798 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.744905 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e649944-a09d-44b1-91d2-fda00b30292d-tmp-dir\") pod \"node-resolver-r5ppb\" (UID: \"0e649944-a09d-44b1-91d2-fda00b30292d\") " pod="openshift-dns/node-resolver-r5ppb" Oct 30 16:22:14.745798 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.744931 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-log-socket\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:14.745798 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.744958 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-ovnkube-script-lib\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:14.745798 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.744983 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3a773f3a-800c-4c84-b18f-7a48cdd92692-etc-sysctl-conf\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:14.745798 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.745012 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3a773f3a-800c-4c84-b18f-7a48cdd92692-run\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:14.745798 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.745037 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-multus-daemon-config\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:14.745798 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.745056 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dc77c50f-abe0-4d9c-a88b-7a186c17dc3e-kubelet-config\") pod \"global-pull-secret-syncer-2lrlr\" (UID: \"dc77c50f-abe0-4d9c-a88b-7a186c17dc3e\") " pod="kube-system/global-pull-secret-syncer-2lrlr" Oct 30 16:22:14.745798 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.745071 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8e02b2b2-c05c-4821-adfe-02f2a6664267-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5jkb7\" (UID: \"8e02b2b2-c05c-4821-adfe-02f2a6664267\") " pod="openshift-multus/multus-additional-cni-plugins-5jkb7" Oct 30 16:22:14.745798 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.745105 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0e649944-a09d-44b1-91d2-fda00b30292d-hosts-file\") pod \"node-resolver-r5ppb\" (UID: \"0e649944-a09d-44b1-91d2-fda00b30292d\") " pod="openshift-dns/node-resolver-r5ppb" Oct 30 16:22:14.745798 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.745130 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-systemd-units\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:14.745798 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.745226 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-host-slash\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:14.746642 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.745243 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c407642e-2f4a-4c47-906e-56137857d162-host-slash\") pod \"iptables-alerter-z7jnf\" (UID: \"c407642e-2f4a-4c47-906e-56137857d162\") " pod="openshift-network-operator/iptables-alerter-z7jnf" Oct 30 16:22:14.746642 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.745259 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-host-cni-netd\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:14.746642 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.745274 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-env-overrides\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:14.746642 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.745291 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a773f3a-800c-4c84-b18f-7a48cdd92692-etc-kubernetes\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:14.746642 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.745314 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-hostroot\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:14.746642 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.745332 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c407642e-2f4a-4c47-906e-56137857d162-iptables-alerter-script\") pod \"iptables-alerter-z7jnf\" (UID: \"c407642e-2f4a-4c47-906e-56137857d162\") " pod="openshift-network-operator/iptables-alerter-z7jnf" Oct 30 16:22:14.746642 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.745355 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8e02b2b2-c05c-4821-adfe-02f2a6664267-cnibin\") pod \"multus-additional-cni-plugins-5jkb7\" (UID: \"8e02b2b2-c05c-4821-adfe-02f2a6664267\") " pod="openshift-multus/multus-additional-cni-plugins-5jkb7" Oct 30 16:22:14.746642 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.745370 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-host-kubelet\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:14.746642 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.745401 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-host-run-ovn-kubernetes\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:14.746642 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.745419 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-ovnkube-config\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:14.746642 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.745449 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-multus-conf-dir\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:14.746642 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.745468 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-run-openvswitch\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:14.746642 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.745493 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-host-cni-bin\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:14.746642 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.745508 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/17bb82ba-5262-47f7-abbc-99f1329500fa-host\") pod \"node-ca-jlr9c\" (UID: \"17bb82ba-5262-47f7-abbc-99f1329500fa\") " pod="openshift-image-registry/node-ca-jlr9c" Oct 30 16:22:14.746642 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.745523 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-etc-kubernetes\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:14.746642 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.745540 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dc77c50f-abe0-4d9c-a88b-7a186c17dc3e-dbus\") pod \"global-pull-secret-syncer-2lrlr\" (UID: \"dc77c50f-abe0-4d9c-a88b-7a186c17dc3e\") " pod="kube-system/global-pull-secret-syncer-2lrlr" Oct 30 16:22:14.746642 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.745555 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/394d816a-1e33-433f-abb1-586089620763-socket-dir\") pod \"aws-ebs-csi-driver-node-z96s5\" (UID: \"394d816a-1e33-433f-abb1-586089620763\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z96s5" Oct 30 16:22:14.760713 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.760684 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 16:22:14.760815 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.760752 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 16:22:14.760815 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.760762 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 16:22:14.760815 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.760805 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 16:22:14.761019 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.760865 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 16:22:14.761019 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.760959 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 16:22:14.761019 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.760978 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 16:22:14.761191 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.761030 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 16:22:14.761191 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.761099 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 16:22:14.761191 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.761143 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 16:22:14.761191 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.761183 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-h4vt2\"" Oct 30 16:22:14.761408 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.761242 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 16:22:14.761408 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.761243 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 16:22:14.761408 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.761288 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 16:22:14.761408 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.761335 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 16:22:14.761832 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.761656 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 16:22:14.761832 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.761698 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 16:22:14.761832 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.761799 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 16:22:14.762051 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.761894 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-lf54z\"" Oct 30 16:22:14.762051 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.761980 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 16:22:14.762138 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.762055 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-86hj4\"" Oct 30 16:22:14.762182 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.762148 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-5n7vr\"" Oct 30 16:22:14.762269 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.762250 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-m9x9f\"" Oct 30 16:22:14.762385 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.762353 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-d6852\"" Oct 30 16:22:14.762904 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.762888 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rjhbz\"" Oct 30 16:22:14.846073 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.845987 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/394d816a-1e33-433f-abb1-586089620763-socket-dir\") pod \"aws-ebs-csi-driver-node-z96s5\" (UID: \"394d816a-1e33-433f-abb1-586089620763\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z96s5" Oct 30 16:22:14.846073 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.846035 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-host-run-netns\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:14.846073 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.846064 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-run-ovn\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:14.846312 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.846092 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a773f3a-800c-4c84-b18f-7a48cdd92692-var-lib-kubelet\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:14.846312 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.846120 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc77c50f-abe0-4d9c-a88b-7a186c17dc3e-original-pull-secret\") pod \"global-pull-secret-syncer-2lrlr\" (UID: \"dc77c50f-abe0-4d9c-a88b-7a186c17dc3e\") " pod="kube-system/global-pull-secret-syncer-2lrlr" Oct 30 16:22:14.846312 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.846146 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e02b2b2-c05c-4821-adfe-02f2a6664267-system-cni-dir\") pod \"multus-additional-cni-plugins-5jkb7\" (UID: \"8e02b2b2-c05c-4821-adfe-02f2a6664267\") " pod="openshift-multus/multus-additional-cni-plugins-5jkb7" Oct 30 16:22:14.846312 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.846172 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7c09760d-2160-41d4-b991-f6324ad5aff7-konnectivity-ca\") pod \"konnectivity-agent-pg4kq\" (UID: \"7c09760d-2160-41d4-b991-f6324ad5aff7\") " pod="kube-system/konnectivity-agent-pg4kq" Oct 30 16:22:14.846312 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.846197 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-multus-cni-dir\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:14.846312 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.846224 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zftsq\" (UniqueName: \"kubernetes.io/projected/c407642e-2f4a-4c47-906e-56137857d162-kube-api-access-zftsq\") pod \"iptables-alerter-z7jnf\" (UID: \"c407642e-2f4a-4c47-906e-56137857d162\") " pod="openshift-network-operator/iptables-alerter-z7jnf" Oct 30 16:22:14.846312 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.846250 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m78c4\" (UniqueName: \"kubernetes.io/projected/8e02b2b2-c05c-4821-adfe-02f2a6664267-kube-api-access-m78c4\") pod \"multus-additional-cni-plugins-5jkb7\" (UID: \"8e02b2b2-c05c-4821-adfe-02f2a6664267\") " pod="openshift-multus/multus-additional-cni-plugins-5jkb7" Oct 30 16:22:14.846312 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.846278 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/394d816a-1e33-433f-abb1-586089620763-sys-fs\") pod \"aws-ebs-csi-driver-node-z96s5\" (UID: \"394d816a-1e33-433f-abb1-586089620763\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z96s5" Oct 30 16:22:14.846312 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.846303 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s64j8\" (UniqueName: \"kubernetes.io/projected/0e649944-a09d-44b1-91d2-fda00b30292d-kube-api-access-s64j8\") pod \"node-resolver-r5ppb\" (UID: \"0e649944-a09d-44b1-91d2-fda00b30292d\") " pod="openshift-dns/node-resolver-r5ppb" Oct 30 16:22:14.846702 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.846327 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-ovn-node-metrics-cert\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:14.846702 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.846351 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3a773f3a-800c-4c84-b18f-7a48cdd92692-sys\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:14.846702 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.846374 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-cnibin\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:14.846702 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.846398 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-host-var-lib-cni-multus\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:14.846702 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.846424 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/394d816a-1e33-433f-abb1-586089620763-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z96s5\" (UID: \"394d816a-1e33-433f-abb1-586089620763\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z96s5" Oct 30 16:22:14.846702 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.846448 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7c09760d-2160-41d4-b991-f6324ad5aff7-agent-certs\") pod \"konnectivity-agent-pg4kq\" (UID: \"7c09760d-2160-41d4-b991-f6324ad5aff7\") " pod="kube-system/konnectivity-agent-pg4kq" Oct 30 16:22:14.846702 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.846475 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:14.846702 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.846511 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5v7j\" (UniqueName: \"kubernetes.io/projected/3a773f3a-800c-4c84-b18f-7a48cdd92692-kube-api-access-m5v7j\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:14.846702 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.846536 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-host-run-k8s-cni-cncf-io\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:14.846702 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.846562 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-host-run-netns\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:14.846702 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.846587 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dlwk\" (UniqueName: \"kubernetes.io/projected/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-kube-api-access-6dlwk\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:14.846702 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.846610 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8e02b2b2-c05c-4821-adfe-02f2a6664267-cni-binary-copy\") pod \"multus-additional-cni-plugins-5jkb7\" (UID: \"8e02b2b2-c05c-4821-adfe-02f2a6664267\") " pod="openshift-multus/multus-additional-cni-plugins-5jkb7" Oct 30 16:22:14.846702 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.846632 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-etc-openvswitch\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:14.846702 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.846648 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxrxb\" (UniqueName: \"kubernetes.io/projected/17bb82ba-5262-47f7-abbc-99f1329500fa-kube-api-access-vxrxb\") pod \"node-ca-jlr9c\" (UID: \"17bb82ba-5262-47f7-abbc-99f1329500fa\") " pod="openshift-image-registry/node-ca-jlr9c" Oct 30 16:22:14.846702 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.846681 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3a773f3a-800c-4c84-b18f-7a48cdd92692-etc-tuned\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:14.846702 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.846696 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-cni-binary-copy\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:14.847512 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.846711 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-multus-socket-dir-parent\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:14.847512 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.846727 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/394d816a-1e33-433f-abb1-586089620763-device-dir\") pod \"aws-ebs-csi-driver-node-z96s5\" (UID: \"394d816a-1e33-433f-abb1-586089620763\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z96s5" Oct 30 16:22:14.847512 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.846743 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3a773f3a-800c-4c84-b18f-7a48cdd92692-etc-modprobe-d\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:14.847512 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.846760 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3a773f3a-800c-4c84-b18f-7a48cdd92692-etc-sysconfig\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:14.847512 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.846776 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b804fc37-0b07-4edf-8720-b31b5698b935-metrics-certs\") pod \"network-metrics-daemon-mbqwf\" (UID: \"b804fc37-0b07-4edf-8720-b31b5698b935\") " pod="openshift-multus/network-metrics-daemon-mbqwf" Oct 30 16:22:14.847512 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.846793 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8e02b2b2-c05c-4821-adfe-02f2a6664267-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5jkb7\" (UID: \"8e02b2b2-c05c-4821-adfe-02f2a6664267\") " pod="openshift-multus/multus-additional-cni-plugins-5jkb7" Oct 30 16:22:14.847512 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.846813 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-run-systemd\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:14.847512 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.846828 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-var-lib-openvswitch\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:14.847512 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.846843 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/17bb82ba-5262-47f7-abbc-99f1329500fa-serviceca\") pod \"node-ca-jlr9c\" (UID: \"17bb82ba-5262-47f7-abbc-99f1329500fa\") " pod="openshift-image-registry/node-ca-jlr9c" Oct 30 16:22:14.847512 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.846876 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a773f3a-800c-4c84-b18f-7a48cdd92692-tmp\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:14.847512 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.846900 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-host-var-lib-cni-bin\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:14.847512 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.846926 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3a773f3a-800c-4c84-b18f-7a48cdd92692-lib-modules\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:14.847512 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.846948 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:14.847512 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.846991 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb9mk\" (UniqueName: \"kubernetes.io/projected/394d816a-1e33-433f-abb1-586089620763-kube-api-access-nb9mk\") pod \"aws-ebs-csi-driver-node-z96s5\" (UID: \"394d816a-1e33-433f-abb1-586089620763\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z96s5" Oct 30 16:22:14.847512 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.847001 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-run-ovn\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:14.847512 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.847010 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-host-run-netns\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:14.847512 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.847014 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3a773f3a-800c-4c84-b18f-7a48cdd92692-sys\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:14.848355 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.847026 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vrcg\" (UniqueName: \"kubernetes.io/projected/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-kube-api-access-8vrcg\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:14.848355 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.847049 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-cnibin\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:14.848355 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.847075 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3a773f3a-800c-4c84-b18f-7a48cdd92692-etc-sysctl-d\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:14.848355 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.847088 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/394d816a-1e33-433f-abb1-586089620763-sys-fs\") pod \"aws-ebs-csi-driver-node-z96s5\" (UID: \"394d816a-1e33-433f-abb1-586089620763\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z96s5" Oct 30 16:22:14.848355 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.847102 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-host-var-lib-cni-multus\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:14.848355 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.847106 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-system-cni-dir\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:14.848355 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.847148 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8e02b2b2-c05c-4821-adfe-02f2a6664267-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5jkb7\" (UID: \"8e02b2b2-c05c-4821-adfe-02f2a6664267\") " pod="openshift-multus/multus-additional-cni-plugins-5jkb7" Oct 30 16:22:14.848355 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.847177 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/394d816a-1e33-433f-abb1-586089620763-registration-dir\") pod \"aws-ebs-csi-driver-node-z96s5\" (UID: \"394d816a-1e33-433f-abb1-586089620763\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z96s5" Oct 30 16:22:14.848355 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.847204 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/394d816a-1e33-433f-abb1-586089620763-etc-selinux\") pod \"aws-ebs-csi-driver-node-z96s5\" (UID: \"394d816a-1e33-433f-abb1-586089620763\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z96s5" Oct 30 16:22:14.848355 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.847233 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srfcg\" (UniqueName: \"kubernetes.io/projected/78b9342d-5361-4a8c-a771-0c1aea26a48d-kube-api-access-srfcg\") pod \"network-check-target-426gj\" (UID: \"78b9342d-5361-4a8c-a771-0c1aea26a48d\") " pod="openshift-network-diagnostics/network-check-target-426gj" Oct 30 16:22:14.848355 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.847236 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e02b2b2-c05c-4821-adfe-02f2a6664267-system-cni-dir\") pod \"multus-additional-cni-plugins-5jkb7\" (UID: \"8e02b2b2-c05c-4821-adfe-02f2a6664267\") " pod="openshift-multus/multus-additional-cni-plugins-5jkb7" Oct 30 16:22:14.848355 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.847259 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-node-log\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:14.848355 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.847285 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3a773f3a-800c-4c84-b18f-7a48cdd92692-etc-systemd\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:14.848355 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.847301 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3a773f3a-800c-4c84-b18f-7a48cdd92692-etc-sysconfig\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:14.848355 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.847311 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3a773f3a-800c-4c84-b18f-7a48cdd92692-host\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:14.848355 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.847349 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-host-run-k8s-cni-cncf-io\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:14.848355 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.847350 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/394d816a-1e33-433f-abb1-586089620763-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z96s5\" (UID: \"394d816a-1e33-433f-abb1-586089620763\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z96s5" Oct 30 16:22:14.849363 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.847358 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-run-systemd\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:14.849363 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.847376 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3a773f3a-800c-4c84-b18f-7a48cdd92692-host\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:14.849363 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.847405 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-var-lib-openvswitch\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:14.849363 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:14.847410 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:22:14.849363 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.847433 2575 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 16:22:14.849363 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:14.847484 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b804fc37-0b07-4edf-8720-b31b5698b935-metrics-certs podName:b804fc37-0b07-4edf-8720-b31b5698b935 nodeName:}" failed. No retries permitted until 2026-10-30 16:22:15.347460523 +0000 UTC m=+3.075873161 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b804fc37-0b07-4edf-8720-b31b5698b935-metrics-certs") pod "network-metrics-daemon-mbqwf" (UID: "b804fc37-0b07-4edf-8720-b31b5698b935") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:22:14.849363 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.847503 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/394d816a-1e33-433f-abb1-586089620763-socket-dir\") pod \"aws-ebs-csi-driver-node-z96s5\" (UID: \"394d816a-1e33-433f-abb1-586089620763\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z96s5" Oct 30 16:22:14.849363 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.847589 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3a773f3a-800c-4c84-b18f-7a48cdd92692-etc-sysctl-d\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:14.849363 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.847620 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-system-cni-dir\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:14.849363 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.847617 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-multus-cni-dir\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:14.849363 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:14.847701 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 16:22:14.849363 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.847728 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8e02b2b2-c05c-4821-adfe-02f2a6664267-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5jkb7\" (UID: \"8e02b2b2-c05c-4821-adfe-02f2a6664267\") " pod="openshift-multus/multus-additional-cni-plugins-5jkb7" Oct 30 16:22:14.849363 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:14.847760 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc77c50f-abe0-4d9c-a88b-7a186c17dc3e-original-pull-secret podName:dc77c50f-abe0-4d9c-a88b-7a186c17dc3e nodeName:}" failed. No retries permitted until 2026-10-30 16:22:15.347742474 +0000 UTC m=+3.076155111 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dc77c50f-abe0-4d9c-a88b-7a186c17dc3e-original-pull-secret") pod "global-pull-secret-syncer-2lrlr" (UID: "dc77c50f-abe0-4d9c-a88b-7a186c17dc3e") : object "kube-system"/"original-pull-secret" not registered Oct 30 16:22:14.849363 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.847787 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/394d816a-1e33-433f-abb1-586089620763-registration-dir\") pod \"aws-ebs-csi-driver-node-z96s5\" (UID: \"394d816a-1e33-433f-abb1-586089620763\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z96s5" Oct 30 16:22:14.849363 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.847819 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a773f3a-800c-4c84-b18f-7a48cdd92692-var-lib-kubelet\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:14.849363 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.847832 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-node-log\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:14.849363 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.847887 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-host-run-netns\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:14.850288 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.847904 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3a773f3a-800c-4c84-b18f-7a48cdd92692-etc-systemd\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:14.850288 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.847914 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-multus-socket-dir-parent\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:14.850288 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.847951 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/394d816a-1e33-433f-abb1-586089620763-device-dir\") pod \"aws-ebs-csi-driver-node-z96s5\" (UID: \"394d816a-1e33-433f-abb1-586089620763\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z96s5" Oct 30 16:22:14.850288 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.847975 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7c09760d-2160-41d4-b991-f6324ad5aff7-konnectivity-ca\") pod \"konnectivity-agent-pg4kq\" (UID: \"7c09760d-2160-41d4-b991-f6324ad5aff7\") " pod="kube-system/konnectivity-agent-pg4kq" Oct 30 16:22:14.850288 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.848001 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/394d816a-1e33-433f-abb1-586089620763-etc-selinux\") pod \"aws-ebs-csi-driver-node-z96s5\" (UID: \"394d816a-1e33-433f-abb1-586089620763\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z96s5" Oct 30 16:22:14.850288 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.848001 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-host-var-lib-kubelet\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:14.850288 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.848034 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-host-run-multus-certs\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:14.850288 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.848076 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3a773f3a-800c-4c84-b18f-7a48cdd92692-etc-modprobe-d\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:14.850288 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.847975 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8e02b2b2-c05c-4821-adfe-02f2a6664267-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5jkb7\" (UID: \"8e02b2b2-c05c-4821-adfe-02f2a6664267\") " pod="openshift-multus/multus-additional-cni-plugins-5jkb7" Oct 30 16:22:14.850288 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.848112 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3a773f3a-800c-4c84-b18f-7a48cdd92692-lib-modules\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:14.850288 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.848200 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8vll\" (UniqueName: \"kubernetes.io/projected/b804fc37-0b07-4edf-8720-b31b5698b935-kube-api-access-m8vll\") pod \"network-metrics-daemon-mbqwf\" (UID: \"b804fc37-0b07-4edf-8720-b31b5698b935\") " pod="openshift-multus/network-metrics-daemon-mbqwf" Oct 30 16:22:14.850288 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.848207 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-host-var-lib-cni-bin\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:14.850288 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.848248 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e649944-a09d-44b1-91d2-fda00b30292d-tmp-dir\") pod \"node-resolver-r5ppb\" (UID: \"0e649944-a09d-44b1-91d2-fda00b30292d\") " pod="openshift-dns/node-resolver-r5ppb" Oct 30 16:22:14.850288 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.848277 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-log-socket\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:14.850288 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.848300 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-ovnkube-script-lib\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:14.850288 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.848319 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-host-var-lib-kubelet\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:14.850288 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.848342 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-etc-openvswitch\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:14.851144 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.848441 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-host-run-multus-certs\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:14.851144 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.848613 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e649944-a09d-44b1-91d2-fda00b30292d-tmp-dir\") pod \"node-resolver-r5ppb\" (UID: \"0e649944-a09d-44b1-91d2-fda00b30292d\") " pod="openshift-dns/node-resolver-r5ppb" Oct 30 16:22:14.851144 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.848672 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-log-socket\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:14.851144 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.848788 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/17bb82ba-5262-47f7-abbc-99f1329500fa-serviceca\") pod \"node-ca-jlr9c\" (UID: \"17bb82ba-5262-47f7-abbc-99f1329500fa\") " pod="openshift-image-registry/node-ca-jlr9c" Oct 30 16:22:14.851144 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.848964 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8e02b2b2-c05c-4821-adfe-02f2a6664267-cni-binary-copy\") pod \"multus-additional-cni-plugins-5jkb7\" (UID: \"8e02b2b2-c05c-4821-adfe-02f2a6664267\") " pod="openshift-multus/multus-additional-cni-plugins-5jkb7" Oct 30 16:22:14.851144 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.849201 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3a773f3a-800c-4c84-b18f-7a48cdd92692-etc-sysctl-conf\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:14.851144 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.849243 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3a773f3a-800c-4c84-b18f-7a48cdd92692-run\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:14.851144 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.849274 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-multus-daemon-config\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:14.851144 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.849326 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dc77c50f-abe0-4d9c-a88b-7a186c17dc3e-kubelet-config\") pod \"global-pull-secret-syncer-2lrlr\" (UID: \"dc77c50f-abe0-4d9c-a88b-7a186c17dc3e\") " pod="kube-system/global-pull-secret-syncer-2lrlr" Oct 30 16:22:14.851144 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.849354 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8e02b2b2-c05c-4821-adfe-02f2a6664267-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5jkb7\" (UID: \"8e02b2b2-c05c-4821-adfe-02f2a6664267\") " pod="openshift-multus/multus-additional-cni-plugins-5jkb7" Oct 30 16:22:14.851144 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.849383 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0e649944-a09d-44b1-91d2-fda00b30292d-hosts-file\") pod \"node-resolver-r5ppb\" (UID: \"0e649944-a09d-44b1-91d2-fda00b30292d\") " pod="openshift-dns/node-resolver-r5ppb" Oct 30 16:22:14.851144 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.849411 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-systemd-units\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:14.851144 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.849447 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-cni-binary-copy\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:14.851144 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.849497 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-host-slash\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:14.851144 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.849529 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c407642e-2f4a-4c47-906e-56137857d162-host-slash\") pod \"iptables-alerter-z7jnf\" (UID: \"c407642e-2f4a-4c47-906e-56137857d162\") " pod="openshift-network-operator/iptables-alerter-z7jnf" Oct 30 16:22:14.851144 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.849553 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-host-cni-netd\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:14.851144 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.849579 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-env-overrides\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:14.851144 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.849606 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a773f3a-800c-4c84-b18f-7a48cdd92692-etc-kubernetes\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:14.851847 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.849641 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dc77c50f-abe0-4d9c-a88b-7a186c17dc3e-kubelet-config\") pod \"global-pull-secret-syncer-2lrlr\" (UID: \"dc77c50f-abe0-4d9c-a88b-7a186c17dc3e\") " pod="kube-system/global-pull-secret-syncer-2lrlr" Oct 30 16:22:14.851847 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.849683 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-hostroot\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:14.851847 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.849712 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c407642e-2f4a-4c47-906e-56137857d162-iptables-alerter-script\") pod \"iptables-alerter-z7jnf\" (UID: \"c407642e-2f4a-4c47-906e-56137857d162\") " pod="openshift-network-operator/iptables-alerter-z7jnf" Oct 30 16:22:14.851847 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.849717 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-host-cni-netd\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:14.851847 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.849731 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0e649944-a09d-44b1-91d2-fda00b30292d-hosts-file\") pod \"node-resolver-r5ppb\" (UID: \"0e649944-a09d-44b1-91d2-fda00b30292d\") " pod="openshift-dns/node-resolver-r5ppb" Oct 30 16:22:14.851847 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.849740 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8e02b2b2-c05c-4821-adfe-02f2a6664267-cnibin\") pod \"multus-additional-cni-plugins-5jkb7\" (UID: \"8e02b2b2-c05c-4821-adfe-02f2a6664267\") " pod="openshift-multus/multus-additional-cni-plugins-5jkb7" Oct 30 16:22:14.851847 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.849754 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-host-slash\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:14.851847 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.849766 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-host-kubelet\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:14.851847 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.849779 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a773f3a-800c-4c84-b18f-7a48cdd92692-etc-kubernetes\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:14.851847 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.849794 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-host-run-ovn-kubernetes\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:14.851847 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.849807 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c407642e-2f4a-4c47-906e-56137857d162-host-slash\") pod \"iptables-alerter-z7jnf\" (UID: \"c407642e-2f4a-4c47-906e-56137857d162\") " pod="openshift-network-operator/iptables-alerter-z7jnf" Oct 30 16:22:14.851847 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.849867 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-hostroot\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:14.851847 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.849886 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8e02b2b2-c05c-4821-adfe-02f2a6664267-cnibin\") pod \"multus-additional-cni-plugins-5jkb7\" (UID: \"8e02b2b2-c05c-4821-adfe-02f2a6664267\") " pod="openshift-multus/multus-additional-cni-plugins-5jkb7" Oct 30 16:22:14.851847 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.849914 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3a773f3a-800c-4c84-b18f-7a48cdd92692-etc-sysctl-conf\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:14.851847 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.849924 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-host-kubelet\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:14.851847 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.849942 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-systemd-units\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:14.851847 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.849968 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-host-run-ovn-kubernetes\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:14.851847 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.850008 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-ovnkube-config\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:14.852606 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.850175 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-multus-conf-dir\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:14.852606 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.850218 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-run-openvswitch\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:14.852606 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.850246 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-host-cni-bin\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:14.852606 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.850271 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/17bb82ba-5262-47f7-abbc-99f1329500fa-host\") pod \"node-ca-jlr9c\" (UID: \"17bb82ba-5262-47f7-abbc-99f1329500fa\") " pod="openshift-image-registry/node-ca-jlr9c" Oct 30 16:22:14.852606 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.850321 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-etc-kubernetes\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:14.852606 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.850348 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dc77c50f-abe0-4d9c-a88b-7a186c17dc3e-dbus\") pod \"global-pull-secret-syncer-2lrlr\" (UID: \"dc77c50f-abe0-4d9c-a88b-7a186c17dc3e\") " pod="kube-system/global-pull-secret-syncer-2lrlr" Oct 30 16:22:14.852606 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.850451 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dc77c50f-abe0-4d9c-a88b-7a186c17dc3e-dbus\") pod \"global-pull-secret-syncer-2lrlr\" (UID: \"dc77c50f-abe0-4d9c-a88b-7a186c17dc3e\") " pod="kube-system/global-pull-secret-syncer-2lrlr" Oct 30 16:22:14.852606 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.850489 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/17bb82ba-5262-47f7-abbc-99f1329500fa-host\") pod \"node-ca-jlr9c\" (UID: \"17bb82ba-5262-47f7-abbc-99f1329500fa\") " pod="openshift-image-registry/node-ca-jlr9c" Oct 30 16:22:14.852606 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.850526 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-multus-conf-dir\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:14.852606 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.850548 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8e02b2b2-c05c-4821-adfe-02f2a6664267-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5jkb7\" (UID: \"8e02b2b2-c05c-4821-adfe-02f2a6664267\") " pod="openshift-multus/multus-additional-cni-plugins-5jkb7" Oct 30 16:22:14.852606 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.850565 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-run-openvswitch\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:14.852606 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.850601 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-host-cni-bin\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:14.852606 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.850619 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-etc-kubernetes\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:14.852606 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.850679 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3a773f3a-800c-4c84-b18f-7a48cdd92692-run\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:14.852606 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.851016 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-ovnkube-script-lib\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:14.852606 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.851349 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-env-overrides\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:14.852606 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.851449 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-ovnkube-config\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:14.852606 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.851677 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a773f3a-800c-4c84-b18f-7a48cdd92692-tmp\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:14.853462 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.851765 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3a773f3a-800c-4c84-b18f-7a48cdd92692-etc-tuned\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:14.853462 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.851790 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c407642e-2f4a-4c47-906e-56137857d162-iptables-alerter-script\") pod \"iptables-alerter-z7jnf\" (UID: \"c407642e-2f4a-4c47-906e-56137857d162\") " pod="openshift-network-operator/iptables-alerter-z7jnf" Oct 30 16:22:14.853462 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.851973 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-ovn-node-metrics-cert\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:14.853462 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.852376 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7c09760d-2160-41d4-b991-f6324ad5aff7-agent-certs\") pod \"konnectivity-agent-pg4kq\" (UID: \"7c09760d-2160-41d4-b991-f6324ad5aff7\") " pod="kube-system/konnectivity-agent-pg4kq" Oct 30 16:22:14.853972 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.853948 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-multus-daemon-config\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:14.871089 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.871060 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zftsq\" (UniqueName: \"kubernetes.io/projected/c407642e-2f4a-4c47-906e-56137857d162-kube-api-access-zftsq\") pod \"iptables-alerter-z7jnf\" (UID: \"c407642e-2f4a-4c47-906e-56137857d162\") " pod="openshift-network-operator/iptables-alerter-z7jnf" Oct 30 16:22:14.871286 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.871262 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vrcg\" (UniqueName: \"kubernetes.io/projected/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-kube-api-access-8vrcg\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:14.871733 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.871710 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5v7j\" (UniqueName: \"kubernetes.io/projected/3a773f3a-800c-4c84-b18f-7a48cdd92692-kube-api-access-m5v7j\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:14.872129 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.872105 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s64j8\" (UniqueName: \"kubernetes.io/projected/0e649944-a09d-44b1-91d2-fda00b30292d-kube-api-access-s64j8\") pod \"node-resolver-r5ppb\" (UID: \"0e649944-a09d-44b1-91d2-fda00b30292d\") " pod="openshift-dns/node-resolver-r5ppb" Oct 30 16:22:14.873995 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.873962 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxrxb\" (UniqueName: \"kubernetes.io/projected/17bb82ba-5262-47f7-abbc-99f1329500fa-kube-api-access-vxrxb\") pod \"node-ca-jlr9c\" (UID: \"17bb82ba-5262-47f7-abbc-99f1329500fa\") " pod="openshift-image-registry/node-ca-jlr9c" Oct 30 16:22:14.875163 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:14.875140 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 16:22:14.875270 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:14.875167 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 16:22:14.875270 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:14.875191 2575 projected.go:196] Error preparing data for projected volume kube-api-access-srfcg for pod openshift-network-diagnostics/network-check-target-426gj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:22:14.875270 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:14.875246 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/78b9342d-5361-4a8c-a771-0c1aea26a48d-kube-api-access-srfcg podName:78b9342d-5361-4a8c-a771-0c1aea26a48d nodeName:}" failed. No retries permitted until 2026-10-30 16:22:15.375229819 +0000 UTC m=+3.103642457 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-srfcg" (UniqueName: "kubernetes.io/projected/78b9342d-5361-4a8c-a771-0c1aea26a48d-kube-api-access-srfcg") pod "network-check-target-426gj" (UID: "78b9342d-5361-4a8c-a771-0c1aea26a48d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:22:14.877331 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.877306 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8vll\" (UniqueName: \"kubernetes.io/projected/b804fc37-0b07-4edf-8720-b31b5698b935-kube-api-access-m8vll\") pod \"network-metrics-daemon-mbqwf\" (UID: \"b804fc37-0b07-4edf-8720-b31b5698b935\") " pod="openshift-multus/network-metrics-daemon-mbqwf" Oct 30 16:22:14.877412 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.877362 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m78c4\" (UniqueName: \"kubernetes.io/projected/8e02b2b2-c05c-4821-adfe-02f2a6664267-kube-api-access-m78c4\") pod \"multus-additional-cni-plugins-5jkb7\" (UID: \"8e02b2b2-c05c-4821-adfe-02f2a6664267\") " pod="openshift-multus/multus-additional-cni-plugins-5jkb7" Oct 30 16:22:14.877927 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.877845 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb9mk\" (UniqueName: \"kubernetes.io/projected/394d816a-1e33-433f-abb1-586089620763-kube-api-access-nb9mk\") pod \"aws-ebs-csi-driver-node-z96s5\" (UID: \"394d816a-1e33-433f-abb1-586089620763\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z96s5" Oct 30 16:22:14.879449 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.879427 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dlwk\" (UniqueName: \"kubernetes.io/projected/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-kube-api-access-6dlwk\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:14.969905 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.969650 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jlr9c" Oct 30 16:22:14.972757 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.972734 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:14.976766 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.976745 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:14.982175 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.982154 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9nfhv" Oct 30 16:22:14.982588 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:14.982561 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17bb82ba_5262_47f7_abbc_99f1329500fa.slice/crio-fb8fe2ee28869222ba653799b2b57a388e05e8a844ed94e8609eda7f5308e961 WatchSource:0}: Error finding container fb8fe2ee28869222ba653799b2b57a388e05e8a844ed94e8609eda7f5308e961: Status 404 returned error can't find the container with id fb8fe2ee28869222ba653799b2b57a388e05e8a844ed94e8609eda7f5308e961 Oct 30 16:22:14.983224 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:14.983164 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3a1c417_2698_4462_b7ff_1d2e66d7a10f.slice/crio-2050322e2b6f53fddf1dd87469e9e1ff6688187934056115715470205362f52d WatchSource:0}: Error finding container 2050322e2b6f53fddf1dd87469e9e1ff6688187934056115715470205362f52d: Status 404 returned error can't find the container with id 2050322e2b6f53fddf1dd87469e9e1ff6688187934056115715470205362f52d Oct 30 16:22:14.989735 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:14.989711 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a773f3a_800c_4c84_b18f_7a48cdd92692.slice/crio-b34b17e821c184a60024ab0bcb8637c9272116ed7fd1b50819ab74e2d25e471a WatchSource:0}: Error finding container b34b17e821c184a60024ab0bcb8637c9272116ed7fd1b50819ab74e2d25e471a: Status 404 returned error can't find the container with id b34b17e821c184a60024ab0bcb8637c9272116ed7fd1b50819ab74e2d25e471a Oct 30 16:22:14.997147 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:14.997120 2575 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 16:22:15.000879 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:15.000859 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe9cbcfe_25a7_4da2_bd0d_cb313273c846.slice/crio-83b1759279a811cd0f4c435481c00f17690c838c2c5b49eb63647a502c3cc295 WatchSource:0}: Error finding container 83b1759279a811cd0f4c435481c00f17690c838c2c5b49eb63647a502c3cc295: Status 404 returned error can't find the container with id 83b1759279a811cd0f4c435481c00f17690c838c2c5b49eb63647a502c3cc295 Oct 30 16:22:15.004583 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:15.004565 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5jkb7" Oct 30 16:22:15.008255 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:15.008059 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-z7jnf" Oct 30 16:22:15.015262 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:15.015111 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z96s5" Oct 30 16:22:15.015417 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:15.015393 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e02b2b2_c05c_4821_adfe_02f2a6664267.slice/crio-b50e9c4749aef7c936214753593f845f779379b915ae058f85a9434ba66e39bc WatchSource:0}: Error finding container b50e9c4749aef7c936214753593f845f779379b915ae058f85a9434ba66e39bc: Status 404 returned error can't find the container with id b50e9c4749aef7c936214753593f845f779379b915ae058f85a9434ba66e39bc Oct 30 16:22:15.017823 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:15.017789 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pg4kq" Oct 30 16:22:15.018119 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:15.018006 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc407642e_2f4a_4c47_906e_56137857d162.slice/crio-8b07c6de4cf29ea898ddc5c111d581be149d4fc1cbc734a4ff91f705c3cc8de0 WatchSource:0}: Error finding container 8b07c6de4cf29ea898ddc5c111d581be149d4fc1cbc734a4ff91f705c3cc8de0: Status 404 returned error can't find the container with id 8b07c6de4cf29ea898ddc5c111d581be149d4fc1cbc734a4ff91f705c3cc8de0 Oct 30 16:22:15.024124 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:15.024106 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-r5ppb" Oct 30 16:22:15.050235 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:15.050205 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod394d816a_1e33_433f_abb1_586089620763.slice/crio-393260634ffef513017fae87c8817de2c1238100f6eacfe5a677b6244e5cc141 WatchSource:0}: Error finding container 393260634ffef513017fae87c8817de2c1238100f6eacfe5a677b6244e5cc141: Status 404 returned error can't find the container with id 393260634ffef513017fae87c8817de2c1238100f6eacfe5a677b6244e5cc141 Oct 30 16:22:15.059457 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:15.059428 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e649944_a09d_44b1_91d2_fda00b30292d.slice/crio-1d9c953d9c05dddbe9ba30df4c18df0bd67104dbf43b27c995246f24c34cf607 WatchSource:0}: Error finding container 1d9c953d9c05dddbe9ba30df4c18df0bd67104dbf43b27c995246f24c34cf607: Status 404 returned error can't find the container with id 1d9c953d9c05dddbe9ba30df4c18df0bd67104dbf43b27c995246f24c34cf607 Oct 30 16:22:15.353615 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:15.353451 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc77c50f-abe0-4d9c-a88b-7a186c17dc3e-original-pull-secret\") pod \"global-pull-secret-syncer-2lrlr\" (UID: \"dc77c50f-abe0-4d9c-a88b-7a186c17dc3e\") " pod="kube-system/global-pull-secret-syncer-2lrlr" Oct 30 16:22:15.353615 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:15.353611 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 16:22:15.353774 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:15.353649 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b804fc37-0b07-4edf-8720-b31b5698b935-metrics-certs\") pod \"network-metrics-daemon-mbqwf\" (UID: \"b804fc37-0b07-4edf-8720-b31b5698b935\") " pod="openshift-multus/network-metrics-daemon-mbqwf" Oct 30 16:22:15.353774 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:15.353669 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc77c50f-abe0-4d9c-a88b-7a186c17dc3e-original-pull-secret podName:dc77c50f-abe0-4d9c-a88b-7a186c17dc3e nodeName:}" failed. No retries permitted until 2026-10-30 16:22:16.353654251 +0000 UTC m=+4.082066885 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dc77c50f-abe0-4d9c-a88b-7a186c17dc3e-original-pull-secret") pod "global-pull-secret-syncer-2lrlr" (UID: "dc77c50f-abe0-4d9c-a88b-7a186c17dc3e") : object "kube-system"/"original-pull-secret" not registered Oct 30 16:22:15.353774 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:15.353727 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:22:15.353902 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:15.353780 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b804fc37-0b07-4edf-8720-b31b5698b935-metrics-certs podName:b804fc37-0b07-4edf-8720-b31b5698b935 nodeName:}" failed. No retries permitted until 2026-10-30 16:22:16.353764793 +0000 UTC m=+4.082177433 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b804fc37-0b07-4edf-8720-b31b5698b935-metrics-certs") pod "network-metrics-daemon-mbqwf" (UID: "b804fc37-0b07-4edf-8720-b31b5698b935") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:22:15.454674 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:15.454628 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srfcg\" (UniqueName: \"kubernetes.io/projected/78b9342d-5361-4a8c-a771-0c1aea26a48d-kube-api-access-srfcg\") pod \"network-check-target-426gj\" (UID: \"78b9342d-5361-4a8c-a771-0c1aea26a48d\") " pod="openshift-network-diagnostics/network-check-target-426gj" Oct 30 16:22:15.454868 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:15.454840 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 16:22:15.454868 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:15.454865 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 16:22:15.454983 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:15.454879 2575 projected.go:196] Error preparing data for projected volume kube-api-access-srfcg for pod openshift-network-diagnostics/network-check-target-426gj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:22:15.454983 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:15.454946 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/78b9342d-5361-4a8c-a771-0c1aea26a48d-kube-api-access-srfcg podName:78b9342d-5361-4a8c-a771-0c1aea26a48d nodeName:}" failed. No retries permitted until 2026-10-30 16:22:16.454926736 +0000 UTC m=+4.183339359 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-srfcg" (UniqueName: "kubernetes.io/projected/78b9342d-5361-4a8c-a771-0c1aea26a48d-kube-api-access-srfcg") pod "network-check-target-426gj" (UID: "78b9342d-5361-4a8c-a771-0c1aea26a48d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:22:15.708543 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:15.708396 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 16:17:15 +0000 UTC" deadline="2028-06-05 03:28:11.461774834 +0000 UTC" Oct 30 16:22:15.708543 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:15.708444 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14003h5m55.75333505s" Oct 30 16:22:15.795772 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:15.795728 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-r5ppb" event={"ID":"0e649944-a09d-44b1-91d2-fda00b30292d","Type":"ContainerStarted","Data":"1d9c953d9c05dddbe9ba30df4c18df0bd67104dbf43b27c995246f24c34cf607"} Oct 30 16:22:15.798166 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:15.798134 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z96s5" event={"ID":"394d816a-1e33-433f-abb1-586089620763","Type":"ContainerStarted","Data":"393260634ffef513017fae87c8817de2c1238100f6eacfe5a677b6244e5cc141"} Oct 30 16:22:15.801021 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:15.800994 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-z7jnf" event={"ID":"c407642e-2f4a-4c47-906e-56137857d162","Type":"ContainerStarted","Data":"8b07c6de4cf29ea898ddc5c111d581be149d4fc1cbc734a4ff91f705c3cc8de0"} Oct 30 16:22:15.807639 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:15.807338 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5jkb7" event={"ID":"8e02b2b2-c05c-4821-adfe-02f2a6664267","Type":"ContainerStarted","Data":"b50e9c4749aef7c936214753593f845f779379b915ae058f85a9434ba66e39bc"} Oct 30 16:22:15.816656 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:15.816607 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9nfhv" event={"ID":"fe9cbcfe-25a7-4da2-bd0d-cb313273c846","Type":"ContainerStarted","Data":"83b1759279a811cd0f4c435481c00f17690c838c2c5b49eb63647a502c3cc295"} Oct 30 16:22:15.823963 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:15.823942 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" event={"ID":"3a773f3a-800c-4c84-b18f-7a48cdd92692","Type":"ContainerStarted","Data":"b34b17e821c184a60024ab0bcb8637c9272116ed7fd1b50819ab74e2d25e471a"} Oct 30 16:22:15.827439 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:15.827323 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" event={"ID":"d3a1c417-2698-4462-b7ff-1d2e66d7a10f","Type":"ContainerStarted","Data":"2050322e2b6f53fddf1dd87469e9e1ff6688187934056115715470205362f52d"} Oct 30 16:22:15.831342 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:15.831228 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jlr9c" event={"ID":"17bb82ba-5262-47f7-abbc-99f1329500fa","Type":"ContainerStarted","Data":"fb8fe2ee28869222ba653799b2b57a388e05e8a844ed94e8609eda7f5308e961"} Oct 30 16:22:15.845953 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:15.845137 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-180.ec2.internal" event={"ID":"9a63059ab80503558edad3ed2b226c76","Type":"ContainerStarted","Data":"8346ea4afe0b1b2aa4c2d60811bdcb6608b13280291facd8246e91acc2a6b065"} Oct 30 16:22:15.872269 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:15.871548 2575 generic.go:356] "Generic (PLEG): container finished" podID="83767d666be2011c3ea225fbea981665" containerID="c111e4d5ad6ad2b86fddac4e7ce3fd1f2debe385a1cf820a91d9a652c905a65f" exitCode=0 Oct 30 16:22:15.872269 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:15.871625 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-180.ec2.internal" event={"ID":"83767d666be2011c3ea225fbea981665","Type":"ContainerDied","Data":"c111e4d5ad6ad2b86fddac4e7ce3fd1f2debe385a1cf820a91d9a652c905a65f"} Oct 30 16:22:15.874541 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:15.874469 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pg4kq" event={"ID":"7c09760d-2160-41d4-b991-f6324ad5aff7","Type":"ContainerStarted","Data":"49a61c059f9969e90d591edc4f2c99a030d089f79e91d92054c08408ca492db3"} Oct 30 16:22:15.916959 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:15.916090 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-180.ec2.internal" podStartSLOduration=1.916073925 podStartE2EDuration="1.916073925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:22:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:22:15.880956093 +0000 UTC m=+3.609368740" watchObservedRunningTime="2026-10-30 16:22:15.916073925 +0000 UTC m=+3.644486573" Oct 30 16:22:16.362083 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:16.361998 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b804fc37-0b07-4edf-8720-b31b5698b935-metrics-certs\") pod \"network-metrics-daemon-mbqwf\" (UID: \"b804fc37-0b07-4edf-8720-b31b5698b935\") " pod="openshift-multus/network-metrics-daemon-mbqwf" Oct 30 16:22:16.362307 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:16.362094 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc77c50f-abe0-4d9c-a88b-7a186c17dc3e-original-pull-secret\") pod \"global-pull-secret-syncer-2lrlr\" (UID: \"dc77c50f-abe0-4d9c-a88b-7a186c17dc3e\") " pod="kube-system/global-pull-secret-syncer-2lrlr" Oct 30 16:22:16.362376 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:16.362338 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 16:22:16.362428 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:16.362407 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc77c50f-abe0-4d9c-a88b-7a186c17dc3e-original-pull-secret podName:dc77c50f-abe0-4d9c-a88b-7a186c17dc3e nodeName:}" failed. No retries permitted until 2026-10-30 16:22:18.362386597 +0000 UTC m=+6.090799235 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dc77c50f-abe0-4d9c-a88b-7a186c17dc3e-original-pull-secret") pod "global-pull-secret-syncer-2lrlr" (UID: "dc77c50f-abe0-4d9c-a88b-7a186c17dc3e") : object "kube-system"/"original-pull-secret" not registered Oct 30 16:22:16.362957 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:16.362937 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:22:16.363048 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:16.363011 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b804fc37-0b07-4edf-8720-b31b5698b935-metrics-certs podName:b804fc37-0b07-4edf-8720-b31b5698b935 nodeName:}" failed. No retries permitted until 2026-10-30 16:22:18.362984064 +0000 UTC m=+6.091396702 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b804fc37-0b07-4edf-8720-b31b5698b935-metrics-certs") pod "network-metrics-daemon-mbqwf" (UID: "b804fc37-0b07-4edf-8720-b31b5698b935") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:22:16.462777 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:16.462740 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srfcg\" (UniqueName: \"kubernetes.io/projected/78b9342d-5361-4a8c-a771-0c1aea26a48d-kube-api-access-srfcg\") pod \"network-check-target-426gj\" (UID: \"78b9342d-5361-4a8c-a771-0c1aea26a48d\") " pod="openshift-network-diagnostics/network-check-target-426gj" Oct 30 16:22:16.462986 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:16.462966 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 16:22:16.463052 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:16.462987 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 16:22:16.463052 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:16.463007 2575 projected.go:196] Error preparing data for projected volume kube-api-access-srfcg for pod openshift-network-diagnostics/network-check-target-426gj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:22:16.463148 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:16.463070 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/78b9342d-5361-4a8c-a771-0c1aea26a48d-kube-api-access-srfcg podName:78b9342d-5361-4a8c-a771-0c1aea26a48d nodeName:}" failed. No retries permitted until 2026-10-30 16:22:18.463050558 +0000 UTC m=+6.191463200 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-srfcg" (UniqueName: "kubernetes.io/projected/78b9342d-5361-4a8c-a771-0c1aea26a48d-kube-api-access-srfcg") pod "network-check-target-426gj" (UID: "78b9342d-5361-4a8c-a771-0c1aea26a48d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:22:16.709124 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:16.708769 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 16:17:15 +0000 UTC" deadline="2028-08-14 00:28:45.149102074 +0000 UTC" Oct 30 16:22:16.709124 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:16.708846 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15680h6m28.440260846s" Oct 30 16:22:16.771777 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:16.771745 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2lrlr" Oct 30 16:22:16.771963 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:16.771895 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2lrlr" podUID="dc77c50f-abe0-4d9c-a88b-7a186c17dc3e" Oct 30 16:22:16.772329 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:16.772306 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-426gj" Oct 30 16:22:16.772434 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:16.772402 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-426gj" podUID="78b9342d-5361-4a8c-a771-0c1aea26a48d" Oct 30 16:22:16.772495 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:16.772472 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mbqwf" Oct 30 16:22:16.772600 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:16.772561 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mbqwf" podUID="b804fc37-0b07-4edf-8720-b31b5698b935" Oct 30 16:22:16.896610 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:16.896535 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-180.ec2.internal" event={"ID":"83767d666be2011c3ea225fbea981665","Type":"ContainerStarted","Data":"15d245eb69d354b0e9e38c68511ac3330174e6cd1f728b0bcc75dbfb92915219"} Oct 30 16:22:18.383455 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:18.383413 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc77c50f-abe0-4d9c-a88b-7a186c17dc3e-original-pull-secret\") pod \"global-pull-secret-syncer-2lrlr\" (UID: \"dc77c50f-abe0-4d9c-a88b-7a186c17dc3e\") " pod="kube-system/global-pull-secret-syncer-2lrlr" Oct 30 16:22:18.383978 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:18.383484 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b804fc37-0b07-4edf-8720-b31b5698b935-metrics-certs\") pod \"network-metrics-daemon-mbqwf\" (UID: \"b804fc37-0b07-4edf-8720-b31b5698b935\") " pod="openshift-multus/network-metrics-daemon-mbqwf" Oct 30 16:22:18.383978 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:18.383641 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:22:18.383978 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:18.383700 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b804fc37-0b07-4edf-8720-b31b5698b935-metrics-certs podName:b804fc37-0b07-4edf-8720-b31b5698b935 nodeName:}" failed. No retries permitted until 2026-10-30 16:22:22.383682018 +0000 UTC m=+10.112094640 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b804fc37-0b07-4edf-8720-b31b5698b935-metrics-certs") pod "network-metrics-daemon-mbqwf" (UID: "b804fc37-0b07-4edf-8720-b31b5698b935") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:22:18.384164 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:18.384119 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 16:22:18.384220 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:18.384171 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc77c50f-abe0-4d9c-a88b-7a186c17dc3e-original-pull-secret podName:dc77c50f-abe0-4d9c-a88b-7a186c17dc3e nodeName:}" failed. No retries permitted until 2026-10-30 16:22:22.384154982 +0000 UTC m=+10.112567621 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dc77c50f-abe0-4d9c-a88b-7a186c17dc3e-original-pull-secret") pod "global-pull-secret-syncer-2lrlr" (UID: "dc77c50f-abe0-4d9c-a88b-7a186c17dc3e") : object "kube-system"/"original-pull-secret" not registered Oct 30 16:22:18.484659 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:18.484619 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srfcg\" (UniqueName: \"kubernetes.io/projected/78b9342d-5361-4a8c-a771-0c1aea26a48d-kube-api-access-srfcg\") pod \"network-check-target-426gj\" (UID: \"78b9342d-5361-4a8c-a771-0c1aea26a48d\") " pod="openshift-network-diagnostics/network-check-target-426gj" Oct 30 16:22:18.484880 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:18.484806 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 16:22:18.484880 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:18.484845 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 16:22:18.484880 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:18.484865 2575 projected.go:196] Error preparing data for projected volume kube-api-access-srfcg for pod openshift-network-diagnostics/network-check-target-426gj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:22:18.485053 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:18.484922 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/78b9342d-5361-4a8c-a771-0c1aea26a48d-kube-api-access-srfcg podName:78b9342d-5361-4a8c-a771-0c1aea26a48d nodeName:}" failed. No retries permitted until 2026-10-30 16:22:22.484903137 +0000 UTC m=+10.213315773 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-srfcg" (UniqueName: "kubernetes.io/projected/78b9342d-5361-4a8c-a771-0c1aea26a48d-kube-api-access-srfcg") pod "network-check-target-426gj" (UID: "78b9342d-5361-4a8c-a771-0c1aea26a48d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:22:18.766015 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:18.765955 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mbqwf" Oct 30 16:22:18.766197 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:18.766102 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mbqwf" podUID="b804fc37-0b07-4edf-8720-b31b5698b935" Oct 30 16:22:18.766197 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:18.766124 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-426gj" Oct 30 16:22:18.766342 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:18.766242 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-426gj" podUID="78b9342d-5361-4a8c-a771-0c1aea26a48d" Oct 30 16:22:18.766342 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:18.765956 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2lrlr" Oct 30 16:22:18.766445 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:18.766349 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2lrlr" podUID="dc77c50f-abe0-4d9c-a88b-7a186c17dc3e" Oct 30 16:22:18.917322 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:18.916380 2575 generic.go:356] "Generic (PLEG): container finished" podID="8e02b2b2-c05c-4821-adfe-02f2a6664267" containerID="ff2ec17cd0c968ad5cdba34fc178bae568e7be24fa56b6a00692195a7b7eaefe" exitCode=0 Oct 30 16:22:18.917322 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:18.916434 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5jkb7" event={"ID":"8e02b2b2-c05c-4821-adfe-02f2a6664267","Type":"ContainerDied","Data":"ff2ec17cd0c968ad5cdba34fc178bae568e7be24fa56b6a00692195a7b7eaefe"} Oct 30 16:22:18.978625 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:18.978116 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-180.ec2.internal" podStartSLOduration=4.97809603 podStartE2EDuration="4.97809603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:22:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:22:16.938160365 +0000 UTC m=+4.666573010" watchObservedRunningTime="2026-10-30 16:22:18.97809603 +0000 UTC m=+6.706508677" Oct 30 16:22:19.924581 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:19.924258 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pg4kq" event={"ID":"7c09760d-2160-41d4-b991-f6324ad5aff7","Type":"ContainerStarted","Data":"0a204459b9c6502baeee5876dc70f018dad6c0a8d905ada1471944bece22c31b"} Oct 30 16:22:19.957396 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:19.957337 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-pg4kq" podStartSLOduration=3.040558786 podStartE2EDuration="6.957320271s" totalImagesPullingTime="3.916761485s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:22:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:22:19.957069234 +0000 UTC m=+7.685481877" watchObservedRunningTime="2026-10-30 16:22:19.957320271 +0000 UTC m=+7.685732912" Oct 30 16:22:20.767542 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:20.767511 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mbqwf" Oct 30 16:22:20.767702 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:20.767650 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mbqwf" podUID="b804fc37-0b07-4edf-8720-b31b5698b935" Oct 30 16:22:20.768061 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:20.768040 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2lrlr" Oct 30 16:22:20.768163 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:20.768136 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2lrlr" podUID="dc77c50f-abe0-4d9c-a88b-7a186c17dc3e" Oct 30 16:22:20.768225 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:20.768196 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-426gj" Oct 30 16:22:20.768272 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:20.768260 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-426gj" podUID="78b9342d-5361-4a8c-a771-0c1aea26a48d" Oct 30 16:22:20.930559 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:20.930526 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z96s5" event={"ID":"394d816a-1e33-433f-abb1-586089620763","Type":"ContainerStarted","Data":"7de1c41b332e0d9378cb9f44dd2fd339f836bba1eecae0b968d33687b2802df3"} Oct 30 16:22:20.935175 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:20.935021 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jlr9c" event={"ID":"17bb82ba-5262-47f7-abbc-99f1329500fa","Type":"ContainerStarted","Data":"c17425d6103d9762815bcf71979e5e1e21f7103451189d7490a5a73501012131"} Oct 30 16:22:20.968221 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:20.967276 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-jlr9c" podStartSLOduration=2.96913546 podStartE2EDuration="7.967256599s" totalImagesPullingTime="4.998121139s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:22:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:22:20.967112818 +0000 UTC m=+8.695525458" watchObservedRunningTime="2026-10-30 16:22:20.967256599 +0000 UTC m=+8.695669242" Oct 30 16:22:22.418396 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:22.418330 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b804fc37-0b07-4edf-8720-b31b5698b935-metrics-certs\") pod \"network-metrics-daemon-mbqwf\" (UID: \"b804fc37-0b07-4edf-8720-b31b5698b935\") " pod="openshift-multus/network-metrics-daemon-mbqwf" Oct 30 16:22:22.418916 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:22.418421 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc77c50f-abe0-4d9c-a88b-7a186c17dc3e-original-pull-secret\") pod \"global-pull-secret-syncer-2lrlr\" (UID: \"dc77c50f-abe0-4d9c-a88b-7a186c17dc3e\") " pod="kube-system/global-pull-secret-syncer-2lrlr" Oct 30 16:22:22.418916 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:22.418546 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 16:22:22.418916 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:22.418606 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc77c50f-abe0-4d9c-a88b-7a186c17dc3e-original-pull-secret podName:dc77c50f-abe0-4d9c-a88b-7a186c17dc3e nodeName:}" failed. No retries permitted until 2026-10-30 16:22:30.418587271 +0000 UTC m=+18.146999910 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dc77c50f-abe0-4d9c-a88b-7a186c17dc3e-original-pull-secret") pod "global-pull-secret-syncer-2lrlr" (UID: "dc77c50f-abe0-4d9c-a88b-7a186c17dc3e") : object "kube-system"/"original-pull-secret" not registered Oct 30 16:22:22.418916 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:22.418671 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:22:22.418916 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:22.418704 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b804fc37-0b07-4edf-8720-b31b5698b935-metrics-certs podName:b804fc37-0b07-4edf-8720-b31b5698b935 nodeName:}" failed. No retries permitted until 2026-10-30 16:22:30.418692198 +0000 UTC m=+18.147104831 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b804fc37-0b07-4edf-8720-b31b5698b935-metrics-certs") pod "network-metrics-daemon-mbqwf" (UID: "b804fc37-0b07-4edf-8720-b31b5698b935") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:22:22.519217 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:22.519185 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srfcg\" (UniqueName: \"kubernetes.io/projected/78b9342d-5361-4a8c-a771-0c1aea26a48d-kube-api-access-srfcg\") pod \"network-check-target-426gj\" (UID: \"78b9342d-5361-4a8c-a771-0c1aea26a48d\") " pod="openshift-network-diagnostics/network-check-target-426gj" Oct 30 16:22:22.519405 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:22.519383 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 16:22:22.519405 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:22.519410 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 16:22:22.519571 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:22.519423 2575 projected.go:196] Error preparing data for projected volume kube-api-access-srfcg for pod openshift-network-diagnostics/network-check-target-426gj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:22:22.519571 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:22.519478 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/78b9342d-5361-4a8c-a771-0c1aea26a48d-kube-api-access-srfcg podName:78b9342d-5361-4a8c-a771-0c1aea26a48d nodeName:}" failed. No retries permitted until 2026-10-30 16:22:30.519464527 +0000 UTC m=+18.247877163 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-srfcg" (UniqueName: "kubernetes.io/projected/78b9342d-5361-4a8c-a771-0c1aea26a48d-kube-api-access-srfcg") pod "network-check-target-426gj" (UID: "78b9342d-5361-4a8c-a771-0c1aea26a48d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:22:22.589084 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:22.589062 2575 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 16:22:22.701992 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:22.701896 2575 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T16:22:22.589081583Z","UUID":"3d68a172-50d4-4ad5-b8f4-e293b0204e2e","Handler":null,"Name":"","Endpoint":""} Oct 30 16:22:22.703582 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:22.703560 2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 16:22:22.703690 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:22.703591 2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 16:22:22.767388 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:22.767355 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mbqwf" Oct 30 16:22:22.767550 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:22.767476 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mbqwf" podUID="b804fc37-0b07-4edf-8720-b31b5698b935" Oct 30 16:22:22.767835 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:22.767813 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2lrlr" Oct 30 16:22:22.767934 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:22.767913 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2lrlr" podUID="dc77c50f-abe0-4d9c-a88b-7a186c17dc3e" Oct 30 16:22:22.768011 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:22.767976 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-426gj" Oct 30 16:22:22.768066 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:22.768022 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-426gj" podUID="78b9342d-5361-4a8c-a771-0c1aea26a48d" Oct 30 16:22:22.942206 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:22.942178 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9nfhv" event={"ID":"fe9cbcfe-25a7-4da2-bd0d-cb313273c846","Type":"ContainerStarted","Data":"71f53acc5b5c3c5a1937e128942660cda430a0ce04dff5dd2dc86246b6e7981e"} Oct 30 16:22:22.945764 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:22.945734 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z96s5" event={"ID":"394d816a-1e33-433f-abb1-586089620763","Type":"ContainerStarted","Data":"0abcfe83b924a513b99af6f5754c83ca17346a7d74093e51a3a3ecf5f4c66bf0"} Oct 30 16:22:23.949976 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:23.949686 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-r5ppb" event={"ID":"0e649944-a09d-44b1-91d2-fda00b30292d","Type":"ContainerStarted","Data":"4a92653b63a40c16b3a678a4ccf85016911c926ca8e8c2b169712ae0692bed7f"} Oct 30 16:22:23.952529 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:23.952500 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z96s5" event={"ID":"394d816a-1e33-433f-abb1-586089620763","Type":"ContainerStarted","Data":"c911fcceab0aac978fc24177ec6277be876cee875de66b2ae5ab9754f28f4e88"} Oct 30 16:22:23.956258 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:23.956236 2575 generic.go:356] "Generic (PLEG): container finished" podID="8e02b2b2-c05c-4821-adfe-02f2a6664267" containerID="cf420fd2f91aafb332053258ecd786aa6c9b9382434793ec68f67020fe439df6" exitCode=0 Oct 30 16:22:23.956365 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:23.956318 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5jkb7" event={"ID":"8e02b2b2-c05c-4821-adfe-02f2a6664267","Type":"ContainerDied","Data":"cf420fd2f91aafb332053258ecd786aa6c9b9382434793ec68f67020fe439df6"} Oct 30 16:22:23.968534 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:23.968508 2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-pg4kq" Oct 30 16:22:23.969232 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:23.969197 2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-pg4kq" Oct 30 16:22:23.982337 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:23.982297 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-9nfhv" podStartSLOduration=3.385544097 podStartE2EDuration="10.982283495s" totalImagesPullingTime="7.596739398s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:22:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:22:22.981116873 +0000 UTC m=+10.709529513" watchObservedRunningTime="2026-10-30 16:22:23.982283495 +0000 UTC m=+11.710696137" Oct 30 16:22:23.982762 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:23.982734 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-r5ppb" podStartSLOduration=2.890232947 podStartE2EDuration="10.982723026s" totalImagesPullingTime="8.092490079s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:22:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:22:23.981940435 +0000 UTC m=+11.710353076" watchObservedRunningTime="2026-10-30 16:22:23.982723026 +0000 UTC m=+11.711135666" Oct 30 16:22:24.053596 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:24.053141 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z96s5" podStartSLOduration=2.890443259 podStartE2EDuration="11.053122476s" totalImagesPullingTime="8.162679217s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:22:13 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:22:24.052866036 +0000 UTC m=+11.781278676" watchObservedRunningTime="2026-10-30 16:22:24.053122476 +0000 UTC m=+11.781535118" Oct 30 16:22:24.766013 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:24.765915 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mbqwf" Oct 30 16:22:24.766216 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:24.766086 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mbqwf" podUID="b804fc37-0b07-4edf-8720-b31b5698b935" Oct 30 16:22:24.766446 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:24.766414 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2lrlr" Oct 30 16:22:24.766553 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:24.766505 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2lrlr" podUID="dc77c50f-abe0-4d9c-a88b-7a186c17dc3e" Oct 30 16:22:24.766654 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:24.766640 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-426gj" Oct 30 16:22:24.766716 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:24.766706 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-426gj" podUID="78b9342d-5361-4a8c-a771-0c1aea26a48d" Oct 30 16:22:24.960196 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:24.960156 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-z7jnf" event={"ID":"c407642e-2f4a-4c47-906e-56137857d162","Type":"ContainerStarted","Data":"37c6155e186db86fc9135b06cc1aa3fd52b9cdfb0969a07a51d85505696b86e3"} Oct 30 16:22:24.963020 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:24.962988 2575 generic.go:356] "Generic (PLEG): container finished" podID="8e02b2b2-c05c-4821-adfe-02f2a6664267" containerID="82795c0397cc845c188df9b4cc15eba9edb9c015377158c20145ff2c7115d27e" exitCode=0 Oct 30 16:22:24.963150 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:24.963101 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5jkb7" event={"ID":"8e02b2b2-c05c-4821-adfe-02f2a6664267","Type":"ContainerDied","Data":"82795c0397cc845c188df9b4cc15eba9edb9c015377158c20145ff2c7115d27e"} Oct 30 16:22:25.028211 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:25.028129 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-z7jnf" podStartSLOduration=4.143684754 podStartE2EDuration="12.028111362s" totalImagesPullingTime="7.884426608s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:22:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:22:24.992043587 +0000 UTC m=+12.720456220" watchObservedRunningTime="2026-10-30 16:22:25.028111362 +0000 UTC m=+12.756524005" Oct 30 16:22:25.967943 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:25.967898 2575 generic.go:356] "Generic (PLEG): container finished" podID="8e02b2b2-c05c-4821-adfe-02f2a6664267" containerID="9503d2db5a1004a8aabb3765560bd4ff03029165a8fab7dd1c18ec09c986bda2" exitCode=0 Oct 30 16:22:25.968626 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:25.967993 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5jkb7" event={"ID":"8e02b2b2-c05c-4821-adfe-02f2a6664267","Type":"ContainerDied","Data":"9503d2db5a1004a8aabb3765560bd4ff03029165a8fab7dd1c18ec09c986bda2"} Oct 30 16:22:25.969673 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:25.969636 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" event={"ID":"3a773f3a-800c-4c84-b18f-7a48cdd92692","Type":"ContainerStarted","Data":"5c3c2bbb26427b62a1845f8e0561959f8e17fa8b3810b72ce6f12b54c605a205"} Oct 30 16:22:26.050617 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:26.050568 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" podStartSLOduration=2.938223255 podStartE2EDuration="13.050554427s" totalImagesPullingTime="10.112331172s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:22:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:22:26.050394073 +0000 UTC m=+13.778806712" watchObservedRunningTime="2026-10-30 16:22:26.050554427 +0000 UTC m=+13.778967067" Oct 30 16:22:26.765411 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:26.764982 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mbqwf" Oct 30 16:22:26.765411 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:26.764998 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-426gj" Oct 30 16:22:26.765411 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:26.765105 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mbqwf" podUID="b804fc37-0b07-4edf-8720-b31b5698b935" Oct 30 16:22:26.765411 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:26.765185 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-426gj" podUID="78b9342d-5361-4a8c-a771-0c1aea26a48d" Oct 30 16:22:26.765411 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:26.765249 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2lrlr" Oct 30 16:22:26.765411 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:26.765299 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2lrlr" podUID="dc77c50f-abe0-4d9c-a88b-7a186c17dc3e" Oct 30 16:22:27.978263 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:27.978013 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5jkb7" event={"ID":"8e02b2b2-c05c-4821-adfe-02f2a6664267","Type":"ContainerStarted","Data":"4ab11267cb789025c62198ca6a265c3534cf1701c3298f9837fd7dbaad8c8cf6"} Oct 30 16:22:28.765330 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:28.765294 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2lrlr" Oct 30 16:22:28.765496 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:28.765310 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mbqwf" Oct 30 16:22:28.765496 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:28.765413 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2lrlr" podUID="dc77c50f-abe0-4d9c-a88b-7a186c17dc3e" Oct 30 16:22:28.765496 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:28.765430 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-426gj" Oct 30 16:22:28.765670 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:28.765555 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mbqwf" podUID="b804fc37-0b07-4edf-8720-b31b5698b935" Oct 30 16:22:28.765670 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:28.765646 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-426gj" podUID="78b9342d-5361-4a8c-a771-0c1aea26a48d" Oct 30 16:22:28.983069 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:28.983030 2575 generic.go:356] "Generic (PLEG): container finished" podID="8e02b2b2-c05c-4821-adfe-02f2a6664267" containerID="4ab11267cb789025c62198ca6a265c3534cf1701c3298f9837fd7dbaad8c8cf6" exitCode=0 Oct 30 16:22:28.983530 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:28.983092 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5jkb7" event={"ID":"8e02b2b2-c05c-4821-adfe-02f2a6664267","Type":"ContainerDied","Data":"4ab11267cb789025c62198ca6a265c3534cf1701c3298f9837fd7dbaad8c8cf6"} Oct 30 16:22:29.987945 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:29.987695 2575 generic.go:356] "Generic (PLEG): container finished" podID="8e02b2b2-c05c-4821-adfe-02f2a6664267" containerID="221492ecc01a9080632eaa800c0b5ffa8b5cac3e45879b2e5a9c8d7a59667968" exitCode=0 Oct 30 16:22:29.987945 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:29.987800 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5jkb7" event={"ID":"8e02b2b2-c05c-4821-adfe-02f2a6664267","Type":"ContainerDied","Data":"221492ecc01a9080632eaa800c0b5ffa8b5cac3e45879b2e5a9c8d7a59667968"} Oct 30 16:22:29.991006 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:29.990986 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-56v9s_d3a1c417-2698-4462-b7ff-1d2e66d7a10f/ovn-acl-logging/0.log" Oct 30 16:22:29.991365 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:29.991329 2575 generic.go:356] "Generic (PLEG): container finished" podID="d3a1c417-2698-4462-b7ff-1d2e66d7a10f" containerID="c3dc910d7bf7e2b1a74e7fe9a25b6f7fcc4147625b9d41326f740453e5382364" exitCode=1 Oct 30 16:22:29.991365 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:29.991360 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" event={"ID":"d3a1c417-2698-4462-b7ff-1d2e66d7a10f","Type":"ContainerStarted","Data":"7d0e49e1f57757b2a2f4af0dbff8665cfbbb6fc6cf073d240ff07692b9f50a3f"} Oct 30 16:22:29.991466 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:29.991402 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" event={"ID":"d3a1c417-2698-4462-b7ff-1d2e66d7a10f","Type":"ContainerStarted","Data":"28b35d8171e698eba46d8b5a2434aaf418e0536887c361522403f8b5f54a90c3"} Oct 30 16:22:29.991466 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:29.991454 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" event={"ID":"d3a1c417-2698-4462-b7ff-1d2e66d7a10f","Type":"ContainerStarted","Data":"4f06f90a6a1f343ffa66b44bd8ec0a94a1fe5556a4eb7eee2f5d3855a3e748dc"} Oct 30 16:22:29.991564 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:29.991481 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" event={"ID":"d3a1c417-2698-4462-b7ff-1d2e66d7a10f","Type":"ContainerStarted","Data":"e68b67caa6a496d4a3ab1aa44507548388318915d5d8a3c915353ac9a56492f3"} Oct 30 16:22:29.991564 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:29.991509 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" event={"ID":"d3a1c417-2698-4462-b7ff-1d2e66d7a10f","Type":"ContainerDied","Data":"c3dc910d7bf7e2b1a74e7fe9a25b6f7fcc4147625b9d41326f740453e5382364"} Oct 30 16:22:29.991564 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:29.991536 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" event={"ID":"d3a1c417-2698-4462-b7ff-1d2e66d7a10f","Type":"ContainerStarted","Data":"e6d0b7cb4d2a284dd627ff426d8c136eec766794f245f9e2f35dc35946d94a03"} Oct 30 16:22:30.476943 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:30.476919 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc77c50f-abe0-4d9c-a88b-7a186c17dc3e-original-pull-secret\") pod \"global-pull-secret-syncer-2lrlr\" (UID: \"dc77c50f-abe0-4d9c-a88b-7a186c17dc3e\") " pod="kube-system/global-pull-secret-syncer-2lrlr" Oct 30 16:22:30.477096 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:30.476954 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b804fc37-0b07-4edf-8720-b31b5698b935-metrics-certs\") pod \"network-metrics-daemon-mbqwf\" (UID: \"b804fc37-0b07-4edf-8720-b31b5698b935\") " pod="openshift-multus/network-metrics-daemon-mbqwf" Oct 30 16:22:30.477096 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:30.477059 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:22:30.477096 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:30.477064 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 16:22:30.477188 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:30.477105 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b804fc37-0b07-4edf-8720-b31b5698b935-metrics-certs podName:b804fc37-0b07-4edf-8720-b31b5698b935 nodeName:}" failed. No retries permitted until 2026-10-30 16:22:46.477092721 +0000 UTC m=+34.205505352 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b804fc37-0b07-4edf-8720-b31b5698b935-metrics-certs") pod "network-metrics-daemon-mbqwf" (UID: "b804fc37-0b07-4edf-8720-b31b5698b935") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:22:30.477188 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:30.477117 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc77c50f-abe0-4d9c-a88b-7a186c17dc3e-original-pull-secret podName:dc77c50f-abe0-4d9c-a88b-7a186c17dc3e nodeName:}" failed. No retries permitted until 2026-10-30 16:22:46.47711112 +0000 UTC m=+34.205523738 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dc77c50f-abe0-4d9c-a88b-7a186c17dc3e-original-pull-secret") pod "global-pull-secret-syncer-2lrlr" (UID: "dc77c50f-abe0-4d9c-a88b-7a186c17dc3e") : object "kube-system"/"original-pull-secret" not registered Oct 30 16:22:30.577978 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:30.577942 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srfcg\" (UniqueName: \"kubernetes.io/projected/78b9342d-5361-4a8c-a771-0c1aea26a48d-kube-api-access-srfcg\") pod \"network-check-target-426gj\" (UID: \"78b9342d-5361-4a8c-a771-0c1aea26a48d\") " pod="openshift-network-diagnostics/network-check-target-426gj" Oct 30 16:22:30.578139 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:30.578092 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 16:22:30.578139 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:30.578110 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 16:22:30.578139 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:30.578119 2575 projected.go:196] Error preparing data for projected volume kube-api-access-srfcg for pod openshift-network-diagnostics/network-check-target-426gj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:22:30.578220 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:30.578164 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/78b9342d-5361-4a8c-a771-0c1aea26a48d-kube-api-access-srfcg podName:78b9342d-5361-4a8c-a771-0c1aea26a48d nodeName:}" failed. No retries permitted until 2026-10-30 16:22:46.578153307 +0000 UTC m=+34.306565944 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-srfcg" (UniqueName: "kubernetes.io/projected/78b9342d-5361-4a8c-a771-0c1aea26a48d-kube-api-access-srfcg") pod "network-check-target-426gj" (UID: "78b9342d-5361-4a8c-a771-0c1aea26a48d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:22:30.765199 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:30.765127 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mbqwf" Oct 30 16:22:30.765199 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:30.765167 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-426gj" Oct 30 16:22:30.765365 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:30.765130 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2lrlr" Oct 30 16:22:30.765365 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:30.765261 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mbqwf" podUID="b804fc37-0b07-4edf-8720-b31b5698b935" Oct 30 16:22:30.765365 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:30.765324 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-426gj" podUID="78b9342d-5361-4a8c-a771-0c1aea26a48d" Oct 30 16:22:30.765454 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:30.765436 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2lrlr" podUID="dc77c50f-abe0-4d9c-a88b-7a186c17dc3e" Oct 30 16:22:30.996329 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:30.996287 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5jkb7" event={"ID":"8e02b2b2-c05c-4821-adfe-02f2a6664267","Type":"ContainerStarted","Data":"19894bb1a6f9b63ce29b5df68bf5bbe4615c9b9fd65b0d5b9204c16d1e07a604"} Oct 30 16:22:31.033216 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:31.033138 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-5jkb7" podStartSLOduration=7.771447416 podStartE2EDuration="18.033126241s" totalImagesPullingTime="10.261678825s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:22:13 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:22:31.032540336 +0000 UTC m=+18.760952976" watchObservedRunningTime="2026-10-30 16:22:31.033126241 +0000 UTC m=+18.761538881" Oct 30 16:22:32.001098 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:32.001071 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-56v9s_d3a1c417-2698-4462-b7ff-1d2e66d7a10f/ovn-acl-logging/0.log" Oct 30 16:22:32.001529 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:32.001459 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" event={"ID":"d3a1c417-2698-4462-b7ff-1d2e66d7a10f","Type":"ContainerStarted","Data":"9363f7240f80c0be9d74371b8d4d4199dc47496abd386afd6ecd8849f7585556"} Oct 30 16:22:32.766129 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:32.766094 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mbqwf" Oct 30 16:22:32.766302 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:32.766209 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2lrlr" Oct 30 16:22:32.766302 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:32.766225 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mbqwf" podUID="b804fc37-0b07-4edf-8720-b31b5698b935" Oct 30 16:22:32.766302 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:32.766262 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2lrlr" podUID="dc77c50f-abe0-4d9c-a88b-7a186c17dc3e" Oct 30 16:22:32.766447 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:32.766320 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-426gj" Oct 30 16:22:32.766447 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:32.766390 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-426gj" podUID="78b9342d-5361-4a8c-a771-0c1aea26a48d" Oct 30 16:22:34.009445 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:34.009418 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-56v9s_d3a1c417-2698-4462-b7ff-1d2e66d7a10f/ovn-acl-logging/0.log" Oct 30 16:22:34.010298 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:34.009768 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" event={"ID":"d3a1c417-2698-4462-b7ff-1d2e66d7a10f","Type":"ContainerStarted","Data":"090b50c7ce14739bbad8f290aaa7601e244aad90cb2a80c4c02c65092e81b746"} Oct 30 16:22:34.010298 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:34.010028 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:34.010298 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:34.010102 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:34.010298 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:34.010193 2575 scope.go:122] "RemoveContainer" containerID="c3dc910d7bf7e2b1a74e7fe9a25b6f7fcc4147625b9d41326f740453e5382364" Oct 30 16:22:34.029382 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:34.029275 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:34.030948 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:34.030926 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:34.766042 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:34.765835 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2lrlr" Oct 30 16:22:34.766213 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:34.765851 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-426gj" Oct 30 16:22:34.766213 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:34.766118 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2lrlr" podUID="dc77c50f-abe0-4d9c-a88b-7a186c17dc3e" Oct 30 16:22:34.766282 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:34.765866 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mbqwf" Oct 30 16:22:34.766351 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:34.766247 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-426gj" podUID="78b9342d-5361-4a8c-a771-0c1aea26a48d" Oct 30 16:22:34.766439 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:34.766417 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mbqwf" podUID="b804fc37-0b07-4edf-8720-b31b5698b935" Oct 30 16:22:35.015219 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:35.015192 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-56v9s_d3a1c417-2698-4462-b7ff-1d2e66d7a10f/ovn-acl-logging/0.log" Oct 30 16:22:35.015685 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:35.015519 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" event={"ID":"d3a1c417-2698-4462-b7ff-1d2e66d7a10f","Type":"ContainerStarted","Data":"15603f4a67babc3d478da9da5daee89f9a1ff3b55efe6faaf6cbff2eb95622de"} Oct 30 16:22:35.015685 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:35.015619 2575 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 16:22:35.088913 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:35.088801 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" podStartSLOduration=7.613644107 podStartE2EDuration="22.08876946s" totalImagesPullingTime="14.475125353s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:22:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:22:35.088275732 +0000 UTC m=+22.816688372" watchObservedRunningTime="2026-10-30 16:22:35.08876946 +0000 UTC m=+22.817182121" Oct 30 16:22:36.017403 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:36.017342 2575 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 16:22:36.075566 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:36.075534 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-426gj"] Oct 30 16:22:36.075733 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:36.075647 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-426gj" Oct 30 16:22:36.075837 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:36.075750 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-426gj" podUID="78b9342d-5361-4a8c-a771-0c1aea26a48d" Oct 30 16:22:36.078433 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:36.078409 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2lrlr"] Oct 30 16:22:36.078561 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:36.078496 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2lrlr" Oct 30 16:22:36.078629 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:36.078610 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2lrlr" podUID="dc77c50f-abe0-4d9c-a88b-7a186c17dc3e" Oct 30 16:22:36.081331 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:36.081309 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mbqwf"] Oct 30 16:22:36.081412 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:36.081382 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mbqwf" Oct 30 16:22:36.081469 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:36.081455 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mbqwf" podUID="b804fc37-0b07-4edf-8720-b31b5698b935" Oct 30 16:22:37.765103 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:37.765064 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mbqwf" Oct 30 16:22:37.765581 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:37.765082 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2lrlr" Oct 30 16:22:37.765581 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:37.765192 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mbqwf" podUID="b804fc37-0b07-4edf-8720-b31b5698b935" Oct 30 16:22:37.765581 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:37.765236 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-426gj" Oct 30 16:22:37.765581 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:37.765307 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2lrlr" podUID="dc77c50f-abe0-4d9c-a88b-7a186c17dc3e" Oct 30 16:22:37.765581 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:37.765369 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-426gj" podUID="78b9342d-5361-4a8c-a771-0c1aea26a48d" Oct 30 16:22:39.105013 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:39.104975 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-pg4kq" Oct 30 16:22:39.105575 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:39.105136 2575 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 16:22:39.105575 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:39.105520 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-pg4kq" Oct 30 16:22:39.765571 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:39.765541 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mbqwf" Oct 30 16:22:39.765571 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:39.765569 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-426gj" Oct 30 16:22:39.765770 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:39.765645 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mbqwf" podUID="b804fc37-0b07-4edf-8720-b31b5698b935" Oct 30 16:22:39.765770 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:39.765697 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2lrlr" Oct 30 16:22:39.765770 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:39.765763 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2lrlr" podUID="dc77c50f-abe0-4d9c-a88b-7a186c17dc3e" Oct 30 16:22:39.765924 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:39.765843 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-426gj" podUID="78b9342d-5361-4a8c-a771-0c1aea26a48d" Oct 30 16:22:41.765160 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:41.765116 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mbqwf" Oct 30 16:22:41.765541 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:41.765180 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-426gj" Oct 30 16:22:41.765541 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:41.765145 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2lrlr" Oct 30 16:22:41.765541 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:41.765248 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mbqwf" podUID="b804fc37-0b07-4edf-8720-b31b5698b935" Oct 30 16:22:41.765541 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:41.765298 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2lrlr" podUID="dc77c50f-abe0-4d9c-a88b-7a186c17dc3e" Oct 30 16:22:41.765541 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:41.765368 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-426gj" podUID="78b9342d-5361-4a8c-a771-0c1aea26a48d" Oct 30 16:22:41.962009 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:41.961977 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:41.962204 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:41.962192 2575 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 16:22:41.974372 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:41.974347 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:42.101804 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:42.101527 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-180.ec2.internal" event="NodeReady" Oct 30 16:22:42.102010 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:42.101994 2575 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 16:22:42.178916 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:42.178891 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-hlfhx"] Oct 30 16:22:42.181981 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:42.181959 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hlfhx" Oct 30 16:22:42.182828 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:42.182806 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-jvqzq"] Oct 30 16:22:42.185736 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:42.185719 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jvqzq" Oct 30 16:22:42.194330 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:42.194308 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 16:22:42.194582 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:42.194560 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 16:22:42.194685 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:42.194612 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 16:22:42.194685 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:42.194612 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-z9zpr\"" Oct 30 16:22:42.204446 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:42.204426 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hlfhx"] Oct 30 16:22:42.208537 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:42.208520 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jvqzq"] Oct 30 16:22:42.262449 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:42.262425 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fdbc\" (UniqueName: \"kubernetes.io/projected/8a7b4cd2-b289-42b0-a54b-d868de87d7d3-kube-api-access-8fdbc\") pod \"ingress-canary-hlfhx\" (UID: \"8a7b4cd2-b289-42b0-a54b-d868de87d7d3\") " pod="openshift-ingress-canary/ingress-canary-hlfhx" Oct 30 16:22:42.262551 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:42.262461 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f99rt\" (UniqueName: \"kubernetes.io/projected/7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc-kube-api-access-f99rt\") pod \"dns-default-jvqzq\" (UID: \"7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc\") " pod="openshift-dns/dns-default-jvqzq" Oct 30 16:22:42.262551 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:42.262485 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8a7b4cd2-b289-42b0-a54b-d868de87d7d3-cert\") pod \"ingress-canary-hlfhx\" (UID: \"8a7b4cd2-b289-42b0-a54b-d868de87d7d3\") " pod="openshift-ingress-canary/ingress-canary-hlfhx" Oct 30 16:22:42.262551 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:42.262514 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc-metrics-tls\") pod \"dns-default-jvqzq\" (UID: \"7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc\") " pod="openshift-dns/dns-default-jvqzq" Oct 30 16:22:42.262813 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:42.262684 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc-tmp-dir\") pod \"dns-default-jvqzq\" (UID: \"7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc\") " pod="openshift-dns/dns-default-jvqzq" Oct 30 16:22:42.263063 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:42.262921 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc-config-volume\") pod \"dns-default-jvqzq\" (UID: \"7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc\") " pod="openshift-dns/dns-default-jvqzq" Oct 30 16:22:42.295706 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:42.295615 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 16:22:42.295706 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:42.295673 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 16:22:42.295865 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:42.295823 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-65hn8\"" Oct 30 16:22:42.364108 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:42.364079 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fdbc\" (UniqueName: \"kubernetes.io/projected/8a7b4cd2-b289-42b0-a54b-d868de87d7d3-kube-api-access-8fdbc\") pod \"ingress-canary-hlfhx\" (UID: \"8a7b4cd2-b289-42b0-a54b-d868de87d7d3\") " pod="openshift-ingress-canary/ingress-canary-hlfhx" Oct 30 16:22:42.364108 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:42.364110 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f99rt\" (UniqueName: \"kubernetes.io/projected/7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc-kube-api-access-f99rt\") pod \"dns-default-jvqzq\" (UID: \"7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc\") " pod="openshift-dns/dns-default-jvqzq" Oct 30 16:22:42.364299 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:42.364128 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8a7b4cd2-b289-42b0-a54b-d868de87d7d3-cert\") pod \"ingress-canary-hlfhx\" (UID: \"8a7b4cd2-b289-42b0-a54b-d868de87d7d3\") " pod="openshift-ingress-canary/ingress-canary-hlfhx" Oct 30 16:22:42.364299 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:42.364143 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc-metrics-tls\") pod \"dns-default-jvqzq\" (UID: \"7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc\") " pod="openshift-dns/dns-default-jvqzq" Oct 30 16:22:42.364410 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:42.364324 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 16:22:42.364410 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:42.364390 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc-metrics-tls podName:7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc nodeName:}" failed. No retries permitted until 2026-10-30 16:22:42.864371722 +0000 UTC m=+30.592784358 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc-metrics-tls") pod "dns-default-jvqzq" (UID: "7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc") : secret "dns-default-metrics-tls" not found Oct 30 16:22:42.364519 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:42.364410 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc-tmp-dir\") pod \"dns-default-jvqzq\" (UID: \"7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc\") " pod="openshift-dns/dns-default-jvqzq" Oct 30 16:22:42.364519 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:42.364461 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc-config-volume\") pod \"dns-default-jvqzq\" (UID: \"7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc\") " pod="openshift-dns/dns-default-jvqzq" Oct 30 16:22:42.364817 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:42.364778 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc-tmp-dir\") pod \"dns-default-jvqzq\" (UID: \"7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc\") " pod="openshift-dns/dns-default-jvqzq" Oct 30 16:22:42.365008 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:42.364993 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc-config-volume\") pod \"dns-default-jvqzq\" (UID: \"7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc\") " pod="openshift-dns/dns-default-jvqzq" Oct 30 16:22:42.368399 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:42.368375 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8a7b4cd2-b289-42b0-a54b-d868de87d7d3-cert\") pod \"ingress-canary-hlfhx\" (UID: \"8a7b4cd2-b289-42b0-a54b-d868de87d7d3\") " pod="openshift-ingress-canary/ingress-canary-hlfhx" Oct 30 16:22:42.385750 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:42.385724 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f99rt\" (UniqueName: \"kubernetes.io/projected/7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc-kube-api-access-f99rt\") pod \"dns-default-jvqzq\" (UID: \"7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc\") " pod="openshift-dns/dns-default-jvqzq" Oct 30 16:22:42.385880 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:42.385864 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fdbc\" (UniqueName: \"kubernetes.io/projected/8a7b4cd2-b289-42b0-a54b-d868de87d7d3-kube-api-access-8fdbc\") pod \"ingress-canary-hlfhx\" (UID: \"8a7b4cd2-b289-42b0-a54b-d868de87d7d3\") " pod="openshift-ingress-canary/ingress-canary-hlfhx" Oct 30 16:22:42.494885 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:42.494860 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hlfhx" Oct 30 16:22:42.797252 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:42.797224 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hlfhx"] Oct 30 16:22:42.802244 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:42.802197 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a7b4cd2_b289_42b0_a54b_d868de87d7d3.slice/crio-4bb20d60d50b5c682dbd00d98d336b88192dabc3052c9dea60bb4c802633c659 WatchSource:0}: Error finding container 4bb20d60d50b5c682dbd00d98d336b88192dabc3052c9dea60bb4c802633c659: Status 404 returned error can't find the container with id 4bb20d60d50b5c682dbd00d98d336b88192dabc3052c9dea60bb4c802633c659 Oct 30 16:22:42.867946 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:42.867922 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc-metrics-tls\") pod \"dns-default-jvqzq\" (UID: \"7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc\") " pod="openshift-dns/dns-default-jvqzq" Oct 30 16:22:42.868073 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:42.868059 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 16:22:42.868123 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:42.868111 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc-metrics-tls podName:7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc nodeName:}" failed. No retries permitted until 2026-10-30 16:22:43.868099275 +0000 UTC m=+31.596511906 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc-metrics-tls") pod "dns-default-jvqzq" (UID: "7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc") : secret "dns-default-metrics-tls" not found Oct 30 16:22:43.036622 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:43.036592 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hlfhx" event={"ID":"8a7b4cd2-b289-42b0-a54b-d868de87d7d3","Type":"ContainerStarted","Data":"4bb20d60d50b5c682dbd00d98d336b88192dabc3052c9dea60bb4c802633c659"} Oct 30 16:22:43.765325 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:43.765283 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-426gj" Oct 30 16:22:43.765325 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:43.765309 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mbqwf" Oct 30 16:22:43.765586 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:43.765553 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2lrlr" Oct 30 16:22:43.778762 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:43.778739 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 16:22:43.779069 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:43.778998 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 16:22:43.779069 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:43.779008 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-vhqb4\"" Oct 30 16:22:43.779069 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:43.778997 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 16:22:43.788732 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:43.788714 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hpqxb\"" Oct 30 16:22:43.788863 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:43.788714 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-9txpl\"" Oct 30 16:22:43.788863 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:43.788810 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 16:22:43.874841 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:43.874809 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc-metrics-tls\") pod \"dns-default-jvqzq\" (UID: \"7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc\") " pod="openshift-dns/dns-default-jvqzq" Oct 30 16:22:43.875229 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:43.874937 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 16:22:43.875229 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:43.874988 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc-metrics-tls podName:7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc nodeName:}" failed. No retries permitted until 2026-10-30 16:22:45.874971156 +0000 UTC m=+33.603383792 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc-metrics-tls") pod "dns-default-jvqzq" (UID: "7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc") : secret "dns-default-metrics-tls" not found Oct 30 16:22:45.047138 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:45.047106 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-fftpj"] Oct 30 16:22:45.079109 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:45.079078 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-fftpj"] Oct 30 16:22:45.079282 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:45.079199 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fftpj" Oct 30 16:22:45.166714 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:45.166647 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 16:22:45.166918 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:45.166732 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 16:22:45.166918 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:45.166757 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-bls9m\"" Oct 30 16:22:45.184269 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:45.184236 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hgtv\" (UniqueName: \"kubernetes.io/projected/21fd140d-96ea-462e-9f8b-ef6d480f8407-kube-api-access-5hgtv\") pod \"migrator-66df5d496b-fftpj\" (UID: \"21fd140d-96ea-462e-9f8b-ef6d480f8407\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fftpj" Oct 30 16:22:45.285018 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:45.284984 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hgtv\" (UniqueName: \"kubernetes.io/projected/21fd140d-96ea-462e-9f8b-ef6d480f8407-kube-api-access-5hgtv\") pod \"migrator-66df5d496b-fftpj\" (UID: \"21fd140d-96ea-462e-9f8b-ef6d480f8407\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fftpj" Oct 30 16:22:45.306541 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:45.306488 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hgtv\" (UniqueName: \"kubernetes.io/projected/21fd140d-96ea-462e-9f8b-ef6d480f8407-kube-api-access-5hgtv\") pod \"migrator-66df5d496b-fftpj\" (UID: \"21fd140d-96ea-462e-9f8b-ef6d480f8407\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fftpj" Oct 30 16:22:45.401118 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:45.401101 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fftpj" Oct 30 16:22:45.709612 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:45.709581 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-fftpj"] Oct 30 16:22:45.714460 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:45.714437 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21fd140d_96ea_462e_9f8b_ef6d480f8407.slice/crio-6fa32b3a14c77ca3475739bb9a49d813fb5e7bcb2e4a30439e74f1ee02956b21 WatchSource:0}: Error finding container 6fa32b3a14c77ca3475739bb9a49d813fb5e7bcb2e4a30439e74f1ee02956b21: Status 404 returned error can't find the container with id 6fa32b3a14c77ca3475739bb9a49d813fb5e7bcb2e4a30439e74f1ee02956b21 Oct 30 16:22:45.889338 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:45.889311 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc-metrics-tls\") pod \"dns-default-jvqzq\" (UID: \"7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc\") " pod="openshift-dns/dns-default-jvqzq" Oct 30 16:22:45.889456 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:45.889430 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 16:22:45.889493 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:45.889481 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc-metrics-tls podName:7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc nodeName:}" failed. No retries permitted until 2026-10-30 16:22:49.889466215 +0000 UTC m=+37.617878847 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc-metrics-tls") pod "dns-default-jvqzq" (UID: "7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc") : secret "dns-default-metrics-tls" not found Oct 30 16:22:45.893098 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:45.893078 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-c24kh"] Oct 30 16:22:45.926319 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:45.926295 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-c24kh" Oct 30 16:22:45.927190 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:45.927169 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-c24kh"] Oct 30 16:22:46.021067 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:46.021051 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 16:22:46.021155 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:46.021134 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 16:22:46.021201 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:46.021157 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 16:22:46.021450 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:46.021434 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 16:22:46.021551 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:46.021539 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-lpvkt\"" Oct 30 16:22:46.044909 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:46.044883 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hlfhx" event={"ID":"8a7b4cd2-b289-42b0-a54b-d868de87d7d3","Type":"ContainerStarted","Data":"5dc5109bfb3d9caa8fa2646f22be4eb864ae1503e6fb7d03929908d6326f0393"} Oct 30 16:22:46.046009 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:46.045986 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fftpj" event={"ID":"21fd140d-96ea-462e-9f8b-ef6d480f8407","Type":"ContainerStarted","Data":"6fa32b3a14c77ca3475739bb9a49d813fb5e7bcb2e4a30439e74f1ee02956b21"} Oct 30 16:22:46.089245 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:46.089206 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-hlfhx" podStartSLOduration=1.5757343590000001 podStartE2EDuration="4.089193839s" totalImagesPullingTime="2.51345948s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:22:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:22:46.087922332 +0000 UTC m=+33.816334971" watchObservedRunningTime="2026-10-30 16:22:46.089193839 +0000 UTC m=+33.817606473" Oct 30 16:22:46.090980 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:46.090959 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c24kh\" (UID: \"8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c\") " pod="openshift-insights/insights-runtime-extractor-c24kh" Oct 30 16:22:46.091056 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:46.090990 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c-data-volume\") pod \"insights-runtime-extractor-c24kh\" (UID: \"8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c\") " pod="openshift-insights/insights-runtime-extractor-c24kh" Oct 30 16:22:46.091056 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:46.091032 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c24kh\" (UID: \"8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c\") " pod="openshift-insights/insights-runtime-extractor-c24kh" Oct 30 16:22:46.091171 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:46.091148 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c-crio-socket\") pod \"insights-runtime-extractor-c24kh\" (UID: \"8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c\") " pod="openshift-insights/insights-runtime-extractor-c24kh" Oct 30 16:22:46.091284 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:46.091176 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j5vz\" (UniqueName: \"kubernetes.io/projected/8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c-kube-api-access-5j5vz\") pod \"insights-runtime-extractor-c24kh\" (UID: \"8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c\") " pod="openshift-insights/insights-runtime-extractor-c24kh" Oct 30 16:22:46.191966 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:46.191904 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c24kh\" (UID: \"8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c\") " pod="openshift-insights/insights-runtime-extractor-c24kh" Oct 30 16:22:46.192169 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:46.192057 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 16:22:46.192233 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:46.192172 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c-insights-runtime-extractor-tls podName:8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c nodeName:}" failed. No retries permitted until 2026-10-30 16:22:46.692147921 +0000 UTC m=+34.420560561 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c24kh" (UID: "8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c") : secret "insights-runtime-extractor-tls" not found Oct 30 16:22:46.192575 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:46.192543 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c-data-volume\") pod \"insights-runtime-extractor-c24kh\" (UID: \"8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c\") " pod="openshift-insights/insights-runtime-extractor-c24kh" Oct 30 16:22:46.195443 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:46.192631 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c24kh\" (UID: \"8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c\") " pod="openshift-insights/insights-runtime-extractor-c24kh" Oct 30 16:22:46.195443 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:46.192759 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c-crio-socket\") pod \"insights-runtime-extractor-c24kh\" (UID: \"8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c\") " pod="openshift-insights/insights-runtime-extractor-c24kh" Oct 30 16:22:46.195443 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:46.192806 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j5vz\" (UniqueName: \"kubernetes.io/projected/8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c-kube-api-access-5j5vz\") pod \"insights-runtime-extractor-c24kh\" (UID: \"8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c\") " pod="openshift-insights/insights-runtime-extractor-c24kh" Oct 30 16:22:46.195443 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:46.193610 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c-data-volume\") pod \"insights-runtime-extractor-c24kh\" (UID: \"8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c\") " pod="openshift-insights/insights-runtime-extractor-c24kh" Oct 30 16:22:46.195443 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:46.193721 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 16:22:46.195443 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:46.193802 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c-kube-rbac-proxy-cm podName:8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c nodeName:}" failed. No retries permitted until 2026-10-30 16:22:46.693766323 +0000 UTC m=+34.422178965 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c24kh" (UID: "8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c") : configmap "kube-rbac-proxy" not found Oct 30 16:22:46.195443 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:46.194897 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c-crio-socket\") pod \"insights-runtime-extractor-c24kh\" (UID: \"8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c\") " pod="openshift-insights/insights-runtime-extractor-c24kh" Oct 30 16:22:46.215704 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:46.215680 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j5vz\" (UniqueName: \"kubernetes.io/projected/8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c-kube-api-access-5j5vz\") pod \"insights-runtime-extractor-c24kh\" (UID: \"8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c\") " pod="openshift-insights/insights-runtime-extractor-c24kh" Oct 30 16:22:46.229360 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:46.229339 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-5m7b4"] Oct 30 16:22:46.258713 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:46.258689 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-5m7b4"] Oct 30 16:22:46.258863 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:46.258811 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-5m7b4" Oct 30 16:22:46.271757 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:46.271695 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 16:22:46.271876 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:46.271822 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 16:22:46.271876 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:46.271847 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-cmtxm\"" Oct 30 16:22:46.271876 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:46.271847 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 16:22:46.272003 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:46.271852 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 16:22:46.395205 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:46.395167 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/315c49d6-d546-4fc2-a5c0-3b987269e5a9-signing-cabundle\") pod \"service-ca-5fffc97bdf-5m7b4\" (UID: \"315c49d6-d546-4fc2-a5c0-3b987269e5a9\") " pod="openshift-service-ca/service-ca-5fffc97bdf-5m7b4" Oct 30 16:22:46.395374 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:46.395229 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/315c49d6-d546-4fc2-a5c0-3b987269e5a9-signing-key\") pod \"service-ca-5fffc97bdf-5m7b4\" (UID: \"315c49d6-d546-4fc2-a5c0-3b987269e5a9\") " pod="openshift-service-ca/service-ca-5fffc97bdf-5m7b4" Oct 30 16:22:46.395428 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:46.395381 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntggj\" (UniqueName: \"kubernetes.io/projected/315c49d6-d546-4fc2-a5c0-3b987269e5a9-kube-api-access-ntggj\") pod \"service-ca-5fffc97bdf-5m7b4\" (UID: \"315c49d6-d546-4fc2-a5c0-3b987269e5a9\") " pod="openshift-service-ca/service-ca-5fffc97bdf-5m7b4" Oct 30 16:22:46.495902 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:46.495872 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b804fc37-0b07-4edf-8720-b31b5698b935-metrics-certs\") pod \"network-metrics-daemon-mbqwf\" (UID: \"b804fc37-0b07-4edf-8720-b31b5698b935\") " pod="openshift-multus/network-metrics-daemon-mbqwf" Oct 30 16:22:46.496082 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:46.495959 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/315c49d6-d546-4fc2-a5c0-3b987269e5a9-signing-cabundle\") pod \"service-ca-5fffc97bdf-5m7b4\" (UID: \"315c49d6-d546-4fc2-a5c0-3b987269e5a9\") " pod="openshift-service-ca/service-ca-5fffc97bdf-5m7b4" Oct 30 16:22:46.496082 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:46.496001 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/315c49d6-d546-4fc2-a5c0-3b987269e5a9-signing-key\") pod \"service-ca-5fffc97bdf-5m7b4\" (UID: \"315c49d6-d546-4fc2-a5c0-3b987269e5a9\") " pod="openshift-service-ca/service-ca-5fffc97bdf-5m7b4" Oct 30 16:22:46.496082 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:46.496025 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc77c50f-abe0-4d9c-a88b-7a186c17dc3e-original-pull-secret\") pod \"global-pull-secret-syncer-2lrlr\" (UID: \"dc77c50f-abe0-4d9c-a88b-7a186c17dc3e\") " pod="kube-system/global-pull-secret-syncer-2lrlr" Oct 30 16:22:46.496082 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:46.496066 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntggj\" (UniqueName: \"kubernetes.io/projected/315c49d6-d546-4fc2-a5c0-3b987269e5a9-kube-api-access-ntggj\") pod \"service-ca-5fffc97bdf-5m7b4\" (UID: \"315c49d6-d546-4fc2-a5c0-3b987269e5a9\") " pod="openshift-service-ca/service-ca-5fffc97bdf-5m7b4" Oct 30 16:22:46.496289 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:46.496024 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 16:22:46.496289 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:46.496212 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b804fc37-0b07-4edf-8720-b31b5698b935-metrics-certs podName:b804fc37-0b07-4edf-8720-b31b5698b935 nodeName:}" failed. No retries permitted until 2026-10-30 16:23:18.496193194 +0000 UTC m=+66.224605817 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b804fc37-0b07-4edf-8720-b31b5698b935-metrics-certs") pod "network-metrics-daemon-mbqwf" (UID: "b804fc37-0b07-4edf-8720-b31b5698b935") : secret "metrics-daemon-secret" not found Oct 30 16:22:46.496977 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:46.496954 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/315c49d6-d546-4fc2-a5c0-3b987269e5a9-signing-cabundle\") pod \"service-ca-5fffc97bdf-5m7b4\" (UID: \"315c49d6-d546-4fc2-a5c0-3b987269e5a9\") " pod="openshift-service-ca/service-ca-5fffc97bdf-5m7b4" Oct 30 16:22:46.499170 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:46.499151 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc77c50f-abe0-4d9c-a88b-7a186c17dc3e-original-pull-secret\") pod \"global-pull-secret-syncer-2lrlr\" (UID: \"dc77c50f-abe0-4d9c-a88b-7a186c17dc3e\") " pod="kube-system/global-pull-secret-syncer-2lrlr" Oct 30 16:22:46.499287 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:46.499268 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/315c49d6-d546-4fc2-a5c0-3b987269e5a9-signing-key\") pod \"service-ca-5fffc97bdf-5m7b4\" (UID: \"315c49d6-d546-4fc2-a5c0-3b987269e5a9\") " pod="openshift-service-ca/service-ca-5fffc97bdf-5m7b4" Oct 30 16:22:46.517741 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:46.517722 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntggj\" (UniqueName: \"kubernetes.io/projected/315c49d6-d546-4fc2-a5c0-3b987269e5a9-kube-api-access-ntggj\") pod \"service-ca-5fffc97bdf-5m7b4\" (UID: \"315c49d6-d546-4fc2-a5c0-3b987269e5a9\") " pod="openshift-service-ca/service-ca-5fffc97bdf-5m7b4" Oct 30 16:22:46.551740 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:46.551679 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2lrlr" Oct 30 16:22:46.582476 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:46.582453 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-5m7b4" Oct 30 16:22:46.596858 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:46.596832 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srfcg\" (UniqueName: \"kubernetes.io/projected/78b9342d-5361-4a8c-a771-0c1aea26a48d-kube-api-access-srfcg\") pod \"network-check-target-426gj\" (UID: \"78b9342d-5361-4a8c-a771-0c1aea26a48d\") " pod="openshift-network-diagnostics/network-check-target-426gj" Oct 30 16:22:46.599551 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:46.599528 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srfcg\" (UniqueName: \"kubernetes.io/projected/78b9342d-5361-4a8c-a771-0c1aea26a48d-kube-api-access-srfcg\") pod \"network-check-target-426gj\" (UID: \"78b9342d-5361-4a8c-a771-0c1aea26a48d\") " pod="openshift-network-diagnostics/network-check-target-426gj" Oct 30 16:22:46.697554 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:46.697521 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c24kh\" (UID: \"8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c\") " pod="openshift-insights/insights-runtime-extractor-c24kh" Oct 30 16:22:46.697705 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:46.697598 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c24kh\" (UID: \"8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c\") " pod="openshift-insights/insights-runtime-extractor-c24kh" Oct 30 16:22:46.697771 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:46.697712 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 16:22:46.697771 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:46.697714 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 16:22:46.697907 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:46.697795 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c-kube-rbac-proxy-cm podName:8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c nodeName:}" failed. No retries permitted until 2026-10-30 16:22:47.697766162 +0000 UTC m=+35.426178794 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c24kh" (UID: "8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c") : configmap "kube-rbac-proxy" not found Oct 30 16:22:46.697907 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:46.697815 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c-insights-runtime-extractor-tls podName:8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c nodeName:}" failed. No retries permitted until 2026-10-30 16:22:47.6978072 +0000 UTC m=+35.426219819 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c24kh" (UID: "8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c") : secret "insights-runtime-extractor-tls" not found Oct 30 16:22:46.805149 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:46.805065 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-426gj" Oct 30 16:22:46.863027 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:46.862997 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2lrlr"] Oct 30 16:22:46.909149 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:46.909120 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc77c50f_abe0_4d9c_a88b_7a186c17dc3e.slice/crio-285446f717ed93d43eb8db91ddceed824babc5ca8a476aa8aeb18ee857b9381d WatchSource:0}: Error finding container 285446f717ed93d43eb8db91ddceed824babc5ca8a476aa8aeb18ee857b9381d: Status 404 returned error can't find the container with id 285446f717ed93d43eb8db91ddceed824babc5ca8a476aa8aeb18ee857b9381d Oct 30 16:22:46.917661 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:46.917642 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-5m7b4"] Oct 30 16:22:46.921756 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:46.921736 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod315c49d6_d546_4fc2_a5c0_3b987269e5a9.slice/crio-3f1e11ba28da1d45a99744adc81ee6df04c856ee09e809cf19589bad3b33ebef WatchSource:0}: Error finding container 3f1e11ba28da1d45a99744adc81ee6df04c856ee09e809cf19589bad3b33ebef: Status 404 returned error can't find the container with id 3f1e11ba28da1d45a99744adc81ee6df04c856ee09e809cf19589bad3b33ebef Oct 30 16:22:47.049377 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:47.049344 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2lrlr" event={"ID":"dc77c50f-abe0-4d9c-a88b-7a186c17dc3e","Type":"ContainerStarted","Data":"285446f717ed93d43eb8db91ddceed824babc5ca8a476aa8aeb18ee857b9381d"} Oct 30 16:22:47.050604 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:47.050581 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-5m7b4" event={"ID":"315c49d6-d546-4fc2-a5c0-3b987269e5a9","Type":"ContainerStarted","Data":"3f1e11ba28da1d45a99744adc81ee6df04c856ee09e809cf19589bad3b33ebef"} Oct 30 16:22:47.122606 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:47.122554 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-426gj"] Oct 30 16:22:47.249393 ip-10-0-132-180 kubenswrapper[2575]: W1030 16:22:47.249368 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78b9342d_5361_4a8c_a771_0c1aea26a48d.slice/crio-414f44d6a75e2e5ffcc124804fba274175dca7f97a137c7980160c4d84a0d8fe WatchSource:0}: Error finding container 414f44d6a75e2e5ffcc124804fba274175dca7f97a137c7980160c4d84a0d8fe: Status 404 returned error can't find the container with id 414f44d6a75e2e5ffcc124804fba274175dca7f97a137c7980160c4d84a0d8fe Oct 30 16:22:47.705399 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:47.705352 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c24kh\" (UID: \"8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c\") " pod="openshift-insights/insights-runtime-extractor-c24kh" Oct 30 16:22:47.705562 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:47.705418 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c24kh\" (UID: \"8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c\") " pod="openshift-insights/insights-runtime-extractor-c24kh" Oct 30 16:22:47.705638 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:47.705561 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 16:22:47.705638 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:47.705621 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c-kube-rbac-proxy-cm podName:8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c nodeName:}" failed. No retries permitted until 2026-10-30 16:22:49.705604658 +0000 UTC m=+37.434017284 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c24kh" (UID: "8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c") : configmap "kube-rbac-proxy" not found Oct 30 16:22:47.705928 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:47.705900 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 16:22:47.706037 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:47.705971 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c-insights-runtime-extractor-tls podName:8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c nodeName:}" failed. No retries permitted until 2026-10-30 16:22:49.7059535 +0000 UTC m=+37.434366132 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c24kh" (UID: "8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c") : secret "insights-runtime-extractor-tls" not found Oct 30 16:22:48.055983 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:48.055902 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-426gj" event={"ID":"78b9342d-5361-4a8c-a771-0c1aea26a48d","Type":"ContainerStarted","Data":"414f44d6a75e2e5ffcc124804fba274175dca7f97a137c7980160c4d84a0d8fe"} Oct 30 16:22:48.057981 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:48.057954 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fftpj" event={"ID":"21fd140d-96ea-462e-9f8b-ef6d480f8407","Type":"ContainerStarted","Data":"3649ac9a25961336f8dd4b81706baf7f3d05bbe7128e9542da49417441aa17fa"} Oct 30 16:22:48.093871 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:48.093815 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fftpj" podStartSLOduration=1.535164467 podStartE2EDuration="3.09380094s" totalImagesPullingTime="1.558636473s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:22:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:22:48.093522531 +0000 UTC m=+35.821935171" watchObservedRunningTime="2026-10-30 16:22:48.09380094 +0000 UTC m=+35.822213574" Oct 30 16:22:49.723213 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:49.723056 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c24kh\" (UID: \"8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c\") " pod="openshift-insights/insights-runtime-extractor-c24kh" Oct 30 16:22:49.723213 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:49.723122 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c24kh\" (UID: \"8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c\") " pod="openshift-insights/insights-runtime-extractor-c24kh" Oct 30 16:22:49.723213 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:49.723208 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 16:22:49.723213 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:49.723215 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 16:22:49.723700 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:49.723272 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c-kube-rbac-proxy-cm podName:8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c nodeName:}" failed. No retries permitted until 2026-10-30 16:22:53.72325581 +0000 UTC m=+41.451668450 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c24kh" (UID: "8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c") : configmap "kube-rbac-proxy" not found Oct 30 16:22:49.723700 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:49.723287 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c-insights-runtime-extractor-tls podName:8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c nodeName:}" failed. No retries permitted until 2026-10-30 16:22:53.72328125 +0000 UTC m=+41.451693868 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c24kh" (UID: "8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c") : secret "insights-runtime-extractor-tls" not found Oct 30 16:22:49.924595 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:49.924561 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc-metrics-tls\") pod \"dns-default-jvqzq\" (UID: \"7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc\") " pod="openshift-dns/dns-default-jvqzq" Oct 30 16:22:49.924770 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:49.924679 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 16:22:49.924770 ip-10-0-132-180 kubenswrapper[2575]: E1030 16:22:49.924731 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc-metrics-tls podName:7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc nodeName:}" failed. No retries permitted until 2026-10-30 16:22:57.924716876 +0000 UTC m=+45.653129494 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc-metrics-tls") pod "dns-default-jvqzq" (UID: "7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc") : secret "dns-default-metrics-tls" not found Oct 30 16:22:50.857360 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:50.857285 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pg4kq_7c09760d-2160-41d4-b991-f6324ad5aff7/konnectivity-agent/0.log" limit=1024 Oct 30 16:22:51.067722 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:51.067696 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-5m7b4" event={"ID":"315c49d6-d546-4fc2-a5c0-3b987269e5a9","Type":"ContainerStarted","Data":"19f79c491ae73e021a4620933fe42f1dcb3f0cce77e8b92f2b3148706619ec3a"} Oct 30 16:22:51.070035 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:51.070006 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-426gj" event={"ID":"78b9342d-5361-4a8c-a771-0c1aea26a48d","Type":"ContainerStarted","Data":"87d039dffb0da101783ef3a096823ba3048b7bbb6c4328b9bec0396416aea515"} Oct 30 16:22:51.106559 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:51.106514 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-5m7b4" podStartSLOduration=1.8312715000000002 podStartE2EDuration="5.106502448s" totalImagesPullingTime="3.275230948s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:22:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:22:51.105904719 +0000 UTC m=+38.834317360" watchObservedRunningTime="2026-10-30 16:22:51.106502448 +0000 UTC m=+38.834915135" Oct 30 16:22:52.072724 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:52.072688 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-426gj" Oct 30 16:22:52.111899 ip-10-0-132-180 kubenswrapper[2575]: I1030 16:22:52.111857 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-426gj" podStartSLOduration=35.41643075 podStartE2EDuration="39.11184359s" totalImagesPullingTime="3.69541284s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:22:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:22:52.110862905 +0000 UTC m=+39.839275545" watchObservedRunningTime="2026-10-30 16:22:52.11184359 +0000 UTC m=+39.840256271" Oct 30 16:22:52.702519 ip-10-0-132-180 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 16:22:52.724167 ip-10-0-132-180 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 16:22:52.724338 ip-10-0-132-180 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 16:22:52.724392 ip-10-0-132-180 systemd[1]: kubelet.service: Consumed 2.328s CPU time. Oct 30 16:22:52.743300 ip-10-0-132-180 systemd[1]: Starting Kubernetes Kubelet... Oct 30 16:22:52.838621 ip-10-0-132-180 kubenswrapper[4493]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 16:22:52.838621 ip-10-0-132-180 kubenswrapper[4493]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 16:22:52.838621 ip-10-0-132-180 kubenswrapper[4493]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 16:22:52.838621 ip-10-0-132-180 kubenswrapper[4493]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 16:22:52.841541 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841453 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:22:52.841541 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841537 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:22:52.841541 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841541 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:22:52.841541 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841544 4493 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:22:52.841541 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841546 4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:22:52.841541 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841549 4493 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:22:52.841901 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841552 4493 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:22:52.841901 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841556 4493 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:22:52.841901 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841558 4493 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:22:52.841901 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841561 4493 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:22:52.841901 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841563 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:22:52.841901 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841566 4493 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:22:52.841901 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841568 4493 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:22:52.841901 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841571 4493 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:22:52.841901 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841573 4493 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:22:52.841901 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841576 4493 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:22:52.841901 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841579 4493 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:22:52.841901 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841581 4493 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:22:52.841901 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841583 4493 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:22:52.841901 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841586 4493 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:22:52.841901 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841588 4493 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:22:52.841901 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841591 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:22:52.841901 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841593 4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:22:52.841901 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841596 4493 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:22:52.841901 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841598 4493 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:22:52.841901 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841601 4493 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:22:52.842621 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841604 4493 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:22:52.842621 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841606 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:22:52.842621 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841610 4493 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:22:52.842621 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841612 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:22:52.842621 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841615 4493 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:22:52.842621 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841618 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:22:52.842621 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841621 4493 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:22:52.842621 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841623 4493 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:22:52.842621 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841626 4493 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:22:52.842621 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841629 4493 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:22:52.842621 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841632 4493 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:22:52.842621 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841634 4493 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:22:52.842621 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841637 4493 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:22:52.842621 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841639 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:22:52.842621 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841642 4493 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:22:52.842621 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841644 4493 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:22:52.842621 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841647 4493 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:22:52.842621 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841649 4493 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:22:52.842621 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841652 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:22:52.842621 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841654 4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:22:52.843144 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841657 4493 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:22:52.843144 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841659 4493 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:22:52.843144 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841662 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:22:52.843144 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841664 4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:22:52.843144 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841674 4493 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:22:52.843144 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841677 4493 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:22:52.843144 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841679 4493 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:22:52.843144 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841682 4493 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:22:52.843144 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841684 4493 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:22:52.843144 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841687 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:22:52.843144 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841690 4493 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:22:52.843144 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841693 4493 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:22:52.843144 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841695 4493 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:22:52.843144 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841698 4493 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:22:52.843144 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841701 4493 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:22:52.843144 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841703 4493 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:22:52.843144 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841706 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:22:52.843144 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841709 4493 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:22:52.843144 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841712 4493 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:22:52.843591 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841714 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:22:52.843591 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841717 4493 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:22:52.843591 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841720 4493 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:22:52.843591 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841722 4493 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:22:52.843591 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841725 4493 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:22:52.843591 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841729 4493 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:22:52.843591 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841733 4493 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:22:52.843591 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841736 4493 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:22:52.843591 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841739 4493 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:22:52.843591 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841741 4493 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:22:52.843591 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841744 4493 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:22:52.843591 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841746 4493 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:22:52.843591 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841748 4493 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:22:52.843591 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841751 4493 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:22:52.843591 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841753 4493 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:22:52.843591 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841756 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:22:52.843591 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841758 4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:22:52.843591 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841761 4493 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:22:52.843591 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841765 4493 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:22:52.844078 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841771 4493 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:22:52.844078 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841776 4493 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:22:52.844078 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841795 4493 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:22:52.844078 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841800 4493 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:22:52.844078 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841803 4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:22:52.844078 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841807 4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:22:52.844078 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841811 4493 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:22:52.844078 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841815 4493 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:22:52.844078 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841822 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:22:52.844078 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841825 4493 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:22:52.844078 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841828 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:22:52.844078 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841831 4493 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:22:52.844078 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841834 4493 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:22:52.844078 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841836 4493 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:22:52.844078 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841839 4493 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:22:52.844078 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841841 4493 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:22:52.844078 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841844 4493 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:22:52.844078 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841846 4493 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:22:52.844078 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841849 4493 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:22:52.844078 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841858 4493 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:22:52.844598 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841862 4493 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:22:52.844598 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.841864 4493 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:22:52.844598 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843148 4493 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:22:52.844598 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843160 4493 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:22:52.844598 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843165 4493 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:22:52.844598 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843169 4493 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:22:52.844598 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843174 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:22:52.844598 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843180 4493 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:22:52.844598 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843186 4493 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:22:52.844598 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843191 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:22:52.844598 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843196 4493 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:22:52.844598 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843201 4493 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:22:52.844598 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843206 4493 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:22:52.844598 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843210 4493 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:22:52.844598 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843214 4493 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:22:52.844598 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843218 4493 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:22:52.844598 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843222 4493 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:22:52.844598 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843226 4493 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:22:52.844598 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843230 4493 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:22:52.844598 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843234 4493 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:22:52.845098 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843240 4493 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:22:52.845098 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843244 4493 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:22:52.845098 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843248 4493 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:22:52.845098 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843253 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:22:52.845098 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843257 4493 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:22:52.845098 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843261 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:22:52.845098 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843265 4493 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:22:52.845098 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843269 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:22:52.845098 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843273 4493 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:22:52.845098 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843277 4493 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:22:52.845098 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843281 4493 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:22:52.845098 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843285 4493 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:22:52.845098 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843289 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:22:52.845098 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843293 4493 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:22:52.845098 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843297 4493 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:22:52.845098 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843301 4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:22:52.845098 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843305 4493 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:22:52.845098 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843309 4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:22:52.845098 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843313 4493 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:22:52.845098 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843317 4493 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:22:52.845598 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843321 4493 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:22:52.845598 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843325 4493 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:22:52.845598 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843329 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:22:52.845598 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843333 4493 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:22:52.845598 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843337 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:22:52.845598 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843342 4493 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:22:52.845598 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843346 4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:22:52.845598 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843350 4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:22:52.845598 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843354 4493 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:22:52.845598 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843358 4493 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:22:52.845598 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843362 4493 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:22:52.845598 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843366 4493 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:22:52.845598 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843370 4493 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:22:52.845598 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843375 4493 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:22:52.845598 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843381 4493 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:22:52.845598 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843385 4493 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:22:52.845598 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843389 4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:22:52.845598 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843394 4493 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:22:52.845598 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843397 4493 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:22:52.845598 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843402 4493 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:22:52.846094 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843405 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:22:52.846094 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843410 4493 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:22:52.846094 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843414 4493 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:22:52.846094 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843418 4493 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:22:52.846094 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843422 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:22:52.846094 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843426 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:22:52.846094 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843430 4493 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:22:52.846094 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843434 4493 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:22:52.846094 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843438 4493 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:22:52.846094 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843445 4493 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:22:52.846094 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843452 4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:22:52.846094 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843456 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:22:52.846094 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843460 4493 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:22:52.846094 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843464 4493 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:22:52.846094 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843469 4493 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:22:52.846094 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843473 4493 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:22:52.846094 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843477 4493 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:22:52.846094 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843481 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:22:52.846532 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843486 4493 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:22:52.846532 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843491 4493 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:22:52.846532 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843496 4493 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:22:52.846532 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843500 4493 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:22:52.846532 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843504 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:22:52.846532 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843508 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:22:52.846532 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843512 4493 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:22:52.846532 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843516 4493 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:22:52.846532 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843522 4493 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:22:52.846532 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843526 4493 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:22:52.846532 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843531 4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:22:52.846532 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843535 4493 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:22:52.846532 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843539 4493 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:22:52.846532 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843543 4493 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:22:52.846532 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843547 4493 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:22:52.846532 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843551 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:22:52.846532 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843556 4493 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:22:52.846532 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843560 4493 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:22:52.846532 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843564 4493 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:22:52.847011 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843567 4493 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:22:52.847011 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843571 4493 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:22:52.847011 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843575 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:22:52.847011 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843579 4493 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:22:52.847011 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843583 4493 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:22:52.847011 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843587 4493 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:22:52.847011 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843590 4493 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:22:52.847011 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843594 4493 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:22:52.847011 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843599 4493 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:22:52.847011 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843602 4493 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:22:52.847011 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.843606 4493 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:22:52.847011 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844384 4493 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:22:52.847011 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844393 4493 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:22:52.847011 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844398 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:22:52.847011 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844403 4493 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:22:52.847011 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844407 4493 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:22:52.847011 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844411 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:22:52.847011 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844415 4493 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:22:52.847011 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844419 4493 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:22:52.847011 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844424 4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:22:52.847491 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844428 4493 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:22:52.847491 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844433 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:22:52.847491 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844437 4493 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:22:52.847491 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844441 4493 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:22:52.847491 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844445 4493 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:22:52.847491 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844449 4493 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:22:52.847491 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844453 4493 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:22:52.847491 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844457 4493 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:22:52.847491 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844461 4493 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:22:52.847491 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844466 4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:22:52.847491 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844470 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:22:52.847491 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844474 4493 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:22:52.847491 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844478 4493 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:22:52.847491 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844483 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:22:52.847491 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844486 4493 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:22:52.847491 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844490 4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:22:52.847491 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844494 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:22:52.847491 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844498 4493 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:22:52.847491 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844503 4493 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:22:52.847953 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844506 4493 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:22:52.847953 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844513 4493 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:22:52.847953 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844517 4493 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:22:52.847953 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844521 4493 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:22:52.847953 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844525 4493 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:22:52.847953 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844528 4493 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:22:52.847953 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844532 4493 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:22:52.847953 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844537 4493 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:22:52.847953 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844542 4493 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:22:52.847953 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844546 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:22:52.847953 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844550 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:22:52.847953 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844554 4493 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:22:52.847953 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844558 4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:22:52.847953 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844562 4493 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:22:52.847953 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844566 4493 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:22:52.847953 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844572 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:22:52.847953 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844576 4493 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:22:52.847953 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844580 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:22:52.847953 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844584 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:22:52.848412 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844588 4493 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:22:52.848412 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844592 4493 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:22:52.848412 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844597 4493 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:22:52.848412 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844601 4493 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:22:52.848412 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844605 4493 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:22:52.848412 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844609 4493 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:22:52.848412 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844615 4493 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:22:52.848412 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844621 4493 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:22:52.848412 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844626 4493 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:22:52.848412 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844631 4493 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:22:52.848412 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844636 4493 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:22:52.848412 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844641 4493 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:22:52.848412 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844647 4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:22:52.848412 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844651 4493 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:22:52.848412 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844656 4493 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:22:52.848412 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844660 4493 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:22:52.848412 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844664 4493 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:22:52.848412 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844668 4493 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:22:52.848412 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844672 4493 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:22:52.848412 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844676 4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:22:52.848929 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844680 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:22:52.848929 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844685 4493 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:22:52.848929 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844689 4493 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:22:52.848929 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844693 4493 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:22:52.848929 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844697 4493 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:22:52.848929 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844701 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:22:52.848929 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844705 4493 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:22:52.848929 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844709 4493 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:22:52.848929 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844713 4493 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:22:52.848929 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844717 4493 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:22:52.848929 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844722 4493 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:22:52.848929 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844726 4493 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:22:52.848929 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844730 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:22:52.848929 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844733 4493 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:22:52.848929 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844737 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:22:52.848929 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844741 4493 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:22:52.848929 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844745 4493 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:22:52.848929 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844749 4493 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:22:52.848929 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844754 4493 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:22:52.848929 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844758 4493 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:22:52.849446 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844762 4493 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:22:52.849446 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844765 4493 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:22:52.849446 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844769 4493 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:22:52.849446 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844773 4493 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:22:52.849446 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844777 4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:22:52.849446 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844796 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:22:52.849446 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844802 4493 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:22:52.849446 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844806 4493 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:22:52.849446 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844810 4493 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:22:52.849446 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844814 4493 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:22:52.849446 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844818 4493 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:22:52.849446 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844822 4493 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:22:52.849446 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844827 4493 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:22:52.849446 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844831 4493 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:22:52.849446 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844836 4493 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:22:52.849446 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844840 4493 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:22:52.849446 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844844 4493 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:22:52.849446 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844848 4493 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:22:52.849446 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.844852 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:22:52.849446 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.844973 4493 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 16:22:52.850001 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.844991 4493 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 16:22:52.850001 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845000 4493 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 16:22:52.850001 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845007 4493 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 16:22:52.850001 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845013 4493 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 16:22:52.850001 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845017 4493 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 16:22:52.850001 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845023 4493 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 16:22:52.850001 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845029 4493 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 16:22:52.850001 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845034 4493 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 16:22:52.850001 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845038 4493 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 16:22:52.850001 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845044 4493 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 16:22:52.850001 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845049 4493 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 16:22:52.850001 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845054 4493 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 16:22:52.850001 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845058 4493 flags.go:64] FLAG: --cgroup-root="" Oct 30 16:22:52.850001 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845062 4493 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 16:22:52.850001 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845067 4493 flags.go:64] FLAG: --client-ca-file="" Oct 30 16:22:52.850001 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845071 4493 flags.go:64] FLAG: --cloud-provider="external" Oct 30 16:22:52.850001 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845076 4493 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 16:22:52.850001 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845081 4493 flags.go:64] FLAG: --cluster-domain="" Oct 30 16:22:52.850001 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845090 4493 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 16:22:52.850001 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845095 4493 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 16:22:52.850001 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845099 4493 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 16:22:52.850001 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845104 4493 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 16:22:52.850001 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845110 4493 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 16:22:52.850001 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845115 4493 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 16:22:52.850709 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845119 4493 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 16:22:52.850709 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845124 4493 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 16:22:52.850709 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845129 4493 flags.go:64] FLAG: --contention-profiling="false" Oct 30 16:22:52.850709 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845134 4493 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 16:22:52.850709 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845139 4493 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 16:22:52.850709 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845143 4493 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 16:22:52.850709 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845148 4493 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 16:22:52.850709 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845153 4493 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 16:22:52.850709 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845158 4493 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 16:22:52.850709 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845163 4493 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 16:22:52.850709 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845167 4493 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 16:22:52.850709 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845172 4493 flags.go:64] FLAG: --enable-server="true" Oct 30 16:22:52.850709 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845176 4493 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 16:22:52.850709 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845183 4493 flags.go:64] FLAG: --event-burst="100" Oct 30 16:22:52.850709 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845188 4493 flags.go:64] FLAG: --event-qps="50" Oct 30 16:22:52.850709 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845192 4493 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 16:22:52.850709 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845197 4493 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 16:22:52.850709 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845201 4493 flags.go:64] FLAG: --eviction-hard="" Oct 30 16:22:52.850709 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845207 4493 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 16:22:52.850709 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845211 4493 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 16:22:52.850709 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845216 4493 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 16:22:52.850709 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845221 4493 flags.go:64] FLAG: --eviction-soft="" Oct 30 16:22:52.850709 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845225 4493 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 16:22:52.850709 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845230 4493 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 16:22:52.850709 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845234 4493 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 16:22:52.851385 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845239 4493 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 16:22:52.851385 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845247 4493 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 16:22:52.851385 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845252 4493 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 16:22:52.851385 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845256 4493 flags.go:64] FLAG: --feature-gates="" Oct 30 16:22:52.851385 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845262 4493 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 16:22:52.851385 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845267 4493 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 16:22:52.851385 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845272 4493 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 16:22:52.851385 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845276 4493 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 16:22:52.851385 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845281 4493 flags.go:64] FLAG: --healthz-port="10248" Oct 30 16:22:52.851385 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845286 4493 flags.go:64] FLAG: --help="false" Oct 30 16:22:52.851385 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845290 4493 flags.go:64] FLAG: --hostname-override="ip-10-0-132-180.ec2.internal" Oct 30 16:22:52.851385 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845295 4493 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 16:22:52.851385 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845300 4493 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 16:22:52.851385 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845305 4493 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 16:22:52.851385 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845310 4493 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 16:22:52.851385 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845315 4493 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 16:22:52.851385 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845319 4493 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 16:22:52.851385 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845324 4493 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 16:22:52.851385 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845328 4493 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 16:22:52.851385 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845332 4493 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 16:22:52.851385 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845337 4493 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 16:22:52.851385 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845342 4493 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 16:22:52.851385 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845346 4493 flags.go:64] FLAG: --kube-reserved="" Oct 30 16:22:52.851385 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845350 4493 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 16:22:52.851969 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845354 4493 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 16:22:52.851969 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845359 4493 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 16:22:52.851969 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845363 4493 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 16:22:52.851969 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845368 4493 flags.go:64] FLAG: --lock-file="" Oct 30 16:22:52.851969 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845372 4493 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 16:22:52.851969 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845376 4493 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 16:22:52.851969 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845381 4493 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 16:22:52.851969 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845389 4493 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 16:22:52.851969 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845394 4493 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 16:22:52.851969 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845402 4493 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 16:22:52.851969 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845407 4493 flags.go:64] FLAG: --logging-format="text" Oct 30 16:22:52.851969 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845411 4493 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 16:22:52.851969 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845416 4493 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 16:22:52.851969 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845420 4493 flags.go:64] FLAG: --manifest-url="" Oct 30 16:22:52.851969 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845425 4493 flags.go:64] FLAG: --manifest-url-header="" Oct 30 16:22:52.851969 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845431 4493 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 16:22:52.851969 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845435 4493 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 16:22:52.851969 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845441 4493 flags.go:64] FLAG: --max-pods="110" Oct 30 16:22:52.851969 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845445 4493 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 16:22:52.851969 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845450 4493 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 16:22:52.851969 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845458 4493 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 16:22:52.851969 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845463 4493 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 16:22:52.851969 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845468 4493 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 16:22:52.851969 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845472 4493 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 16:22:52.851969 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845477 4493 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 16:22:52.852612 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845487 4493 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 16:22:52.852612 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845491 4493 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 16:22:52.852612 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845496 4493 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 16:22:52.852612 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845501 4493 flags.go:64] FLAG: --pod-cidr="" Oct 30 16:22:52.852612 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845505 4493 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 16:22:52.852612 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845509 4493 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 16:22:52.852612 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845514 4493 flags.go:64] FLAG: --pods-per-core="0" Oct 30 16:22:52.852612 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845518 4493 flags.go:64] FLAG: --port="10250" Oct 30 16:22:52.852612 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845523 4493 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 16:22:52.852612 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845527 4493 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c2f018e142dba3de" Oct 30 16:22:52.852612 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845532 4493 flags.go:64] FLAG: --qos-reserved="" Oct 30 16:22:52.852612 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845536 4493 flags.go:64] FLAG: --read-only-port="10255" Oct 30 16:22:52.852612 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845541 4493 flags.go:64] FLAG: --register-node="true" Oct 30 16:22:52.852612 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845545 4493 flags.go:64] FLAG: --register-with-taints="" Oct 30 16:22:52.852612 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845551 4493 flags.go:64] FLAG: --registry-burst="10" Oct 30 16:22:52.852612 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845555 4493 flags.go:64] FLAG: --registry-qps="5" Oct 30 16:22:52.852612 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845563 4493 flags.go:64] FLAG: --reserved-cpus="" Oct 30 16:22:52.852612 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845567 4493 flags.go:64] FLAG: --reserved-memory="" Oct 30 16:22:52.852612 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845573 4493 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 16:22:52.852612 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845577 4493 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 16:22:52.852612 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845582 4493 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 16:22:52.852612 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845586 4493 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 16:22:52.852612 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845591 4493 flags.go:64] FLAG: --runonce="false" Oct 30 16:22:52.852612 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845595 4493 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 16:22:52.852612 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845600 4493 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 16:22:52.852612 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845605 4493 flags.go:64] FLAG: --seccomp-default="false" Oct 30 16:22:52.852612 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845610 4493 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 16:22:52.853284 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845614 4493 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 16:22:52.853284 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845619 4493 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 16:22:52.853284 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845624 4493 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 16:22:52.853284 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845628 4493 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 16:22:52.853284 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845633 4493 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 16:22:52.853284 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845637 4493 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 16:22:52.853284 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845641 4493 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 16:22:52.853284 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845645 4493 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 16:22:52.853284 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845650 4493 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 16:22:52.853284 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845655 4493 flags.go:64] FLAG: --system-cgroups="" Oct 30 16:22:52.853284 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845659 4493 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 16:22:52.853284 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845667 4493 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 16:22:52.853284 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845671 4493 flags.go:64] FLAG: --tls-cert-file="" Oct 30 16:22:52.853284 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845675 4493 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 16:22:52.853284 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845681 4493 flags.go:64] FLAG: --tls-min-version="" Oct 30 16:22:52.853284 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845685 4493 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 16:22:52.853284 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845689 4493 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 16:22:52.853284 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845693 4493 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 16:22:52.853284 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845698 4493 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 16:22:52.853284 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845702 4493 flags.go:64] FLAG: --v="2" Oct 30 16:22:52.853284 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845708 4493 flags.go:64] FLAG: --version="false" Oct 30 16:22:52.853284 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845717 4493 flags.go:64] FLAG: --vmodule="" Oct 30 16:22:52.853284 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845722 4493 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 16:22:52.853284 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.845727 4493 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 16:22:52.853284 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.845961 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:22:52.853920 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.845967 4493 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:22:52.853920 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.845971 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:22:52.853920 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.845975 4493 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:22:52.853920 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.845978 4493 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:22:52.853920 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.845982 4493 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:22:52.853920 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.845987 4493 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:22:52.853920 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.845990 4493 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:22:52.853920 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.845994 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:22:52.853920 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.845998 4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:22:52.853920 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846010 4493 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:22:52.853920 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846015 4493 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:22:52.853920 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846019 4493 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:22:52.853920 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846023 4493 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:22:52.853920 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846027 4493 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:22:52.853920 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846031 4493 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:22:52.853920 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846036 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:22:52.853920 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846040 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:22:52.853920 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846044 4493 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:22:52.853920 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846048 4493 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:22:52.854385 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846052 4493 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:22:52.854385 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846057 4493 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:22:52.854385 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846060 4493 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:22:52.854385 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846064 4493 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:22:52.854385 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846068 4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:22:52.854385 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846072 4493 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:22:52.854385 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846076 4493 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:22:52.854385 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846080 4493 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:22:52.854385 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846084 4493 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:22:52.854385 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846091 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:22:52.854385 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846095 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:22:52.854385 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846099 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:22:52.854385 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846103 4493 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:22:52.854385 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846108 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:22:52.854385 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846112 4493 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:22:52.854385 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846116 4493 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:22:52.854385 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846120 4493 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:22:52.854385 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846124 4493 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:22:52.854385 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846128 4493 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:22:52.854857 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846132 4493 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:22:52.854857 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846136 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:22:52.854857 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846140 4493 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:22:52.854857 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846144 4493 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:22:52.854857 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846149 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:22:52.854857 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846153 4493 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:22:52.854857 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846157 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:22:52.854857 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846161 4493 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:22:52.854857 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846165 4493 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:22:52.854857 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846169 4493 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:22:52.854857 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846173 4493 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:22:52.854857 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846177 4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:22:52.854857 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846181 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:22:52.854857 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846185 4493 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:22:52.854857 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846189 4493 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:22:52.854857 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846193 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:22:52.854857 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846198 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:22:52.854857 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846202 4493 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:22:52.854857 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846206 4493 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:22:52.855325 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846210 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:22:52.855325 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846215 4493 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:22:52.855325 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846219 4493 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:22:52.855325 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846228 4493 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:22:52.855325 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846232 4493 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:22:52.855325 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846236 4493 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:22:52.855325 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846240 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:22:52.855325 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846244 4493 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:22:52.855325 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846248 4493 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:22:52.855325 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846252 4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:22:52.855325 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846256 4493 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:22:52.855325 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846261 4493 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:22:52.855325 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846265 4493 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:22:52.855325 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846269 4493 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:22:52.855325 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846273 4493 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:22:52.855325 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846277 4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:22:52.855325 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846281 4493 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:22:52.855325 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846285 4493 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:22:52.855325 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846290 4493 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:22:52.855325 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846294 4493 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:22:52.855865 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846298 4493 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:22:52.855865 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846302 4493 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:22:52.855865 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846306 4493 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:22:52.855865 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846310 4493 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:22:52.855865 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846314 4493 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:22:52.855865 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846318 4493 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:22:52.855865 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846322 4493 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:22:52.855865 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846326 4493 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:22:52.855865 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846330 4493 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:22:52.855865 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846334 4493 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:22:52.855865 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846340 4493 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:22:52.855865 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846347 4493 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:22:52.855865 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846352 4493 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:22:52.855865 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846358 4493 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:22:52.855865 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846364 4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:22:52.855865 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846372 4493 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:22:52.855865 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846376 4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:22:52.855865 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846381 4493 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:22:52.855865 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846385 4493 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:22:52.855865 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846389 4493 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:22:52.856391 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846393 4493 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:22:52.856391 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846397 4493 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:22:52.856391 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846401 4493 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:22:52.856391 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846406 4493 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:22:52.856391 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846410 4493 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:22:52.856391 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846414 4493 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:22:52.856391 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846418 4493 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:22:52.856391 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846423 4493 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:22:52.856391 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.846461 4493 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 16:22:52.856391 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846513 4493 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:22:52.856391 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846519 4493 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:22:52.856391 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846528 4493 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:22:52.856391 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846534 4493 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:22:52.856391 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846540 4493 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:22:52.856391 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846545 4493 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:22:52.856391 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846550 4493 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:22:52.857053 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846556 4493 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:22:52.857053 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846561 4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:22:52.857053 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846566 4493 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:22:52.857053 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846570 4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:22:52.857053 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846575 4493 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:22:52.857053 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846579 4493 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:22:52.857053 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846583 4493 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:22:52.857053 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846587 4493 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:22:52.857053 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846591 4493 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:22:52.857053 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846595 4493 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:22:52.857053 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846598 4493 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:22:52.857053 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846603 4493 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:22:52.857053 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846608 4493 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:22:52.857053 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846612 4493 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:22:52.857053 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846617 4493 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:22:52.857053 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846621 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:22:52.857053 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846624 4493 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:22:52.857053 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846629 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:22:52.857053 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846632 4493 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:22:52.857819 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846636 4493 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:22:52.857819 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846640 4493 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:22:52.857819 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846644 4493 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:22:52.857819 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846648 4493 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:22:52.857819 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846652 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:22:52.857819 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846656 4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:22:52.857819 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846660 4493 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:22:52.857819 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846664 4493 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:22:52.857819 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846668 4493 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:22:52.857819 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846679 4493 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:22:52.857819 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846683 4493 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:22:52.857819 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846687 4493 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:22:52.857819 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846691 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:22:52.857819 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846695 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:22:52.857819 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846699 4493 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:22:52.857819 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846703 4493 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:22:52.857819 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846707 4493 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:22:52.857819 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846711 4493 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:22:52.857819 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846716 4493 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:22:52.858541 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846720 4493 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:22:52.858541 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846724 4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:22:52.858541 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846728 4493 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:22:52.858541 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846732 4493 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:22:52.858541 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846735 4493 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:22:52.858541 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846740 4493 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:22:52.858541 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846744 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:22:52.858541 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846748 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:22:52.858541 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846752 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:22:52.858541 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846756 4493 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:22:52.858541 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846760 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:22:52.858541 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846764 4493 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:22:52.858541 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846768 4493 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:22:52.858541 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846772 4493 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:22:52.858541 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846776 4493 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:22:52.858541 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846795 4493 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:22:52.858541 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846800 4493 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:22:52.858541 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846805 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:22:52.858541 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846809 4493 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:22:52.859233 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846813 4493 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:22:52.859233 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846817 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:22:52.859233 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846821 4493 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:22:52.859233 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846825 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:22:52.859233 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846838 4493 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:22:52.859233 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846842 4493 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:22:52.859233 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846846 4493 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:22:52.859233 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846850 4493 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:22:52.859233 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846854 4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:22:52.859233 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846858 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:22:52.859233 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846862 4493 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:22:52.859233 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846866 4493 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:22:52.859233 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846870 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:22:52.859233 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846874 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:22:52.859233 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846878 4493 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:22:52.859233 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846882 4493 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:22:52.859233 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846886 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:22:52.859233 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846890 4493 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:22:52.859233 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846895 4493 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:22:52.859741 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846898 4493 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:22:52.859741 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846903 4493 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:22:52.859741 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846906 4493 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:22:52.859741 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846910 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:22:52.859741 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846914 4493 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:22:52.859741 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846918 4493 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:22:52.859741 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846922 4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:22:52.859741 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846926 4493 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:22:52.859741 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846930 4493 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:22:52.859741 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846934 4493 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:22:52.859741 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846938 4493 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:22:52.859741 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846942 4493 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:22:52.859741 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846946 4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:22:52.859741 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846950 4493 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:22:52.859741 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846954 4493 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:22:52.859741 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846962 4493 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:22:52.859741 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846966 4493 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:22:52.859741 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846970 4493 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:22:52.859741 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846981 4493 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:22:52.859741 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846985 4493 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:22:52.860449 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846989 4493 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:22:52.860449 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846993 4493 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:22:52.860449 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.846997 4493 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:22:52.860449 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.850993 4493 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 16:22:52.860449 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.851009 4493 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 16:22:52.860449 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.851027 4493 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 16:22:52.860449 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.851032 4493 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 16:22:52.860449 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851090 4493 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:22:52.860449 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851094 4493 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:22:52.860449 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851097 4493 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:22:52.860449 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851100 4493 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:22:52.860449 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851102 4493 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:22:52.860449 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851105 4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:22:52.860449 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851108 4493 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:22:52.860449 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851110 4493 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:22:52.860449 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851113 4493 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:22:52.860449 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851116 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:22:52.860449 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851118 4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:22:52.860449 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851121 4493 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:22:52.860449 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851124 4493 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:22:52.861227 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851126 4493 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:22:52.861227 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851129 4493 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:22:52.861227 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851131 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:22:52.861227 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851134 4493 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:22:52.861227 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851136 4493 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:22:52.861227 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851139 4493 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:22:52.861227 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851141 4493 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:22:52.861227 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851143 4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:22:52.861227 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851146 4493 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:22:52.861227 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851149 4493 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:22:52.861227 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851151 4493 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:22:52.861227 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851154 4493 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:22:52.861227 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851157 4493 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:22:52.861227 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851159 4493 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:22:52.861227 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851162 4493 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:22:52.861227 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851165 4493 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:22:52.861227 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851168 4493 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:22:52.861227 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851171 4493 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:22:52.861227 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851173 4493 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:22:52.861976 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851176 4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:22:52.861976 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851178 4493 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:22:52.861976 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851181 4493 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:22:52.861976 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851183 4493 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:22:52.861976 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851186 4493 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:22:52.861976 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851189 4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:22:52.861976 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851191 4493 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:22:52.861976 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851194 4493 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:22:52.861976 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851196 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:22:52.861976 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851199 4493 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:22:52.861976 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851201 4493 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:22:52.861976 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851203 4493 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:22:52.861976 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851206 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:22:52.861976 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851208 4493 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:22:52.861976 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851211 4493 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:22:52.861976 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851213 4493 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:22:52.861976 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851216 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:22:52.861976 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851218 4493 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:22:52.861976 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851221 4493 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:22:52.862714 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851223 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:22:52.862714 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851226 4493 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:22:52.862714 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851228 4493 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:22:52.862714 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851231 4493 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:22:52.862714 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851233 4493 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:22:52.862714 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851236 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:22:52.862714 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851238 4493 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:22:52.862714 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851241 4493 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:22:52.862714 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851244 4493 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:22:52.862714 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851246 4493 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:22:52.862714 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851249 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:22:52.862714 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851252 4493 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:22:52.862714 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851255 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:22:52.862714 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851257 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:22:52.862714 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851260 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:22:52.862714 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851262 4493 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:22:52.862714 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851265 4493 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:22:52.862714 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851267 4493 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:22:52.862714 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851269 4493 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:22:52.863489 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851272 4493 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:22:52.863489 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851274 4493 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:22:52.863489 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851277 4493 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:22:52.863489 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851279 4493 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:22:52.863489 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851282 4493 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:22:52.863489 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851284 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:22:52.863489 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851287 4493 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:22:52.863489 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851289 4493 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:22:52.863489 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851292 4493 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:22:52.863489 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851294 4493 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:22:52.863489 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851297 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:22:52.863489 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851299 4493 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:22:52.863489 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851302 4493 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:22:52.863489 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851305 4493 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:22:52.863489 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851308 4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:22:52.863489 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851310 4493 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:22:52.863489 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851313 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:22:52.863489 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851315 4493 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:22:52.863489 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851318 4493 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:22:52.863489 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851320 4493 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:22:52.864307 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851322 4493 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:22:52.864307 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851325 4493 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:22:52.864307 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851327 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:22:52.864307 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851330 4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:22:52.864307 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851332 4493 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:22:52.864307 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851335 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:22:52.864307 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851337 4493 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:22:52.864307 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851340 4493 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:22:52.864307 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851343 4493 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:22:52.864307 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851345 4493 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:22:52.864307 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851349 4493 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:22:52.864307 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851353 4493 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:22:52.864307 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851358 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:22:52.864307 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851361 4493 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:22:52.864307 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851363 4493 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:22:52.864307 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851366 4493 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:22:52.864950 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.851393 4493 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 16:22:52.864950 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851568 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:22:52.864950 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851573 4493 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:22:52.864950 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851576 4493 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:22:52.864950 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851578 4493 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:22:52.864950 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851581 4493 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:22:52.864950 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851584 4493 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:22:52.864950 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851587 4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:22:52.864950 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851589 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:22:52.864950 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851592 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:22:52.864950 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851594 4493 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:22:52.864950 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851597 4493 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:22:52.864950 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851599 4493 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:22:52.864950 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851602 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:22:52.864950 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851604 4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:22:52.864950 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851607 4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:22:52.865551 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851609 4493 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:22:52.865551 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851612 4493 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:22:52.865551 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851614 4493 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:22:52.865551 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851617 4493 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:22:52.865551 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851619 4493 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:22:52.865551 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851622 4493 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:22:52.865551 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851630 4493 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:22:52.865551 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851633 4493 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:22:52.865551 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851635 4493 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:22:52.865551 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851638 4493 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:22:52.865551 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851640 4493 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:22:52.865551 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851643 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:22:52.865551 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851645 4493 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:22:52.865551 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851648 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:22:52.865551 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851650 4493 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:22:52.865551 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851653 4493 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:22:52.865551 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851655 4493 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:22:52.865551 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851658 4493 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:22:52.865551 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851660 4493 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:22:52.865551 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851663 4493 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:22:52.866434 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851665 4493 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:22:52.866434 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851668 4493 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:22:52.866434 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851671 4493 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:22:52.866434 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851673 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:22:52.866434 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851675 4493 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:22:52.866434 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851678 4493 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:22:52.866434 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851680 4493 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:22:52.866434 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851686 4493 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:22:52.866434 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851689 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:22:52.866434 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851691 4493 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:22:52.866434 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851694 4493 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:22:52.866434 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851696 4493 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:22:52.866434 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851698 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:22:52.866434 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851701 4493 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:22:52.866434 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851703 4493 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:22:52.866434 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851705 4493 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:22:52.866434 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851708 4493 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:22:52.866434 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851711 4493 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:22:52.866434 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851713 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:22:52.867135 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851722 4493 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:22:52.867135 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851725 4493 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:22:52.867135 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851727 4493 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:22:52.867135 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851729 4493 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:22:52.867135 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851732 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:22:52.867135 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851734 4493 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:22:52.867135 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851738 4493 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:22:52.867135 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851742 4493 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:22:52.867135 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851745 4493 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:22:52.867135 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851749 4493 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:22:52.867135 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851753 4493 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:22:52.867135 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851755 4493 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:22:52.867135 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851758 4493 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:22:52.867135 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851760 4493 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:22:52.867135 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851764 4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:22:52.867135 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851766 4493 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:22:52.867135 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851769 4493 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:22:52.867135 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851771 4493 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:22:52.867135 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851774 4493 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:22:52.867694 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851776 4493 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:22:52.867694 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851779 4493 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:22:52.867694 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851794 4493 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:22:52.867694 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851797 4493 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:22:52.867694 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851800 4493 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:22:52.867694 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851802 4493 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:22:52.867694 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851805 4493 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:22:52.867694 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851807 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:22:52.867694 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851810 4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:22:52.867694 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851812 4493 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:22:52.867694 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851814 4493 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:22:52.867694 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851817 4493 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:22:52.867694 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851820 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:22:52.867694 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851822 4493 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:22:52.867694 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851830 4493 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:22:52.867694 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851833 4493 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:22:52.867694 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851836 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:22:52.867694 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851838 4493 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:22:52.867694 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851840 4493 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:22:52.868265 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851843 4493 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:22:52.868265 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851847 4493 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:22:52.868265 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851850 4493 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:22:52.868265 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851852 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:22:52.868265 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851854 4493 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:22:52.868265 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851857 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:22:52.868265 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851859 4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:22:52.868265 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851864 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:22:52.868265 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851867 4493 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:22:52.868265 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851869 4493 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:22:52.868265 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851872 4493 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:22:52.868265 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851874 4493 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:22:52.868265 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851877 4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:22:52.868265 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851879 4493 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:22:52.868265 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.851902 4493 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 16:22:52.868265 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851940 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:22:52.868706 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851944 4493 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:22:52.868706 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851946 4493 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:22:52.868706 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851949 4493 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:22:52.868706 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851951 4493 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:22:52.868706 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851954 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:22:52.868706 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851957 4493 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:22:52.868706 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851966 4493 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:22:52.868706 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851971 4493 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:22:52.868706 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851974 4493 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:22:52.868706 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851978 4493 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:22:52.868706 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851981 4493 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:22:52.868706 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851984 4493 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:22:52.868706 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851993 4493 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:22:52.868706 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851996 4493 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:22:52.868706 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.851999 4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:22:52.868706 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852002 4493 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:22:52.868706 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852004 4493 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:22:52.868706 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852008 4493 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:22:52.868706 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852011 4493 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:22:52.869244 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852013 4493 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:22:52.869244 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852016 4493 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:22:52.869244 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852018 4493 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:22:52.869244 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852021 4493 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:22:52.869244 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852023 4493 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:22:52.869244 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852027 4493 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:22:52.869244 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852029 4493 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:22:52.869244 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852032 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:22:52.869244 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852034 4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:22:52.869244 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852037 4493 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:22:52.869244 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852039 4493 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:22:52.869244 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852042 4493 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:22:52.869244 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852044 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:22:52.869244 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852047 4493 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:22:52.869244 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852050 4493 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:22:52.869244 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852052 4493 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:22:52.869244 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852055 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:22:52.869244 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852057 4493 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:22:52.869244 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852060 4493 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:22:52.869819 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852062 4493 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:22:52.869819 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852064 4493 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:22:52.869819 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852067 4493 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:22:52.869819 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852070 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:22:52.869819 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852072 4493 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:22:52.869819 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852074 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:22:52.869819 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852077 4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:22:52.869819 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852085 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:22:52.869819 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852088 4493 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:22:52.869819 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852090 4493 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:22:52.869819 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852093 4493 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:22:52.869819 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852096 4493 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:22:52.869819 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852099 4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:22:52.869819 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852102 4493 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:22:52.869819 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852104 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:22:52.869819 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852106 4493 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:22:52.869819 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852109 4493 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:22:52.869819 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852111 4493 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:22:52.869819 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852115 4493 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:22:52.869819 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852117 4493 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:22:52.870455 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852120 4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:22:52.870455 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852123 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:22:52.870455 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852125 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:22:52.870455 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852128 4493 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:22:52.870455 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852130 4493 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:22:52.870455 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852133 4493 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:22:52.870455 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852135 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:22:52.870455 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852138 4493 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:22:52.870455 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852140 4493 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:22:52.870455 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852143 4493 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:22:52.870455 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852145 4493 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:22:52.870455 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852147 4493 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:22:52.870455 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852150 4493 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:22:52.870455 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852153 4493 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:22:52.870455 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852156 4493 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:22:52.870455 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852158 4493 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:22:52.870455 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852161 4493 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:22:52.870455 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852163 4493 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:22:52.870455 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852165 4493 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:22:52.870455 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852168 4493 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:22:52.871000 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852175 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:22:52.871000 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852178 4493 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:22:52.871000 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852181 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:22:52.871000 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852184 4493 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:22:52.871000 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852187 4493 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:22:52.871000 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852189 4493 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:22:52.871000 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852192 4493 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:22:52.871000 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852194 4493 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:22:52.871000 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852196 4493 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:22:52.871000 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852199 4493 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:22:52.871000 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852203 4493 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:22:52.871000 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852206 4493 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:22:52.871000 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852208 4493 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:22:52.871000 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852211 4493 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:22:52.871000 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852213 4493 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:22:52.871000 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852216 4493 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:22:52.871000 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852218 4493 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:22:52.871000 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852220 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:22:52.871000 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852223 4493 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:22:52.871488 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852225 4493 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:22:52.871488 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852228 4493 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:22:52.871488 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852230 4493 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:22:52.871488 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852233 4493 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:22:52.871488 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852235 4493 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:22:52.871488 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852237 4493 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:22:52.871488 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852241 4493 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:22:52.871488 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:52.852243 4493 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:22:52.871488 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.852304 4493 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 16:22:52.871488 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.853293 4493 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 16:22:52.871488 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.853346 4493 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 16:22:52.871488 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.853535 4493 server.go:1014] "Starting client certificate rotation" Oct 30 16:22:52.871488 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.853716 4493 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 16:22:52.871488 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.853756 4493 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 16:17:13 +0000 UTC" deadline="2028-04-25 23:13:36.062722234 +0000 UTC" Oct 30 16:22:52.871488 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.853820 4493 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13038h50m43.208905714s" Oct 30 16:22:52.871488 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.854125 4493 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 16:22:52.871930 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.855326 4493 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 16:22:52.871930 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.859190 4493 log.go:25] "Validated CRI v1 runtime API" Oct 30 16:22:52.871930 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.865675 4493 log.go:25] "Validated CRI v1 image API" Oct 30 16:22:52.871930 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.866532 4493 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 16:22:52.871930 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.868236 4493 fs.go:135] Filesystem UUIDs: map[32fc8df6-ceea-42e8-92d2-2990ec3d6eb1:/dev/nvme0n1p4 3f22268a-26c0-4a7c-8700-7676b0d0b5bf:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Oct 30 16:22:52.872076 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.868257 4493 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1d9c953d9c05dddbe9ba30df4c18df0bd67104dbf43b27c995246f24c34cf607/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1d9c953d9c05dddbe9ba30df4c18df0bd67104dbf43b27c995246f24c34cf607/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2050322e2b6f53fddf1dd87469e9e1ff6688187934056115715470205362f52d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2050322e2b6f53fddf1dd87469e9e1ff6688187934056115715470205362f52d/userdata/shm major:0 minor:69 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/285446f717ed93d43eb8db91ddceed824babc5ca8a476aa8aeb18ee857b9381d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/285446f717ed93d43eb8db91ddceed824babc5ca8a476aa8aeb18ee857b9381d/userdata/shm major:0 minor:221 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/393260634ffef513017fae87c8817de2c1238100f6eacfe5a677b6244e5cc141/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/393260634ffef513017fae87c8817de2c1238100f6eacfe5a677b6244e5cc141/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3f1e11ba28da1d45a99744adc81ee6df04c856ee09e809cf19589bad3b33ebef/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3f1e11ba28da1d45a99744adc81ee6df04c856ee09e809cf19589bad3b33ebef/userdata/shm major:0 minor:224 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/414f44d6a75e2e5ffcc124804fba274175dca7f97a137c7980160c4d84a0d8fe/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/414f44d6a75e2e5ffcc124804fba274175dca7f97a137c7980160c4d84a0d8fe/userdata/shm major:0 minor:226 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/49a61c059f9969e90d591edc4f2c99a030d089f79e91d92054c08408ca492db3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/49a61c059f9969e90d591edc4f2c99a030d089f79e91d92054c08408ca492db3/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4bb20d60d50b5c682dbd00d98d336b88192dabc3052c9dea60bb4c802633c659/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4bb20d60d50b5c682dbd00d98d336b88192dabc3052c9dea60bb4c802633c659/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6fa32b3a14c77ca3475739bb9a49d813fb5e7bcb2e4a30439e74f1ee02956b21/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6fa32b3a14c77ca3475739bb9a49d813fb5e7bcb2e4a30439e74f1ee02956b21/userdata/shm major:0 minor:214 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/83b1759279a811cd0f4c435481c00f17690c838c2c5b49eb63647a502c3cc295/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/83b1759279a811cd0f4c435481c00f17690c838c2c5b49eb63647a502c3cc295/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8b07c6de4cf29ea898ddc5c111d581be149d4fc1cbc734a4ff91f705c3cc8de0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8b07c6de4cf29ea898ddc5c111d581be149d4fc1cbc734a4ff91f705c3cc8de0/userdata/shm major:0 minor:85 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b50e9c4749aef7c936214753593f845f779379b915ae058f85a9434ba66e39bc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b50e9c4749aef7c936214753593f845f779379b915ae058f85a9434ba66e39bc/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c45d537cf983095d6f7f6e59329996acbabab075cf91370b12a541c56665f537/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c45d537cf983095d6f7f6e59329996acbabab075cf91370b12a541c56665f537/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ec42a59d26b309a2abc07c1f01503b90314473de1dca75a3ae30db9221dcb9dc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ec42a59d26b309a2abc07c1f01503b90314473de1dca75a3ae30db9221dcb9dc/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fb8fe2ee28869222ba653799b2b57a388e05e8a844ed94e8609eda7f5308e961/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fb8fe2ee28869222ba653799b2b57a388e05e8a844ed94e8609eda7f5308e961/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e649944-a09d-44b1-91d2-fda00b30292d/volumes/kubernetes.io~projected/kube-api-access-s64j8:{mountpoint:/var/lib/kubelet/pods/0e649944-a09d-44b1-91d2-fda00b30292d/volumes/kubernetes.io~projected/kube-api-access-s64j8 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/17bb82ba-5262-47f7-abbc-99f1329500fa/volumes/kubernetes.io~projected/kube-api-access-vxrxb:{mountpoint:/var/lib/kubelet/pods/17bb82ba-5262-47f7-abbc-99f1329500fa/volumes/kubernetes.io~projected/kube-api-access-vxrxb major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/21fd140d-96ea-462e-9f8b-ef6d480f8407/volumes/kubernetes.io~projected/kube-api-access-5hgtv:{mountpoint:/var/lib/kubelet/pods/21fd140d-96ea-462e-9f8b-ef6d480f8407/volumes/kubernetes.io~projected/kube-api-access-5hgtv major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/315c49d6-d546-4fc2-a5c0-3b987269e5a9/volumes/kubernetes.io~projected/kube-api-access-ntggj:{mountpoint:/var/lib/kubelet/pods/315c49d6-d546-4fc2-a5c0-3b987269e5a9/volumes/kubernetes.io~projected/kube-api-access-ntggj major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/315c49d6-d546-4fc2-a5c0-3b987269e5a9/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/315c49d6-d546-4fc2-a5c0-3b987269e5a9/volumes/kubernetes.io~secret/signing-key major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/394d816a-1e33-433f-abb1-586089620763/volumes/kubernetes.io~projected/kube-api-access-nb9mk:{mountpoint:/var/lib/kubelet/pods/394d816a-1e33-433f-abb1-586089620763/volumes/kubernetes.io~projected/kube-api-access-nb9mk major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3a773f3a-800c-4c84-b18f-7a48cdd92692/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/3a773f3a-800c-4c84-b18f-7a48cdd92692/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3a773f3a-800c-4c84-b18f-7a48cdd92692/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/3a773f3a-800c-4c84-b18f-7a48cdd92692/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3a773f3a-800c-4c84-b18f-7a48cdd92692/volumes/kubernetes.io~projected/kube-api-access-m5v7j:{mountpoint:/var/lib/kubelet/pods/3a773f3a-800c-4c84-b18f-7a48cdd92692/volumes/kubernetes.io~projected/kube-api-access-m5v7j major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/78b9342d-5361-4a8c-a771-0c1aea26a48d/volumes/kubernetes.io~projected/kube-api-access-srfcg:{mountpoint:/var/lib/kubelet/pods/78b9342d-5361-4a8c-a771-0c1aea26a48d/volumes/kubernetes.io~projected/kube-api-access-srfcg major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7c09760d-2160-41d4-b991-f6324ad5aff7/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/7c09760d-2160-41d4-b991-f6324ad5aff7/volumes/kubernetes.io~secret/agent-certs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc/volumes/kubernetes.io~projected/kube-api-access-f99rt:{mountpoint:/var/lib/kubelet/pods/7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc/volumes/kubernetes.io~projected/kube-api-access-f99rt major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a7b4cd2-b289-42b0-a54b-d868de87d7d3/volumes/kubernetes.io~projected/kube-api-access-8fdbc:{mountpoint:/var/lib/kubelet/pods/8a7b4cd2-b289-42b0-a54b-d868de87d7d3/volumes/kubernetes.io~projected/kube-api-access-8fdbc major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a7b4cd2-b289-42b0-a54b-d868de87d7d3/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/8a7b4cd2-b289-42b0-a54b-d868de87d7d3/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c/volumes/kubernetes.io~projected/kube-api-access-5j5vz:{mountpoint:/var/lib/kubelet/pods/8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c/volumes/kubernetes.io~projected/kube-api-access-5j5vz major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e02b2b2-c05c-4821-adfe-02f2a6664267/volumes/kubernetes.io~projected/kube-api-access-m78c4:{mountpoint:/var/lib/kubelet/pods/8e02b2b2-c05c-4821-adfe-02f2a6664267/volumes/kubernetes.io~projected/kube-api-access-m78c4 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b804fc37-0b07-4edf-8720-b31b5698b935/volumes/kubernetes.io~projected/kube-api-access-m8vll:{mountpoint:/var/lib/kubelet/pods/b804fc37-0b07-4edf-8720-b31b5698b935/volumes/kubernetes.io~projected/kube-api-access-m8vll major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c407642e-2f4a-4c47-906e-56137857d162/volumes/kubernetes.io~projected/kube-api-access-zftsq:{mountpoint:/var/lib/kubelet/pods/c407642e-2f4a-4c47-906e-56137857d162/volumes/kubernetes.io~projected/kube-api-access-zftsq major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d3a1c417-2698-4462-b7ff-1d2e66d7a10f/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/d3a1c417-2698-4462-b7ff-1d2e66d7a10f/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d3a1c417-2698-4462-b7ff-1d2e66d7a10f/volumes/kubernetes.io~projected/kube-api-access-8vrcg:{mountpoint:/var/lib/kubelet/pods/d3a1c417-2698-4462-b7ff-1d2e66d7a10f/volumes/kubernetes.io~projected/kube-api-access-8vrcg major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d3a1c417-2698-4462-b7ff-1d2e66d7a10f/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/d3a1c417-2698-4462-b7ff-1d2e66d7a10f/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc77c50f-abe0-4d9c-a88b-7a186c17dc3e/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/dc77c50f-abe0-4d9c-a88b-7a186c17dc3e/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fe9cbcfe-25a7-4da2-bd0d-cb313273c846/volumes/kubernetes.io~projected/kube-api-access-6dlwk:{mountpoint:/var/lib/kubelet/pods/fe9cbcfe-25a7-4da2-bd0d-cb313273c846/volumes/kubernetes.io~projected/kube-api-access-6dlwk major:0 minor:67 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/c2113edd5768351993b79ac948206463730e976cff52865422ee83ec3d322835/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/c8e39039ef952ccf7e5c022ea354c8dbe5c4194285c71a93f6595ad223f412f9/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/7955f1e7d92c68f56c33ae383ce96ee5db12b6debb2258cf68c7ab343ce58ef0/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/df107eb0c912d4280de44780ffea2491e132f2c0dd861f28309a0d1bcf1c02fe/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/244c516b8a304f9141c3f484f0adfc92d76bc1cb287820ac28dbdf1beb6d3260/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/d3cdff7ce6711c2aa73f80d60306465a3e5773803e0a967075438fc583fc2de2/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/3e191c9ba82a7bfaf8af923e46b62d4704c5f32be6757597a49a3fa4e5910f95/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/2419168ae6b6a7526600eb1c71015aa11fe2d0725cbc3e629f28b916515cecb8/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/09252b408151b9ad29ced6ec21d51679a5c9c39e9a3b7b0388cd2d161834a7c2/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/e8ab59d40902ef6e3e9eefe4d6921c90a411a37ec2c8b8f14f65767069d64bb8/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/5d7e85c45024a232c9275d92b8579920a988bfa3ecc7b76219adc2ea3834f6e7/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/1f31b8ac4af4f81170eff56344c69eb599daef80156ec71fd9f77689577559cf/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/c9a5990a346827d4c3ff645bd6cf5c34f1b5208dc44452891c34c25f1e351481/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/a9bc123e04b919678847cef8e9cf74a0b197f2d8bbf57ddd4949df6ea1d46274/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/e23c8eb1acda39d3b0c1bc002003ba96c2c7d4f9bd62d9e582070084ac4f6fea/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/7136f151617e2dfb6e207bf43828357564dbb254a5e059829dceb3180cee454b/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/c0e1bb74bf39479764a104deb9b7301a1d138c34f9c8a2dd099aa574a02317b1/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/6b413adc8c3db805851b8e219f944791325df9c125345d8089464d6e73ca2ad0/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/e4352dbdfc23c72794fe76c32b960e61f190024b552b35c31becf35316b973e6/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/247eb5935830ac72cd093f18a58386727f9f2e57227cc1af96f436f3b928dfd3/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/f70943ea055cc50e2b1c97865da30062ff60f0969cdd202afad9d38adc673758/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/9889ad6f92160435f572accb2b34a9c255b3fca9b0955d744a4c0fb216b2184e/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-207:{mountpoint:/var/lib/containers/storage/overlay/381976e263355e9418c73e595ad2c4816fb7dda7105bdb89620c3c4aa820db59/merged major:0 minor:207 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/0d0909f2738dbd29f18b2db1c148c5bd881406a88e7670406288cee3e5c7a68b/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-228:{mountpoint:/var/lib/containers/storage/overlay/76ff4eefef9c470cd8f1b6733a8411d2262feeb80ef331fffddbfcbfb0c99d7e/merged major:0 minor:228 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/db3833a95ad779265a53f88ea3e2abd62e481a16c69e8df6e34769b40e4ceba6/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-232:{mountpoint:/var/lib/containers/storage/overlay/a234cd489cd3a190a6a24a154564fc44c14f7bc91b959550829af824a35bd10c/merged major:0 minor:232 fsType:overlay blockSize:0} overlay_0-234:{mountpoint:/var/lib/containers/storage/overlay/c5be77aaa151572e0301d16d33c30a29baab79ffac8e8f991e5c7538830c4d80/merged major:0 minor:234 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/5e59d8804be661f5c1f7680911bbaed30e88de5b88a3c4b197938af926d06b9a/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-242:{mountpoint:/var/lib/containers/storage/overlay/d94a131645102802e8177e4eaced4395d24897ae2c78049c127bb8335a64493f/merged major:0 minor:242 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/7aac61641b17a51d6d20b6788e6649b472d24884cdd69a57dc07b107fb1077d0/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/863da8ae09a56f4b1fba627b2443bb527e5701c382ef00a45cb4ef50ce8a1929/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/d0add26b98771821779c54e8d0c901e9c2d131b234f699693242d2549db808e2/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-71:{mountpoint:/var/lib/containers/storage/overlay/c49d94e569807b841fea99438c371e5a566abf26828387e7d2c68de8b3e00d30/merged major:0 minor:71 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/332b7012e2c909720c24cec470e49621e8203d12206ef4efb4a70cc00e5474cc/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/265c2b7f25e0e59a869e3d8712c62920ca1fc9bf50a33254e0839eed7cc094c3/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/a620dd3832c3d8766fb907bd16204fc57dcf674335df2addb37299e2e3bea423/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/27f610e660824e5007402a47f7763337c61517a3f3455adcce5f26c19d7b64aa/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/6102e04e38591a756b032c88b873feac5bea0ca76b2aa465cdf5fdb77cfe6fcb/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/ca2fa0c1658e012d3a0848b43971e75596f484c683affb2b0dbfcf027187badf/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/df00c09956cfb784e1b1f51b030a8c6abbbb0e95ca3bf127b667bda7bce28317/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 16:22:52.874887 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.874618 4493 manager.go:217] Machine: {Timestamp:2026-10-30 16:22:52.874162154 +0000 UTC m=+0.101337246 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099203 MemoryCapacity:32812023808 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec273da32cfe0dda36f4161300eb9e18 SystemUUID:ec273da3-2cfe-0dda-36f4-161300eb9e18 BootID:b5e11be0-d6ef-4d05-8d25-f802e219ec89 Filesystems:[{Device:/run/containers/storage/overlay-containers/c45d537cf983095d6f7f6e59329996acbabab075cf91370b12a541c56665f537/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:/run/containers/storage/overlay-containers/49a61c059f9969e90d591edc4f2c99a030d089f79e91d92054c08408ca492db3/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/315c49d6-d546-4fc2-a5c0-3b987269e5a9/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:219 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/3a773f3a-800c-4c84-b18f-7a48cdd92692/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/fe9cbcfe-25a7-4da2-bd0d-cb313273c846/volumes/kubernetes.io~projected/kube-api-access-6dlwk DeviceMajor:0 DeviceMinor:67 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2050322e2b6f53fddf1dd87469e9e1ff6688187934056115715470205362f52d/userdata/shm DeviceMajor:0 DeviceMinor:69 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-242 DeviceMajor:0 DeviceMinor:242 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3a773f3a-800c-4c84-b18f-7a48cdd92692/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/c407642e-2f4a-4c47-906e-56137857d162/volumes/kubernetes.io~projected/kube-api-access-zftsq DeviceMajor:0 DeviceMinor:59 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-71 DeviceMajor:0 DeviceMinor:71 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d3a1c417-2698-4462-b7ff-1d2e66d7a10f/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d3a1c417-2698-4462-b7ff-1d2e66d7a10f/volumes/kubernetes.io~projected/kube-api-access-8vrcg DeviceMajor:0 DeviceMinor:60 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fb8fe2ee28869222ba653799b2b57a388e05e8a844ed94e8609eda7f5308e961/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8a7b4cd2-b289-42b0-a54b-d868de87d7d3/volumes/kubernetes.io~projected/kube-api-access-8fdbc DeviceMajor:0 DeviceMinor:201 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-207 DeviceMajor:0 DeviceMinor:207 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3a773f3a-800c-4c84-b18f-7a48cdd92692/volumes/kubernetes.io~projected/kube-api-access-m5v7j DeviceMajor:0 DeviceMinor:61 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/run/containers/storage/overlay-containers/83b1759279a811cd0f4c435481c00f17690c838c2c5b49eb63647a502c3cc295/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:/run/containers/storage/overlay-containers/393260634ffef513017fae87c8817de2c1238100f6eacfe5a677b6244e5cc141/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc/volumes/kubernetes.io~projected/kube-api-access-f99rt DeviceMajor:0 DeviceMinor:197 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/78b9342d-5361-4a8c-a771-0c1aea26a48d/volumes/kubernetes.io~projected/kube-api-access-srfcg DeviceMajor:0 DeviceMinor:223 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/run/containers/storage/overlay-containers/414f44d6a75e2e5ffcc124804fba274175dca7f97a137c7980160c4d84a0d8fe/userdata/shm DeviceMajor:0 DeviceMinor:226 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406011904 Type:vfs Inodes:4005374 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ec42a59d26b309a2abc07c1f01503b90314473de1dca75a3ae30db9221dcb9dc/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c/volumes/kubernetes.io~projected/kube-api-access-5j5vz DeviceMajor:0 DeviceMinor:217 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/dc77c50f-abe0-4d9c-a88b-7a186c17dc3e/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:218 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/21fd140d-96ea-462e-9f8b-ef6d480f8407/volumes/kubernetes.io~projected/kube-api-access-5hgtv DeviceMajor:0 DeviceMinor:206 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-228 DeviceMajor:0 DeviceMinor:228 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/17bb82ba-5262-47f7-abbc-99f1329500fa/volumes/kubernetes.io~projected/kube-api-access-vxrxb DeviceMajor:0 DeviceMinor:63 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-234 DeviceMajor:0 DeviceMinor:234 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/394d816a-1e33-433f-abb1-586089620763/volumes/kubernetes.io~projected/kube-api-access-nb9mk DeviceMajor:0 DeviceMinor:66 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/315c49d6-d546-4fc2-a5c0-3b987269e5a9/volumes/kubernetes.io~projected/kube-api-access-ntggj DeviceMajor:0 DeviceMinor:220 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8b07c6de4cf29ea898ddc5c111d581be149d4fc1cbc734a4ff91f705c3cc8de0/userdata/shm DeviceMajor:0 DeviceMinor:85 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3f1e11ba28da1d45a99744adc81ee6df04c856ee09e809cf19589bad3b33ebef/userdata/shm DeviceMajor:0 DeviceMinor:224 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/7c09760d-2160-41d4-b991-f6324ad5aff7/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:58 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/0e649944-a09d-44b1-91d2-fda00b30292d/volumes/kubernetes.io~projected/kube-api-access-s64j8 DeviceMajor:0 DeviceMinor:62 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/b804fc37-0b07-4edf-8720-b31b5698b935/volumes/kubernetes.io~projected/kube-api-access-m8vll DeviceMajor:0 DeviceMinor:64 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b50e9c4749aef7c936214753593f845f779379b915ae058f85a9434ba66e39bc/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6fa32b3a14c77ca3475739bb9a49d813fb5e7bcb2e4a30439e74f1ee02956b21/userdata/shm DeviceMajor:0 DeviceMinor:214 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1d9c953d9c05dddbe9ba30df4c18df0bd67104dbf43b27c995246f24c34cf607/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:/run/containers/storage/overlay-containers/285446f717ed93d43eb8db91ddceed824babc5ca8a476aa8aeb18ee857b9381d/userdata/shm DeviceMajor:0 DeviceMinor:221 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-232 DeviceMajor:0 DeviceMinor:232 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d3a1c417-2698-4462-b7ff-1d2e66d7a10f/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:57 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:/var/lib/kubelet/pods/8e02b2b2-c05c-4821-adfe-02f2a6664267/volumes/kubernetes.io~projected/kube-api-access-m78c4 DeviceMajor:0 DeviceMinor:65 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8a7b4cd2-b289-42b0-a54b-d868de87d7d3/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31633424384 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4bb20d60d50b5c682dbd00d98d336b88192dabc3052c9dea60bb4c802633c659/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4005374 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:285446f717ed93d MacAddress:fa:a0:54:d6:e0:f2 Speed:10000 Mtu:8901} {Name:3f1e11ba28da1d4 MacAddress:b6:f2:86:8f:f2:14 Speed:10000 Mtu:8901} {Name:414f44d6a75e2e5 MacAddress:4a:f1:c8:f1:60:42 Speed:10000 Mtu:8901} {Name:4bb20d60d50b5c6 MacAddress:a2:14:47:ed:ad:7c Speed:10000 Mtu:8901} {Name:6fa32b3a14c77ca MacAddress:fa:8a:68:b1:c3:56 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:d9:ac:ba:d1 Speed:0 Mtu:9001} {Name:br-int MacAddress:22:60:2c:63:15:c1 Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:ff:d9:ac:ba:d1 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:f2:06:cb:1a:6c:68 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:16:f6:36:8a:2d:7a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812023808 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 16:22:52.875051 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.874891 4493 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 16:22:52.875051 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.874950 4493 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 16:22:52.875227 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.875206 4493 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 16:22:52.875356 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.875228 4493 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-180.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 16:22:52.875402 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.875359 4493 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 16:22:52.875402 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.875366 4493 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 16:22:52.875402 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.875379 4493 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 16:22:52.875402 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.875392 4493 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 16:22:52.875506 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.875473 4493 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 16:22:52.875642 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.875632 4493 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 16:22:52.875762 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.875753 4493 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 16:22:52.875815 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.875805 4493 kubelet.go:493] "Attempting to sync node with API server" Oct 30 16:22:52.875845 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.875819 4493 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 16:22:52.875845 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.875829 4493 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 16:22:52.875845 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.875838 4493 kubelet.go:405] "Adding apiserver pod source" Oct 30 16:22:52.875954 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.875851 4493 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 16:22:52.876861 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.876837 4493 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 16:22:52.876861 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.876854 4493 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 16:22:52.877517 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.877491 4493 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 16:22:52.880998 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.880954 4493 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 16:22:52.880998 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.880993 4493 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 16:22:52.881138 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.881077 4493 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 16:22:52.881280 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.881266 4493 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 16:22:52.881334 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.881296 4493 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 16:22:52.881388 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.881382 4493 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 16:22:52.881434 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.881394 4493 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 16:22:52.881434 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.881401 4493 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 16:22:52.881434 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.881406 4493 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 16:22:52.881434 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.881411 4493 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 16:22:52.881434 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.881417 4493 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 16:22:52.881434 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.881423 4493 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 16:22:52.881434 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.881428 4493 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 16:22:52.881434 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.881434 4493 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 16:22:52.881710 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.881440 4493 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 16:22:52.881710 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.881448 4493 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 16:22:52.881710 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.881457 4493 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 16:22:52.881710 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.881478 4493 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 16:22:52.881710 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.881484 4493 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 16:22:52.884774 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.884761 4493 server.go:1263] "Started kubelet" Oct 30 16:22:52.884864 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.884839 4493 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 16:22:52.885666 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.884923 4493 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 16:22:52.885666 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.884997 4493 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 16:22:52.885666 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.885453 4493 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 16:22:52.885300 ip-10-0-132-180 systemd[1]: Started Kubernetes Kubelet. Oct 30 16:22:52.886704 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.886680 4493 server.go:317] "Adding debug handlers to kubelet server" Oct 30 16:22:52.887840 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.887826 4493 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 16:22:52.890415 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.888160 4493 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 16:22:52.890415 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.888178 4493 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 16:22:52.890415 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.888174 4493 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 16:22:52.890415 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:22:52.888329 4493 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-180.ec2.internal\" not found" Oct 30 16:22:52.890415 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.888358 4493 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 16:22:52.890415 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.888389 4493 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 16:17:15 +0000 UTC" deadline="2028-05-12 02:53:38.873757951 +0000 UTC" Oct 30 16:22:52.890415 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.888424 4493 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13426h30m45.985337689s" Oct 30 16:22:52.890415 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.889769 4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a773f3a-800c-4c84-b18f-7a48cdd92692" volumeName="kubernetes.io/empty-dir/3a773f3a-800c-4c84-b18f-7a48cdd92692-tmp" seLinuxMountContext="" Oct 30 16:22:52.890415 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.889826 4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78b9342d-5361-4a8c-a771-0c1aea26a48d" volumeName="kubernetes.io/projected/78b9342d-5361-4a8c-a771-0c1aea26a48d-kube-api-access-srfcg" seLinuxMountContext="" Oct 30 16:22:52.890415 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.889836 4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e02b2b2-c05c-4821-adfe-02f2a6664267" volumeName="kubernetes.io/configmap/8e02b2b2-c05c-4821-adfe-02f2a6664267-cni-binary-copy" seLinuxMountContext="" Oct 30 16:22:52.890415 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.889844 4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c407642e-2f4a-4c47-906e-56137857d162" volumeName="kubernetes.io/configmap/c407642e-2f4a-4c47-906e-56137857d162-iptables-alerter-script" seLinuxMountContext="" Oct 30 16:22:52.890415 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.889868 4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17bb82ba-5262-47f7-abbc-99f1329500fa" volumeName="kubernetes.io/configmap/17bb82ba-5262-47f7-abbc-99f1329500fa-serviceca" seLinuxMountContext="" Oct 30 16:22:52.890415 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.889875 4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc" volumeName="kubernetes.io/configmap/7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc-config-volume" seLinuxMountContext="" Oct 30 16:22:52.890415 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.889882 4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c407642e-2f4a-4c47-906e-56137857d162" volumeName="kubernetes.io/projected/c407642e-2f4a-4c47-906e-56137857d162-kube-api-access-zftsq" seLinuxMountContext="" Oct 30 16:22:52.890415 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.889892 4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe9cbcfe-25a7-4da2-bd0d-cb313273c846" volumeName="kubernetes.io/configmap/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-cni-binary-copy" seLinuxMountContext="" Oct 30 16:22:52.890415 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.889901 4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a773f3a-800c-4c84-b18f-7a48cdd92692" volumeName="kubernetes.io/empty-dir/3a773f3a-800c-4c84-b18f-7a48cdd92692-etc-tuned" seLinuxMountContext="" Oct 30 16:22:52.890415 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.889913 4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c" volumeName="kubernetes.io/projected/8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c-kube-api-access-5j5vz" seLinuxMountContext="" Oct 30 16:22:52.890415 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.889939 4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3a1c417-2698-4462-b7ff-1d2e66d7a10f" volumeName="kubernetes.io/configmap/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-ovnkube-script-lib" seLinuxMountContext="" Oct 30 16:22:52.890415 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.889946 4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3a1c417-2698-4462-b7ff-1d2e66d7a10f" volumeName="kubernetes.io/configmap/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-ovnkube-config" seLinuxMountContext="" Oct 30 16:22:52.890415 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.889955 4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21fd140d-96ea-462e-9f8b-ef6d480f8407" volumeName="kubernetes.io/projected/21fd140d-96ea-462e-9f8b-ef6d480f8407-kube-api-access-5hgtv" seLinuxMountContext="" Oct 30 16:22:52.890415 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.889969 4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="315c49d6-d546-4fc2-a5c0-3b987269e5a9" volumeName="kubernetes.io/secret/315c49d6-d546-4fc2-a5c0-3b987269e5a9-signing-key" seLinuxMountContext="" Oct 30 16:22:52.890415 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.889978 4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e02b2b2-c05c-4821-adfe-02f2a6664267" volumeName="kubernetes.io/configmap/8e02b2b2-c05c-4821-adfe-02f2a6664267-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 16:22:52.890415 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.889984 4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3a1c417-2698-4462-b7ff-1d2e66d7a10f" volumeName="kubernetes.io/configmap/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-env-overrides" seLinuxMountContext="" Oct 30 16:22:52.890415 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.889993 4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe9cbcfe-25a7-4da2-bd0d-cb313273c846" volumeName="kubernetes.io/configmap/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-multus-daemon-config" seLinuxMountContext="" Oct 30 16:22:52.890415 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.890000 4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17bb82ba-5262-47f7-abbc-99f1329500fa" volumeName="kubernetes.io/projected/17bb82ba-5262-47f7-abbc-99f1329500fa-kube-api-access-vxrxb" seLinuxMountContext="" Oct 30 16:22:52.890415 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.890007 4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a7b4cd2-b289-42b0-a54b-d868de87d7d3" volumeName="kubernetes.io/secret/8a7b4cd2-b289-42b0-a54b-d868de87d7d3-cert" seLinuxMountContext="" Oct 30 16:22:52.890415 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.890015 4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c" volumeName="kubernetes.io/empty-dir/8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c-data-volume" seLinuxMountContext="" Oct 30 16:22:52.890415 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.890023 4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e02b2b2-c05c-4821-adfe-02f2a6664267" volumeName="kubernetes.io/projected/8e02b2b2-c05c-4821-adfe-02f2a6664267-kube-api-access-m78c4" seLinuxMountContext="" Oct 30 16:22:52.890415 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.890031 4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3a1c417-2698-4462-b7ff-1d2e66d7a10f" volumeName="kubernetes.io/secret/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 16:22:52.890415 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.890042 4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc77c50f-abe0-4d9c-a88b-7a186c17dc3e" volumeName="kubernetes.io/secret/dc77c50f-abe0-4d9c-a88b-7a186c17dc3e-original-pull-secret" seLinuxMountContext="" Oct 30 16:22:52.890415 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.890049 4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e649944-a09d-44b1-91d2-fda00b30292d" volumeName="kubernetes.io/empty-dir/0e649944-a09d-44b1-91d2-fda00b30292d-tmp-dir" seLinuxMountContext="" Oct 30 16:22:52.890415 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.890061 4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="315c49d6-d546-4fc2-a5c0-3b987269e5a9" volumeName="kubernetes.io/configmap/315c49d6-d546-4fc2-a5c0-3b987269e5a9-signing-cabundle" seLinuxMountContext="" Oct 30 16:22:52.890415 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.890075 4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="394d816a-1e33-433f-abb1-586089620763" volumeName="kubernetes.io/projected/394d816a-1e33-433f-abb1-586089620763-kube-api-access-nb9mk" seLinuxMountContext="" Oct 30 16:22:52.890415 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.890083 4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a773f3a-800c-4c84-b18f-7a48cdd92692" volumeName="kubernetes.io/projected/3a773f3a-800c-4c84-b18f-7a48cdd92692-kube-api-access-m5v7j" seLinuxMountContext="" Oct 30 16:22:52.890415 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.890089 4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc" volumeName="kubernetes.io/projected/7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc-kube-api-access-f99rt" seLinuxMountContext="" Oct 30 16:22:52.890415 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.890105 4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3a1c417-2698-4462-b7ff-1d2e66d7a10f" volumeName="kubernetes.io/projected/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-kube-api-access-8vrcg" seLinuxMountContext="" Oct 30 16:22:52.890415 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.890113 4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="315c49d6-d546-4fc2-a5c0-3b987269e5a9" volumeName="kubernetes.io/projected/315c49d6-d546-4fc2-a5c0-3b987269e5a9-kube-api-access-ntggj" seLinuxMountContext="" Oct 30 16:22:52.890415 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.890120 4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c09760d-2160-41d4-b991-f6324ad5aff7" volumeName="kubernetes.io/configmap/7c09760d-2160-41d4-b991-f6324ad5aff7-konnectivity-ca" seLinuxMountContext="" Oct 30 16:22:52.890415 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.890127 4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b804fc37-0b07-4edf-8720-b31b5698b935" volumeName="kubernetes.io/projected/b804fc37-0b07-4edf-8720-b31b5698b935-kube-api-access-m8vll" seLinuxMountContext="" Oct 30 16:22:52.890415 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.890133 4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe9cbcfe-25a7-4da2-bd0d-cb313273c846" volumeName="kubernetes.io/projected/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-kube-api-access-6dlwk" seLinuxMountContext="" Oct 30 16:22:52.890415 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.890140 4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e649944-a09d-44b1-91d2-fda00b30292d" volumeName="kubernetes.io/projected/0e649944-a09d-44b1-91d2-fda00b30292d-kube-api-access-s64j8" seLinuxMountContext="" Oct 30 16:22:52.890415 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.890147 4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c09760d-2160-41d4-b991-f6324ad5aff7" volumeName="kubernetes.io/secret/7c09760d-2160-41d4-b991-f6324ad5aff7-agent-certs" seLinuxMountContext="" Oct 30 16:22:52.890415 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.890155 4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc" volumeName="kubernetes.io/empty-dir/7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc-tmp-dir" seLinuxMountContext="" Oct 30 16:22:52.890415 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.890167 4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a7b4cd2-b289-42b0-a54b-d868de87d7d3" volumeName="kubernetes.io/projected/8a7b4cd2-b289-42b0-a54b-d868de87d7d3-kube-api-access-8fdbc" seLinuxMountContext="" Oct 30 16:22:52.890415 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.890174 4493 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e02b2b2-c05c-4821-adfe-02f2a6664267" volumeName="kubernetes.io/configmap/8e02b2b2-c05c-4821-adfe-02f2a6664267-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 16:22:52.890415 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.890184 4493 reconstruct.go:97] "Volume reconstruction finished" Oct 30 16:22:52.890415 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.890191 4493 reconciler.go:29] "Reconciler: start to sync state" Oct 30 16:22:52.898025 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.898007 4493 factory.go:55] Registering systemd factory Oct 30 16:22:52.898025 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.898028 4493 factory.go:223] Registration of the systemd container factory successfully Oct 30 16:22:52.898687 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.898668 4493 factory.go:153] Registering CRI-O factory Oct 30 16:22:52.898687 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.898686 4493 factory.go:223] Registration of the crio container factory successfully Oct 30 16:22:52.898841 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.898735 4493 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 16:22:52.898841 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.898752 4493 factory.go:103] Registering Raw factory Oct 30 16:22:52.898841 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.898766 4493 manager.go:1199] Started watching for new ooms in manager Oct 30 16:22:52.899365 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.899346 4493 manager.go:319] Starting recovery of all containers Oct 30 16:22:52.916051 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.916029 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-56v9s_d3a1c417-2698-4462-b7ff-1d2e66d7a10f/ovn-acl-logging/0.log" Oct 30 16:22:52.916141 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.916080 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-56v9s_d3a1c417-2698-4462-b7ff-1d2e66d7a10f/ovn-acl-logging/0.log" Oct 30 16:22:52.923410 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.923380 4493 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 16:22:52.924937 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.924921 4493 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 16:22:52.924937 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.924939 4493 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 16:22:52.925055 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.924953 4493 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 16:22:52.925055 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:22:52.924983 4493 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 16:22:52.935360 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.935312 4493 generic.go:356] "Generic (PLEG): container finished" podID="8e02b2b2-c05c-4821-adfe-02f2a6664267" containerID="221492ecc01a9080632eaa800c0b5ffa8b5cac3e45879b2e5a9c8d7a59667968" exitCode=0 Oct 30 16:22:52.935360 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.935360 4493 generic.go:356] "Generic (PLEG): container finished" podID="8e02b2b2-c05c-4821-adfe-02f2a6664267" containerID="4ab11267cb789025c62198ca6a265c3534cf1701c3298f9837fd7dbaad8c8cf6" exitCode=0 Oct 30 16:22:52.935511 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.935370 4493 generic.go:356] "Generic (PLEG): container finished" podID="8e02b2b2-c05c-4821-adfe-02f2a6664267" containerID="9503d2db5a1004a8aabb3765560bd4ff03029165a8fab7dd1c18ec09c986bda2" exitCode=0 Oct 30 16:22:52.935511 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.935382 4493 generic.go:356] "Generic (PLEG): container finished" podID="8e02b2b2-c05c-4821-adfe-02f2a6664267" containerID="82795c0397cc845c188df9b4cc15eba9edb9c015377158c20145ff2c7115d27e" exitCode=0 Oct 30 16:22:52.935511 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.935389 4493 generic.go:356] "Generic (PLEG): container finished" podID="8e02b2b2-c05c-4821-adfe-02f2a6664267" containerID="cf420fd2f91aafb332053258ecd786aa6c9b9382434793ec68f67020fe439df6" exitCode=0 Oct 30 16:22:52.935511 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.935397 4493 generic.go:356] "Generic (PLEG): container finished" podID="8e02b2b2-c05c-4821-adfe-02f2a6664267" containerID="ff2ec17cd0c968ad5cdba34fc178bae568e7be24fa56b6a00692195a7b7eaefe" exitCode=0 Oct 30 16:22:52.935713 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.935697 4493 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 16:22:52.935831 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.935816 4493 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 16:22:52.940673 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.940652 4493 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 16:22:52.945751 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.945731 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-56v9s_d3a1c417-2698-4462-b7ff-1d2e66d7a10f/ovn-acl-logging/0.log" Oct 30 16:22:52.946143 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.946119 4493 generic.go:356] "Generic (PLEG): container finished" podID="d3a1c417-2698-4462-b7ff-1d2e66d7a10f" containerID="c3dc910d7bf7e2b1a74e7fe9a25b6f7fcc4147625b9d41326f740453e5382364" exitCode=1 Oct 30 16:22:52.953162 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.953106 4493 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 16:22:52.953242 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.953211 4493 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 16:22:52.957968 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.957945 4493 generic.go:356] "Generic (PLEG): container finished" podID="83767d666be2011c3ea225fbea981665" containerID="c111e4d5ad6ad2b86fddac4e7ce3fd1f2debe385a1cf820a91d9a652c905a65f" exitCode=0 Oct 30 16:22:52.970479 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.970463 4493 manager.go:324] Recovery completed Oct 30 16:22:52.987034 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.987018 4493 cpu_manager.go:226] "Starting" policy="none" Oct 30 16:22:52.987034 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.987033 4493 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 16:22:52.987153 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.987052 4493 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 16:22:52.987209 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.987192 4493 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 16:22:52.987239 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.987212 4493 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 16:22:52.987239 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.987236 4493 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 16:22:52.987294 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.987244 4493 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 16:22:52.987294 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.987253 4493 policy_none.go:50] "Start" Oct 30 16:22:52.987294 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.987267 4493 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 16:22:52.987294 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.987278 4493 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 16:22:52.987452 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.987442 4493 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 16:22:52.987481 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.987453 4493 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 16:22:52.987481 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.987459 4493 policy_none.go:44] "Start" Oct 30 16:22:52.991286 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.991262 4493 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 16:22:52.993698 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.993686 4493 manager.go:341] "Starting Device Plugin manager" Oct 30 16:22:52.994422 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:22:52.993708 4493 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 16:22:52.994471 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.994428 4493 server.go:87] "Starting device plugin registration server" Oct 30 16:22:52.994810 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.994777 4493 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 16:22:52.994843 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.994819 4493 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 16:22:52.994942 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.994923 4493 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 16:22:52.995051 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.995019 4493 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 16:22:52.995135 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.995059 4493 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 16:22:52.995135 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.995066 4493 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 16:22:52.995222 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.995117 4493 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T16:22:52.995049523Z","UUID":"f3c861ca-e886-4e52-b30c-6bf30817885d","Handler":null,"Name":"","Endpoint":""} Oct 30 16:22:52.996222 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:22:52.996190 4493 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 16:22:52.996677 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.996520 4493 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 16:22:52.996677 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:52.996555 4493 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 16:22:53.025082 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.025051 4493 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-132-180.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-180.ec2.internal"] Oct 30 16:22:53.025590 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.025569 4493 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-180.ec2.internal" Oct 30 16:22:53.025663 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.025581 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-180.ec2.internal" event={"ID":"9a63059ab80503558edad3ed2b226c76","Type":"ContainerStarted","Data":"8346ea4afe0b1b2aa4c2d60811bdcb6608b13280291facd8246e91acc2a6b065"} Oct 30 16:22:53.025663 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.025633 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-180.ec2.internal" event={"ID":"9a63059ab80503558edad3ed2b226c76","Type":"ContainerStarted","Data":"ec42a59d26b309a2abc07c1f01503b90314473de1dca75a3ae30db9221dcb9dc"} Oct 30 16:22:53.025663 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.025650 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-180.ec2.internal" event={"ID":"83767d666be2011c3ea225fbea981665","Type":"ContainerStarted","Data":"15d245eb69d354b0e9e38c68511ac3330174e6cd1f728b0bcc75dbfb92915219"} Oct 30 16:22:53.025663 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.025653 4493 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-180.ec2.internal" Oct 30 16:22:53.025663 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.025662 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-180.ec2.internal" event={"ID":"83767d666be2011c3ea225fbea981665","Type":"ContainerDied","Data":"c111e4d5ad6ad2b86fddac4e7ce3fd1f2debe385a1cf820a91d9a652c905a65f"} Oct 30 16:22:53.025847 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.025675 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-180.ec2.internal" event={"ID":"83767d666be2011c3ea225fbea981665","Type":"ContainerStarted","Data":"c45d537cf983095d6f7f6e59329996acbabab075cf91370b12a541c56665f537"} Oct 30 16:22:53.048930 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.048910 4493 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 16:22:53.049006 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:22:53.048955 4493 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-132-180.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-180.ec2.internal" Oct 30 16:22:53.049543 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.049532 4493 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 16:22:53.049590 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:22:53.049554 4493 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-132-180.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-180.ec2.internal" Oct 30 16:22:53.095657 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.095636 4493 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 16:22:53.096882 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.096865 4493 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-180.ec2.internal" event="NodeHasSufficientMemory" Oct 30 16:22:53.096967 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.096898 4493 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-180.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 16:22:53.096967 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.096911 4493 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-180.ec2.internal" event="NodeHasSufficientPID" Oct 30 16:22:53.096967 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.096951 4493 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-132-180.ec2.internal" Oct 30 16:22:53.131031 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.131011 4493 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-132-180.ec2.internal" Oct 30 16:22:53.131098 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.131059 4493 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-132-180.ec2.internal" Oct 30 16:22:53.137685 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.137661 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/83767d666be2011c3ea225fbea981665-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-180.ec2.internal\" (UID: \"83767d666be2011c3ea225fbea981665\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-180.ec2.internal" Oct 30 16:22:53.137685 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.137686 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9a63059ab80503558edad3ed2b226c76-config\") pod \"kube-apiserver-proxy-ip-10-0-132-180.ec2.internal\" (UID: \"9a63059ab80503558edad3ed2b226c76\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-180.ec2.internal" Oct 30 16:22:53.137812 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.137702 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/83767d666be2011c3ea225fbea981665-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-180.ec2.internal\" (UID: \"83767d666be2011c3ea225fbea981665\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-180.ec2.internal" Oct 30 16:22:53.238300 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.238229 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9a63059ab80503558edad3ed2b226c76-config\") pod \"kube-apiserver-proxy-ip-10-0-132-180.ec2.internal\" (UID: \"9a63059ab80503558edad3ed2b226c76\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-180.ec2.internal" Oct 30 16:22:53.238300 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.238266 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/83767d666be2011c3ea225fbea981665-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-180.ec2.internal\" (UID: \"83767d666be2011c3ea225fbea981665\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-180.ec2.internal" Oct 30 16:22:53.238300 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.238282 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/83767d666be2011c3ea225fbea981665-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-180.ec2.internal\" (UID: \"83767d666be2011c3ea225fbea981665\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-180.ec2.internal" Oct 30 16:22:53.238477 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.238339 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/83767d666be2011c3ea225fbea981665-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-180.ec2.internal\" (UID: \"83767d666be2011c3ea225fbea981665\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-180.ec2.internal" Oct 30 16:22:53.238477 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.238342 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9a63059ab80503558edad3ed2b226c76-config\") pod \"kube-apiserver-proxy-ip-10-0-132-180.ec2.internal\" (UID: \"9a63059ab80503558edad3ed2b226c76\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-180.ec2.internal" Oct 30 16:22:53.238477 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.238346 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/83767d666be2011c3ea225fbea981665-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-180.ec2.internal\" (UID: \"83767d666be2011c3ea225fbea981665\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-180.ec2.internal" Oct 30 16:22:53.876326 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.876291 4493 apiserver.go:52] "Watching apiserver" Oct 30 16:22:53.915639 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.915613 4493 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 16:22:53.916066 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.916045 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-426gj","kube-system/konnectivity-agent-pg4kq","openshift-ovn-kubernetes/ovnkube-node-56v9s","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z96s5","openshift-insights/insights-runtime-extractor-c24kh","openshift-multus/multus-9nfhv","openshift-multus/network-metrics-daemon-mbqwf","openshift-kube-storage-version-migrator/migrator-66df5d496b-fftpj","openshift-cluster-node-tuning-operator/tuned-xbbxh","kube-system/kube-apiserver-proxy-ip-10-0-132-180.ec2.internal","openshift-image-registry/node-ca-jlr9c","openshift-service-ca/service-ca-5fffc97bdf-5m7b4","openshift-dns/dns-default-jvqzq","openshift-ingress-canary/ingress-canary-hlfhx","kube-system/global-pull-secret-syncer-2lrlr","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-180.ec2.internal","openshift-network-operator/iptables-alerter-z7jnf","openshift-multus/multus-additional-cni-plugins-5jkb7","openshift-dns/node-resolver-r5ppb"] Oct 30 16:22:53.916466 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.916448 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mbqwf" Oct 30 16:22:53.921622 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.921601 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jvqzq" Oct 30 16:22:53.923643 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.923628 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-c24kh" Oct 30 16:22:53.931383 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.931350 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 16:22:53.931882 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.931844 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-9txpl\"" Oct 30 16:22:53.931882 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.931870 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 16:22:53.932025 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.931873 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 16:22:53.932025 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.931971 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 16:22:53.932025 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.931981 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hpqxb\"" Oct 30 16:22:53.932025 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.931997 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 16:22:53.932203 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.932035 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 16:22:53.932203 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.932063 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 16:22:53.932203 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.932085 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 16:22:53.932203 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.932087 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9kf5z\"" Oct 30 16:22:53.932343 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.932222 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 16:22:53.932343 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.932231 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 16:22:53.942570 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.942536 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 16:22:53.942653 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.942639 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-vhqb4\"" Oct 30 16:22:53.942709 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.942695 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-5n7vr\"" Oct 30 16:22:53.943110 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.943094 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 16:22:53.943228 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.943213 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-m9x9f\"" Oct 30 16:22:53.943279 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.943254 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-bls9m\"" Oct 30 16:22:53.943331 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.943305 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-z9zpr\"" Oct 30 16:22:53.943372 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.943364 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-lf54z\"" Oct 30 16:22:53.943420 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.943408 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 16:22:53.943463 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.943422 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-d6852\"" Oct 30 16:22:53.943463 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.943460 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 16:22:53.943554 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.943485 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 16:22:53.943554 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.943485 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rjhbz\"" Oct 30 16:22:53.943658 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.943582 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 16:22:53.943658 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.943587 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 16:22:53.943658 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.943559 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 16:22:53.943658 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.943648 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 16:22:53.943658 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.943664 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 16:22:53.943974 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.943590 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 16:22:53.943974 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.943663 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 16:22:53.943974 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.943697 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 16:22:53.943974 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.943722 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 16:22:53.943974 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.943587 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 16:22:53.943974 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.943726 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 16:22:53.943974 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.943764 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 16:22:53.943974 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.943770 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 16:22:53.943974 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.943821 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 16:22:53.943974 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.943826 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-h4vt2\"" Oct 30 16:22:53.943974 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.943833 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-cmtxm\"" Oct 30 16:22:53.943974 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.943861 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 16:22:53.943974 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.943892 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 16:22:53.943974 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.943930 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 16:22:53.943974 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.943972 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 16:22:53.944805 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.944069 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 16:22:53.944805 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.944234 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-65hn8\"" Oct 30 16:22:53.944805 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.944239 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-86hj4\"" Oct 30 16:22:53.944805 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.944268 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 16:22:53.944805 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.944275 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 16:22:53.944805 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.944298 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 16:22:53.944805 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.944321 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tdd2t\"" Oct 30 16:22:53.944805 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.944393 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 16:22:53.944805 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.944469 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 16:22:53.944805 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.944471 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 16:22:53.944805 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.944483 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 16:22:53.944805 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.944518 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-lpvkt\"" Oct 30 16:22:53.944805 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.944594 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 16:22:53.944805 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.944597 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 16:22:53.944805 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.944626 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 16:22:53.944805 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.944645 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 16:22:53.944805 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.944673 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 16:22:53.965374 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.965359 4493 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-180.ec2.internal" Oct 30 16:22:53.965473 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.965462 4493 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-180.ec2.internal" Oct 30 16:22:53.988791 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.988773 4493 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 16:22:53.988887 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:22:53.988837 4493 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-132-180.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-180.ec2.internal" Oct 30 16:22:53.989651 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.989638 4493 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 16:22:53.989692 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:22:53.989672 4493 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-132-180.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-180.ec2.internal" Oct 30 16:22:53.989808 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:53.989798 4493 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 16:22:54.042264 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.042246 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8a7b4cd2-b289-42b0-a54b-d868de87d7d3-cert\") pod \"ingress-canary-hlfhx\" (UID: \"8a7b4cd2-b289-42b0-a54b-d868de87d7d3\") " pod="openshift-ingress-canary/ingress-canary-hlfhx" Oct 30 16:22:54.042382 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.042273 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8e02b2b2-c05c-4821-adfe-02f2a6664267-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5jkb7\" (UID: \"8e02b2b2-c05c-4821-adfe-02f2a6664267\") " pod="openshift-multus/multus-additional-cni-plugins-5jkb7" Oct 30 16:22:54.042382 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.042291 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hgtv\" (UniqueName: \"kubernetes.io/projected/21fd140d-96ea-462e-9f8b-ef6d480f8407-kube-api-access-5hgtv\") pod \"migrator-66df5d496b-fftpj\" (UID: \"21fd140d-96ea-462e-9f8b-ef6d480f8407\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fftpj" Oct 30 16:22:54.042382 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.042306 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a773f3a-800c-4c84-b18f-7a48cdd92692-tmp\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:54.042382 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.042321 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7c09760d-2160-41d4-b991-f6324ad5aff7-agent-certs\") pod \"konnectivity-agent-pg4kq\" (UID: \"7c09760d-2160-41d4-b991-f6324ad5aff7\") " pod="kube-system/konnectivity-agent-pg4kq" Oct 30 16:22:54.042382 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.042338 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-run-openvswitch\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:54.042382 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.042355 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-host-cni-bin\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:54.042382 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.042375 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-multus-daemon-config\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:54.042682 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.042427 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a773f3a-800c-4c84-b18f-7a48cdd92692-tmp\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:54.042682 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.042509 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e02b2b2-c05c-4821-adfe-02f2a6664267-system-cni-dir\") pod \"multus-additional-cni-plugins-5jkb7\" (UID: \"8e02b2b2-c05c-4821-adfe-02f2a6664267\") " pod="openshift-multus/multus-additional-cni-plugins-5jkb7" Oct 30 16:22:54.042682 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.042536 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8e02b2b2-c05c-4821-adfe-02f2a6664267-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5jkb7\" (UID: \"8e02b2b2-c05c-4821-adfe-02f2a6664267\") " pod="openshift-multus/multus-additional-cni-plugins-5jkb7" Oct 30 16:22:54.042682 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.042563 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8e02b2b2-c05c-4821-adfe-02f2a6664267-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5jkb7\" (UID: \"8e02b2b2-c05c-4821-adfe-02f2a6664267\") " pod="openshift-multus/multus-additional-cni-plugins-5jkb7" Oct 30 16:22:54.042682 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.042600 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc-config-volume\") pod \"dns-default-jvqzq\" (UID: \"7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc\") " pod="openshift-dns/dns-default-jvqzq" Oct 30 16:22:54.042935 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.042684 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8e02b2b2-c05c-4821-adfe-02f2a6664267-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5jkb7\" (UID: \"8e02b2b2-c05c-4821-adfe-02f2a6664267\") " pod="openshift-multus/multus-additional-cni-plugins-5jkb7" Oct 30 16:22:54.042935 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.042877 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-multus-daemon-config\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:54.043015 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.042951 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc-tmp-dir\") pod \"dns-default-jvqzq\" (UID: \"7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc\") " pod="openshift-dns/dns-default-jvqzq" Oct 30 16:22:54.043090 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.043034 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8a7b4cd2-b289-42b0-a54b-d868de87d7d3-cert\") pod \"ingress-canary-hlfhx\" (UID: \"8a7b4cd2-b289-42b0-a54b-d868de87d7d3\") " pod="openshift-ingress-canary/ingress-canary-hlfhx" Oct 30 16:22:54.043090 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.042625 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc-tmp-dir\") pod \"dns-default-jvqzq\" (UID: \"7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc\") " pod="openshift-dns/dns-default-jvqzq" Oct 30 16:22:54.043144 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.043096 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-system-cni-dir\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:54.043144 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.043112 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3a773f3a-800c-4c84-b18f-7a48cdd92692-etc-sysctl-d\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:54.043144 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.043126 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc-metrics-tls\") pod \"dns-default-jvqzq\" (UID: \"7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc\") " pod="openshift-dns/dns-default-jvqzq" Oct 30 16:22:54.043144 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.043141 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-node-log\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:54.043289 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.043157 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dlwk\" (UniqueName: \"kubernetes.io/projected/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-kube-api-access-6dlwk\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:54.043289 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.043206 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dc77c50f-abe0-4d9c-a88b-7a186c17dc3e-dbus\") pod \"global-pull-secret-syncer-2lrlr\" (UID: \"dc77c50f-abe0-4d9c-a88b-7a186c17dc3e\") " pod="kube-system/global-pull-secret-syncer-2lrlr" Oct 30 16:22:54.043289 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.043253 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/17bb82ba-5262-47f7-abbc-99f1329500fa-host\") pod \"node-ca-jlr9c\" (UID: \"17bb82ba-5262-47f7-abbc-99f1329500fa\") " pod="openshift-image-registry/node-ca-jlr9c" Oct 30 16:22:54.043435 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.043342 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fdbc\" (UniqueName: \"kubernetes.io/projected/8a7b4cd2-b289-42b0-a54b-d868de87d7d3-kube-api-access-8fdbc\") pod \"ingress-canary-hlfhx\" (UID: \"8a7b4cd2-b289-42b0-a54b-d868de87d7d3\") " pod="openshift-ingress-canary/ingress-canary-hlfhx" Oct 30 16:22:54.043435 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.043364 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8e02b2b2-c05c-4821-adfe-02f2a6664267-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5jkb7\" (UID: \"8e02b2b2-c05c-4821-adfe-02f2a6664267\") " pod="openshift-multus/multus-additional-cni-plugins-5jkb7" Oct 30 16:22:54.043435 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.043386 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc-config-volume\") pod \"dns-default-jvqzq\" (UID: \"7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc\") " pod="openshift-dns/dns-default-jvqzq" Oct 30 16:22:54.043435 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.043390 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c24kh\" (UID: \"8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c\") " pod="openshift-insights/insights-runtime-extractor-c24kh" Oct 30 16:22:54.043435 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.043430 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8e02b2b2-c05c-4821-adfe-02f2a6664267-cnibin\") pod \"multus-additional-cni-plugins-5jkb7\" (UID: \"8e02b2b2-c05c-4821-adfe-02f2a6664267\") " pod="openshift-multus/multus-additional-cni-plugins-5jkb7" Oct 30 16:22:54.043655 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.043461 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a773f3a-800c-4c84-b18f-7a48cdd92692-etc-kubernetes\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:54.043655 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.043484 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7c09760d-2160-41d4-b991-f6324ad5aff7-agent-certs\") pod \"konnectivity-agent-pg4kq\" (UID: \"7c09760d-2160-41d4-b991-f6324ad5aff7\") " pod="kube-system/konnectivity-agent-pg4kq" Oct 30 16:22:54.043655 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.043499 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3a773f3a-800c-4c84-b18f-7a48cdd92692-run\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:54.043655 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.043535 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-run-systemd\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:54.043655 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.043577 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-etc-openvswitch\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:54.043655 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.043615 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-multus-socket-dir-parent\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:54.043655 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.043636 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-host-run-netns\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:54.043938 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.043663 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j5vz\" (UniqueName: \"kubernetes.io/projected/8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c-kube-api-access-5j5vz\") pod \"insights-runtime-extractor-c24kh\" (UID: \"8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c\") " pod="openshift-insights/insights-runtime-extractor-c24kh" Oct 30 16:22:54.043938 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.043698 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c407642e-2f4a-4c47-906e-56137857d162-host-slash\") pod \"iptables-alerter-z7jnf\" (UID: \"c407642e-2f4a-4c47-906e-56137857d162\") " pod="openshift-network-operator/iptables-alerter-z7jnf" Oct 30 16:22:54.043938 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.043797 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3a773f3a-800c-4c84-b18f-7a48cdd92692-etc-systemd\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:54.043938 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.043836 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/315c49d6-d546-4fc2-a5c0-3b987269e5a9-signing-cabundle\") pod \"service-ca-5fffc97bdf-5m7b4\" (UID: \"315c49d6-d546-4fc2-a5c0-3b987269e5a9\") " pod="openshift-service-ca/service-ca-5fffc97bdf-5m7b4" Oct 30 16:22:54.043938 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.043868 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-hostroot\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:54.043938 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.043906 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/394d816a-1e33-433f-abb1-586089620763-registration-dir\") pod \"aws-ebs-csi-driver-node-z96s5\" (UID: \"394d816a-1e33-433f-abb1-586089620763\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z96s5" Oct 30 16:22:54.044192 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.043945 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-run-ovn\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:54.044192 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.043964 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7c09760d-2160-41d4-b991-f6324ad5aff7-konnectivity-ca\") pod \"konnectivity-agent-pg4kq\" (UID: \"7c09760d-2160-41d4-b991-f6324ad5aff7\") " pod="kube-system/konnectivity-agent-pg4kq" Oct 30 16:22:54.044192 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.043985 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/315c49d6-d546-4fc2-a5c0-3b987269e5a9-signing-key\") pod \"service-ca-5fffc97bdf-5m7b4\" (UID: \"315c49d6-d546-4fc2-a5c0-3b987269e5a9\") " pod="openshift-service-ca/service-ca-5fffc97bdf-5m7b4" Oct 30 16:22:54.044192 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.044016 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vrcg\" (UniqueName: \"kubernetes.io/projected/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-kube-api-access-8vrcg\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:54.044192 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.044036 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e649944-a09d-44b1-91d2-fda00b30292d-tmp-dir\") pod \"node-resolver-r5ppb\" (UID: \"0e649944-a09d-44b1-91d2-fda00b30292d\") " pod="openshift-dns/node-resolver-r5ppb" Oct 30 16:22:54.044192 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.044060 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/315c49d6-d546-4fc2-a5c0-3b987269e5a9-signing-cabundle\") pod \"service-ca-5fffc97bdf-5m7b4\" (UID: \"315c49d6-d546-4fc2-a5c0-3b987269e5a9\") " pod="openshift-service-ca/service-ca-5fffc97bdf-5m7b4" Oct 30 16:22:54.044192 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.044063 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-host-run-k8s-cni-cncf-io\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:54.044192 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.044100 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-host-var-lib-cni-multus\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:54.044192 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.044143 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7c09760d-2160-41d4-b991-f6324ad5aff7-konnectivity-ca\") pod \"konnectivity-agent-pg4kq\" (UID: \"7c09760d-2160-41d4-b991-f6324ad5aff7\") " pod="kube-system/konnectivity-agent-pg4kq" Oct 30 16:22:54.044192 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.044178 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-etc-kubernetes\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:54.044558 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.044186 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e649944-a09d-44b1-91d2-fda00b30292d-tmp-dir\") pod \"node-resolver-r5ppb\" (UID: \"0e649944-a09d-44b1-91d2-fda00b30292d\") " pod="openshift-dns/node-resolver-r5ppb" Oct 30 16:22:54.044558 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.044223 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5v7j\" (UniqueName: \"kubernetes.io/projected/3a773f3a-800c-4c84-b18f-7a48cdd92692-kube-api-access-m5v7j\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:54.044558 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.044252 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-host-cni-netd\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:54.044558 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.044265 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/315c49d6-d546-4fc2-a5c0-3b987269e5a9-signing-key\") pod \"service-ca-5fffc97bdf-5m7b4\" (UID: \"315c49d6-d546-4fc2-a5c0-3b987269e5a9\") " pod="openshift-service-ca/service-ca-5fffc97bdf-5m7b4" Oct 30 16:22:54.044558 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.044337 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-host-var-lib-cni-bin\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:54.044558 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.044378 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c-data-volume\") pod \"insights-runtime-extractor-c24kh\" (UID: \"8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c\") " pod="openshift-insights/insights-runtime-extractor-c24kh" Oct 30 16:22:54.044558 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.044426 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c-data-volume\") pod \"insights-runtime-extractor-c24kh\" (UID: \"8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c\") " pod="openshift-insights/insights-runtime-extractor-c24kh" Oct 30 16:22:54.044558 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.044465 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zftsq\" (UniqueName: \"kubernetes.io/projected/c407642e-2f4a-4c47-906e-56137857d162-kube-api-access-zftsq\") pod \"iptables-alerter-z7jnf\" (UID: \"c407642e-2f4a-4c47-906e-56137857d162\") " pod="openshift-network-operator/iptables-alerter-z7jnf" Oct 30 16:22:54.044558 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.044504 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3a773f3a-800c-4c84-b18f-7a48cdd92692-lib-modules\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:54.044558 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.044520 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a773f3a-800c-4c84-b18f-7a48cdd92692-var-lib-kubelet\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:54.044558 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.044548 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srfcg\" (UniqueName: \"kubernetes.io/projected/78b9342d-5361-4a8c-a771-0c1aea26a48d-kube-api-access-srfcg\") pod \"network-check-target-426gj\" (UID: \"78b9342d-5361-4a8c-a771-0c1aea26a48d\") " pod="openshift-network-diagnostics/network-check-target-426gj" Oct 30 16:22:54.044983 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.044579 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-host-run-netns\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:54.044983 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.044599 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:54.044983 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.044714 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s64j8\" (UniqueName: \"kubernetes.io/projected/0e649944-a09d-44b1-91d2-fda00b30292d-kube-api-access-s64j8\") pod \"node-resolver-r5ppb\" (UID: \"0e649944-a09d-44b1-91d2-fda00b30292d\") " pod="openshift-dns/node-resolver-r5ppb" Oct 30 16:22:54.044983 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.044748 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/394d816a-1e33-433f-abb1-586089620763-socket-dir\") pod \"aws-ebs-csi-driver-node-z96s5\" (UID: \"394d816a-1e33-433f-abb1-586089620763\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z96s5" Oct 30 16:22:54.044983 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.044831 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/394d816a-1e33-433f-abb1-586089620763-sys-fs\") pod \"aws-ebs-csi-driver-node-z96s5\" (UID: \"394d816a-1e33-433f-abb1-586089620763\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z96s5" Oct 30 16:22:54.044983 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.044855 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3a773f3a-800c-4c84-b18f-7a48cdd92692-etc-sysctl-conf\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:54.044983 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.044870 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3a773f3a-800c-4c84-b18f-7a48cdd92692-sys\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:54.044983 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.044905 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f99rt\" (UniqueName: \"kubernetes.io/projected/7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc-kube-api-access-f99rt\") pod \"dns-default-jvqzq\" (UID: \"7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc\") " pod="openshift-dns/dns-default-jvqzq" Oct 30 16:22:54.044983 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.044940 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b804fc37-0b07-4edf-8720-b31b5698b935-metrics-certs\") pod \"network-metrics-daemon-mbqwf\" (UID: \"b804fc37-0b07-4edf-8720-b31b5698b935\") " pod="openshift-multus/network-metrics-daemon-mbqwf" Oct 30 16:22:54.045327 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.044988 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8vll\" (UniqueName: \"kubernetes.io/projected/b804fc37-0b07-4edf-8720-b31b5698b935-kube-api-access-m8vll\") pod \"network-metrics-daemon-mbqwf\" (UID: \"b804fc37-0b07-4edf-8720-b31b5698b935\") " pod="openshift-multus/network-metrics-daemon-mbqwf" Oct 30 16:22:54.045327 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.045007 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-var-lib-openvswitch\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:54.045327 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.045024 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8e02b2b2-c05c-4821-adfe-02f2a6664267-cni-binary-copy\") pod \"multus-additional-cni-plugins-5jkb7\" (UID: \"8e02b2b2-c05c-4821-adfe-02f2a6664267\") " pod="openshift-multus/multus-additional-cni-plugins-5jkb7" Oct 30 16:22:54.045327 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.045058 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-env-overrides\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:54.045327 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.045082 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c24kh\" (UID: \"8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c\") " pod="openshift-insights/insights-runtime-extractor-c24kh" Oct 30 16:22:54.045327 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.045097 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3a773f3a-800c-4c84-b18f-7a48cdd92692-etc-modprobe-d\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:54.045327 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.045119 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-host-kubelet\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:54.045327 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.045196 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-ovnkube-config\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:54.045327 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.045231 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8e02b2b2-c05c-4821-adfe-02f2a6664267-cni-binary-copy\") pod \"multus-additional-cni-plugins-5jkb7\" (UID: \"8e02b2b2-c05c-4821-adfe-02f2a6664267\") " pod="openshift-multus/multus-additional-cni-plugins-5jkb7" Oct 30 16:22:54.045327 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.045281 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-env-overrides\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:54.045731 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.045350 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-ovnkube-config\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:54.045731 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.045395 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-ovn-node-metrics-cert\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:54.045731 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.045427 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-multus-cni-dir\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:54.045731 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.045441 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-cnibin\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:54.045731 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.045456 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-host-run-multus-certs\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:54.045731 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.045471 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dc77c50f-abe0-4d9c-a88b-7a186c17dc3e-kubelet-config\") pod \"global-pull-secret-syncer-2lrlr\" (UID: \"dc77c50f-abe0-4d9c-a88b-7a186c17dc3e\") " pod="kube-system/global-pull-secret-syncer-2lrlr" Oct 30 16:22:54.045731 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.045487 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/394d816a-1e33-433f-abb1-586089620763-etc-selinux\") pod \"aws-ebs-csi-driver-node-z96s5\" (UID: \"394d816a-1e33-433f-abb1-586089620763\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z96s5" Oct 30 16:22:54.045731 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.045466 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-ovn-node-metrics-cert\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:54.045731 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.045517 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3a773f3a-800c-4c84-b18f-7a48cdd92692-host\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:54.045731 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.045561 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3a773f3a-800c-4c84-b18f-7a48cdd92692-etc-tuned\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:54.045731 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.045592 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-log-socket\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:54.045731 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.045608 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-cni-binary-copy\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:54.045731 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.045667 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c-crio-socket\") pod \"insights-runtime-extractor-c24kh\" (UID: \"8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c\") " pod="openshift-insights/insights-runtime-extractor-c24kh" Oct 30 16:22:54.045731 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.045673 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3a773f3a-800c-4c84-b18f-7a48cdd92692-etc-tuned\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:54.045731 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.045704 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-host-var-lib-kubelet\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:54.046163 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.045740 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-multus-conf-dir\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:54.046163 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.045763 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m78c4\" (UniqueName: \"kubernetes.io/projected/8e02b2b2-c05c-4821-adfe-02f2a6664267-kube-api-access-m78c4\") pod \"multus-additional-cni-plugins-5jkb7\" (UID: \"8e02b2b2-c05c-4821-adfe-02f2a6664267\") " pod="openshift-multus/multus-additional-cni-plugins-5jkb7" Oct 30 16:22:54.046163 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.045770 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-cni-binary-copy\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:54.046163 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.045841 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/394d816a-1e33-433f-abb1-586089620763-device-dir\") pod \"aws-ebs-csi-driver-node-z96s5\" (UID: \"394d816a-1e33-433f-abb1-586089620763\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z96s5" Oct 30 16:22:54.046163 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.045875 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb9mk\" (UniqueName: \"kubernetes.io/projected/394d816a-1e33-433f-abb1-586089620763-kube-api-access-nb9mk\") pod \"aws-ebs-csi-driver-node-z96s5\" (UID: \"394d816a-1e33-433f-abb1-586089620763\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z96s5" Oct 30 16:22:54.046163 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.045891 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3a773f3a-800c-4c84-b18f-7a48cdd92692-etc-sysconfig\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:54.046163 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.045975 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntggj\" (UniqueName: \"kubernetes.io/projected/315c49d6-d546-4fc2-a5c0-3b987269e5a9-kube-api-access-ntggj\") pod \"service-ca-5fffc97bdf-5m7b4\" (UID: \"315c49d6-d546-4fc2-a5c0-3b987269e5a9\") " pod="openshift-service-ca/service-ca-5fffc97bdf-5m7b4" Oct 30 16:22:54.046163 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.045997 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-host-run-ovn-kubernetes\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:54.046163 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.046018 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-ovnkube-script-lib\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:54.046163 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.046050 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0e649944-a09d-44b1-91d2-fda00b30292d-hosts-file\") pod \"node-resolver-r5ppb\" (UID: \"0e649944-a09d-44b1-91d2-fda00b30292d\") " pod="openshift-dns/node-resolver-r5ppb" Oct 30 16:22:54.046163 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.046071 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c407642e-2f4a-4c47-906e-56137857d162-iptables-alerter-script\") pod \"iptables-alerter-z7jnf\" (UID: \"c407642e-2f4a-4c47-906e-56137857d162\") " pod="openshift-network-operator/iptables-alerter-z7jnf" Oct 30 16:22:54.046163 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.046090 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/394d816a-1e33-433f-abb1-586089620763-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z96s5\" (UID: \"394d816a-1e33-433f-abb1-586089620763\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z96s5" Oct 30 16:22:54.046163 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.046113 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-systemd-units\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:54.046608 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.046217 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-host-slash\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:54.046608 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.046242 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc77c50f-abe0-4d9c-a88b-7a186c17dc3e-original-pull-secret\") pod \"global-pull-secret-syncer-2lrlr\" (UID: \"dc77c50f-abe0-4d9c-a88b-7a186c17dc3e\") " pod="kube-system/global-pull-secret-syncer-2lrlr" Oct 30 16:22:54.046608 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.046261 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/17bb82ba-5262-47f7-abbc-99f1329500fa-serviceca\") pod \"node-ca-jlr9c\" (UID: \"17bb82ba-5262-47f7-abbc-99f1329500fa\") " pod="openshift-image-registry/node-ca-jlr9c" Oct 30 16:22:54.046608 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.046285 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c407642e-2f4a-4c47-906e-56137857d162-iptables-alerter-script\") pod \"iptables-alerter-z7jnf\" (UID: \"c407642e-2f4a-4c47-906e-56137857d162\") " pod="openshift-network-operator/iptables-alerter-z7jnf" Oct 30 16:22:54.046608 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.046310 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-ovnkube-script-lib\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:54.046608 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.046313 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxrxb\" (UniqueName: \"kubernetes.io/projected/17bb82ba-5262-47f7-abbc-99f1329500fa-kube-api-access-vxrxb\") pod \"node-ca-jlr9c\" (UID: \"17bb82ba-5262-47f7-abbc-99f1329500fa\") " pod="openshift-image-registry/node-ca-jlr9c" Oct 30 16:22:54.046608 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.046440 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/17bb82ba-5262-47f7-abbc-99f1329500fa-serviceca\") pod \"node-ca-jlr9c\" (UID: \"17bb82ba-5262-47f7-abbc-99f1329500fa\") " pod="openshift-image-registry/node-ca-jlr9c" Oct 30 16:22:54.046608 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.046496 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dc77c50f-abe0-4d9c-a88b-7a186c17dc3e-original-pull-secret\") pod \"global-pull-secret-syncer-2lrlr\" (UID: \"dc77c50f-abe0-4d9c-a88b-7a186c17dc3e\") " pod="kube-system/global-pull-secret-syncer-2lrlr" Oct 30 16:22:54.062857 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.062837 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hgtv\" (UniqueName: \"kubernetes.io/projected/21fd140d-96ea-462e-9f8b-ef6d480f8407-kube-api-access-5hgtv\") pod \"migrator-66df5d496b-fftpj\" (UID: \"21fd140d-96ea-462e-9f8b-ef6d480f8407\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fftpj" Oct 30 16:22:54.063472 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.063457 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j5vz\" (UniqueName: \"kubernetes.io/projected/8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c-kube-api-access-5j5vz\") pod \"insights-runtime-extractor-c24kh\" (UID: \"8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c\") " pod="openshift-insights/insights-runtime-extractor-c24kh" Oct 30 16:22:54.064201 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.064186 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fdbc\" (UniqueName: \"kubernetes.io/projected/8a7b4cd2-b289-42b0-a54b-d868de87d7d3-kube-api-access-8fdbc\") pod \"ingress-canary-hlfhx\" (UID: \"8a7b4cd2-b289-42b0-a54b-d868de87d7d3\") " pod="openshift-ingress-canary/ingress-canary-hlfhx" Oct 30 16:22:54.064456 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.064440 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dlwk\" (UniqueName: \"kubernetes.io/projected/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-kube-api-access-6dlwk\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:54.066795 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.066765 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f99rt\" (UniqueName: \"kubernetes.io/projected/7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc-kube-api-access-f99rt\") pod \"dns-default-jvqzq\" (UID: \"7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc\") " pod="openshift-dns/dns-default-jvqzq" Oct 30 16:22:54.067923 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.067908 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zftsq\" (UniqueName: \"kubernetes.io/projected/c407642e-2f4a-4c47-906e-56137857d162-kube-api-access-zftsq\") pod \"iptables-alerter-z7jnf\" (UID: \"c407642e-2f4a-4c47-906e-56137857d162\") " pod="openshift-network-operator/iptables-alerter-z7jnf" Oct 30 16:22:54.069572 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.069542 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vrcg\" (UniqueName: \"kubernetes.io/projected/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-kube-api-access-8vrcg\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:54.069626 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.069618 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8vll\" (UniqueName: \"kubernetes.io/projected/b804fc37-0b07-4edf-8720-b31b5698b935-kube-api-access-m8vll\") pod \"network-metrics-daemon-mbqwf\" (UID: \"b804fc37-0b07-4edf-8720-b31b5698b935\") " pod="openshift-multus/network-metrics-daemon-mbqwf" Oct 30 16:22:54.069727 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.069708 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5v7j\" (UniqueName: \"kubernetes.io/projected/3a773f3a-800c-4c84-b18f-7a48cdd92692-kube-api-access-m5v7j\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:54.072414 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.072359 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s64j8\" (UniqueName: \"kubernetes.io/projected/0e649944-a09d-44b1-91d2-fda00b30292d-kube-api-access-s64j8\") pod \"node-resolver-r5ppb\" (UID: \"0e649944-a09d-44b1-91d2-fda00b30292d\") " pod="openshift-dns/node-resolver-r5ppb" Oct 30 16:22:54.074353 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.074331 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srfcg\" (UniqueName: \"kubernetes.io/projected/78b9342d-5361-4a8c-a771-0c1aea26a48d-kube-api-access-srfcg\") pod \"network-check-target-426gj\" (UID: \"78b9342d-5361-4a8c-a771-0c1aea26a48d\") " pod="openshift-network-diagnostics/network-check-target-426gj" Oct 30 16:22:54.076410 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.076390 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb9mk\" (UniqueName: \"kubernetes.io/projected/394d816a-1e33-433f-abb1-586089620763-kube-api-access-nb9mk\") pod \"aws-ebs-csi-driver-node-z96s5\" (UID: \"394d816a-1e33-433f-abb1-586089620763\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z96s5" Oct 30 16:22:54.076913 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.076892 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m78c4\" (UniqueName: \"kubernetes.io/projected/8e02b2b2-c05c-4821-adfe-02f2a6664267-kube-api-access-m78c4\") pod \"multus-additional-cni-plugins-5jkb7\" (UID: \"8e02b2b2-c05c-4821-adfe-02f2a6664267\") " pod="openshift-multus/multus-additional-cni-plugins-5jkb7" Oct 30 16:22:54.076986 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.076912 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxrxb\" (UniqueName: \"kubernetes.io/projected/17bb82ba-5262-47f7-abbc-99f1329500fa-kube-api-access-vxrxb\") pod \"node-ca-jlr9c\" (UID: \"17bb82ba-5262-47f7-abbc-99f1329500fa\") " pod="openshift-image-registry/node-ca-jlr9c" Oct 30 16:22:54.077030 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.077015 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntggj\" (UniqueName: \"kubernetes.io/projected/315c49d6-d546-4fc2-a5c0-3b987269e5a9-kube-api-access-ntggj\") pod \"service-ca-5fffc97bdf-5m7b4\" (UID: \"315c49d6-d546-4fc2-a5c0-3b987269e5a9\") " pod="openshift-service-ca/service-ca-5fffc97bdf-5m7b4" Oct 30 16:22:54.146898 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.146879 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-host-run-netns\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:54.146970 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.146907 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:54.146970 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.146923 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/394d816a-1e33-433f-abb1-586089620763-socket-dir\") pod \"aws-ebs-csi-driver-node-z96s5\" (UID: \"394d816a-1e33-433f-abb1-586089620763\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z96s5" Oct 30 16:22:54.146970 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.146942 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/394d816a-1e33-433f-abb1-586089620763-sys-fs\") pod \"aws-ebs-csi-driver-node-z96s5\" (UID: \"394d816a-1e33-433f-abb1-586089620763\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z96s5" Oct 30 16:22:54.146970 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.146965 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3a773f3a-800c-4c84-b18f-7a48cdd92692-etc-sysctl-conf\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:54.147147 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.146987 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3a773f3a-800c-4c84-b18f-7a48cdd92692-sys\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:54.147147 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.146986 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:54.147147 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.147012 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-host-run-netns\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:54.147147 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.147010 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b804fc37-0b07-4edf-8720-b31b5698b935-metrics-certs\") pod \"network-metrics-daemon-mbqwf\" (UID: \"b804fc37-0b07-4edf-8720-b31b5698b935\") " pod="openshift-multus/network-metrics-daemon-mbqwf" Oct 30 16:22:54.147147 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.147045 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/394d816a-1e33-433f-abb1-586089620763-sys-fs\") pod \"aws-ebs-csi-driver-node-z96s5\" (UID: \"394d816a-1e33-433f-abb1-586089620763\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z96s5" Oct 30 16:22:54.147147 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.147056 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3a773f3a-800c-4c84-b18f-7a48cdd92692-sys\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:54.147147 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.147067 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/394d816a-1e33-433f-abb1-586089620763-socket-dir\") pod \"aws-ebs-csi-driver-node-z96s5\" (UID: \"394d816a-1e33-433f-abb1-586089620763\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z96s5" Oct 30 16:22:54.147147 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.147088 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-var-lib-openvswitch\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:54.147147 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.147110 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3a773f3a-800c-4c84-b18f-7a48cdd92692-etc-sysctl-conf\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:54.147147 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.147121 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-var-lib-openvswitch\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:54.147147 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.147125 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c24kh\" (UID: \"8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c\") " pod="openshift-insights/insights-runtime-extractor-c24kh" Oct 30 16:22:54.147557 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.147158 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3a773f3a-800c-4c84-b18f-7a48cdd92692-etc-modprobe-d\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:54.147557 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.147181 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-host-kubelet\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:54.147557 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.147207 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-multus-cni-dir\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:54.147557 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.147229 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-cnibin\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:54.147557 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:22:54.147222 4493 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 16:22:54.147557 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.147254 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-host-run-multus-certs\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:54.147557 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.147261 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-host-kubelet\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:54.147557 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.147276 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dc77c50f-abe0-4d9c-a88b-7a186c17dc3e-kubelet-config\") pod \"global-pull-secret-syncer-2lrlr\" (UID: \"dc77c50f-abe0-4d9c-a88b-7a186c17dc3e\") " pod="kube-system/global-pull-secret-syncer-2lrlr" Oct 30 16:22:54.147557 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.147287 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-multus-cni-dir\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:54.147557 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.147298 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/394d816a-1e33-433f-abb1-586089620763-etc-selinux\") pod \"aws-ebs-csi-driver-node-z96s5\" (UID: \"394d816a-1e33-433f-abb1-586089620763\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z96s5" Oct 30 16:22:54.147557 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.147302 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3a773f3a-800c-4c84-b18f-7a48cdd92692-etc-modprobe-d\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:54.147557 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.147305 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-cnibin\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:54.147557 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:22:54.147316 4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c-insights-runtime-extractor-tls podName:8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c nodeName:}" failed. No retries permitted until 2026-10-30 16:22:54.647295737 +0000 UTC m=+1.874470832 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c24kh" (UID: "8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c") : secret "insights-runtime-extractor-tls" not found Oct 30 16:22:54.147557 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.147336 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-host-run-multus-certs\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:54.147557 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.147344 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/394d816a-1e33-433f-abb1-586089620763-etc-selinux\") pod \"aws-ebs-csi-driver-node-z96s5\" (UID: \"394d816a-1e33-433f-abb1-586089620763\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z96s5" Oct 30 16:22:54.147557 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.147350 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3a773f3a-800c-4c84-b18f-7a48cdd92692-host\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:54.147557 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.147341 4493 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 16:22:54.147557 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.147375 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-log-socket\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:54.147557 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.147355 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dc77c50f-abe0-4d9c-a88b-7a186c17dc3e-kubelet-config\") pod \"global-pull-secret-syncer-2lrlr\" (UID: \"dc77c50f-abe0-4d9c-a88b-7a186c17dc3e\") " pod="kube-system/global-pull-secret-syncer-2lrlr" Oct 30 16:22:54.147557 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.147402 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c-crio-socket\") pod \"insights-runtime-extractor-c24kh\" (UID: \"8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c\") " pod="openshift-insights/insights-runtime-extractor-c24kh" Oct 30 16:22:54.147557 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.147425 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3a773f3a-800c-4c84-b18f-7a48cdd92692-host\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:54.147557 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.147427 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-log-socket\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:54.147557 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.147427 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-host-var-lib-kubelet\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:54.147557 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.147458 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c-crio-socket\") pod \"insights-runtime-extractor-c24kh\" (UID: \"8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c\") " pod="openshift-insights/insights-runtime-extractor-c24kh" Oct 30 16:22:54.147557 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.147474 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-multus-conf-dir\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:54.147557 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.147497 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/394d816a-1e33-433f-abb1-586089620763-device-dir\") pod \"aws-ebs-csi-driver-node-z96s5\" (UID: \"394d816a-1e33-433f-abb1-586089620763\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z96s5" Oct 30 16:22:54.147557 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.147503 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-multus-conf-dir\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:54.147557 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.147513 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3a773f3a-800c-4c84-b18f-7a48cdd92692-etc-sysconfig\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:54.147557 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.147535 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/394d816a-1e33-433f-abb1-586089620763-device-dir\") pod \"aws-ebs-csi-driver-node-z96s5\" (UID: \"394d816a-1e33-433f-abb1-586089620763\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z96s5" Oct 30 16:22:54.147557 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.147534 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-host-run-ovn-kubernetes\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:54.147557 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.147459 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-host-var-lib-kubelet\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:54.147557 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.147567 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-host-run-ovn-kubernetes\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:54.147557 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.147568 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3a773f3a-800c-4c84-b18f-7a48cdd92692-etc-sysconfig\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:54.147557 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.147577 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0e649944-a09d-44b1-91d2-fda00b30292d-hosts-file\") pod \"node-resolver-r5ppb\" (UID: \"0e649944-a09d-44b1-91d2-fda00b30292d\") " pod="openshift-dns/node-resolver-r5ppb" Oct 30 16:22:54.148875 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.147602 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0e649944-a09d-44b1-91d2-fda00b30292d-hosts-file\") pod \"node-resolver-r5ppb\" (UID: \"0e649944-a09d-44b1-91d2-fda00b30292d\") " pod="openshift-dns/node-resolver-r5ppb" Oct 30 16:22:54.148875 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.147603 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/394d816a-1e33-433f-abb1-586089620763-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z96s5\" (UID: \"394d816a-1e33-433f-abb1-586089620763\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z96s5" Oct 30 16:22:54.148875 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.147626 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/394d816a-1e33-433f-abb1-586089620763-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z96s5\" (UID: \"394d816a-1e33-433f-abb1-586089620763\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z96s5" Oct 30 16:22:54.148875 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.147629 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-systemd-units\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:54.148875 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.147649 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-host-slash\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:54.148875 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.147667 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-run-openvswitch\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:54.148875 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.147681 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-host-cni-bin\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:54.148875 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.147678 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-systemd-units\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:54.148875 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.147708 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-host-slash\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:54.148875 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.147703 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e02b2b2-c05c-4821-adfe-02f2a6664267-system-cni-dir\") pod \"multus-additional-cni-plugins-5jkb7\" (UID: \"8e02b2b2-c05c-4821-adfe-02f2a6664267\") " pod="openshift-multus/multus-additional-cni-plugins-5jkb7" Oct 30 16:22:54.148875 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.147735 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8e02b2b2-c05c-4821-adfe-02f2a6664267-system-cni-dir\") pod \"multus-additional-cni-plugins-5jkb7\" (UID: \"8e02b2b2-c05c-4821-adfe-02f2a6664267\") " pod="openshift-multus/multus-additional-cni-plugins-5jkb7" Oct 30 16:22:54.148875 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.147724 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-host-cni-bin\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:54.148875 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.147765 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-run-openvswitch\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:54.148875 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.147750 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8e02b2b2-c05c-4821-adfe-02f2a6664267-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5jkb7\" (UID: \"8e02b2b2-c05c-4821-adfe-02f2a6664267\") " pod="openshift-multus/multus-additional-cni-plugins-5jkb7" Oct 30 16:22:54.148875 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.147834 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-system-cni-dir\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:54.148875 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.147857 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3a773f3a-800c-4c84-b18f-7a48cdd92692-etc-sysctl-d\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:54.148875 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.147876 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-system-cni-dir\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:54.148875 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.147886 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc-metrics-tls\") pod \"dns-default-jvqzq\" (UID: \"7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc\") " pod="openshift-dns/dns-default-jvqzq" Oct 30 16:22:54.148875 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.147909 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-node-log\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:54.148875 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.147931 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dc77c50f-abe0-4d9c-a88b-7a186c17dc3e-dbus\") pod \"global-pull-secret-syncer-2lrlr\" (UID: \"dc77c50f-abe0-4d9c-a88b-7a186c17dc3e\") " pod="kube-system/global-pull-secret-syncer-2lrlr" Oct 30 16:22:54.148875 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.147947 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3a773f3a-800c-4c84-b18f-7a48cdd92692-etc-sysctl-d\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:54.148875 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.147957 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/17bb82ba-5262-47f7-abbc-99f1329500fa-host\") pod \"node-ca-jlr9c\" (UID: \"17bb82ba-5262-47f7-abbc-99f1329500fa\") " pod="openshift-image-registry/node-ca-jlr9c" Oct 30 16:22:54.148875 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.147963 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-node-log\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:54.148875 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.147980 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c24kh\" (UID: \"8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c\") " pod="openshift-insights/insights-runtime-extractor-c24kh" Oct 30 16:22:54.148875 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.147958 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8e02b2b2-c05c-4821-adfe-02f2a6664267-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5jkb7\" (UID: \"8e02b2b2-c05c-4821-adfe-02f2a6664267\") " pod="openshift-multus/multus-additional-cni-plugins-5jkb7" Oct 30 16:22:54.148875 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.148003 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dc77c50f-abe0-4d9c-a88b-7a186c17dc3e-dbus\") pod \"global-pull-secret-syncer-2lrlr\" (UID: \"dc77c50f-abe0-4d9c-a88b-7a186c17dc3e\") " pod="kube-system/global-pull-secret-syncer-2lrlr" Oct 30 16:22:54.148875 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.148007 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/17bb82ba-5262-47f7-abbc-99f1329500fa-host\") pod \"node-ca-jlr9c\" (UID: \"17bb82ba-5262-47f7-abbc-99f1329500fa\") " pod="openshift-image-registry/node-ca-jlr9c" Oct 30 16:22:54.148875 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.148004 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8e02b2b2-c05c-4821-adfe-02f2a6664267-cnibin\") pod \"multus-additional-cni-plugins-5jkb7\" (UID: \"8e02b2b2-c05c-4821-adfe-02f2a6664267\") " pod="openshift-multus/multus-additional-cni-plugins-5jkb7" Oct 30 16:22:54.148875 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.148040 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8e02b2b2-c05c-4821-adfe-02f2a6664267-cnibin\") pod \"multus-additional-cni-plugins-5jkb7\" (UID: \"8e02b2b2-c05c-4821-adfe-02f2a6664267\") " pod="openshift-multus/multus-additional-cni-plugins-5jkb7" Oct 30 16:22:54.148875 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.148088 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a773f3a-800c-4c84-b18f-7a48cdd92692-etc-kubernetes\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:54.148875 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.148042 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a773f3a-800c-4c84-b18f-7a48cdd92692-etc-kubernetes\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:54.148875 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.148120 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3a773f3a-800c-4c84-b18f-7a48cdd92692-run\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:54.148875 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.148144 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-run-systemd\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:54.148875 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.148168 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-etc-openvswitch\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:54.148875 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.148191 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-multus-socket-dir-parent\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:54.148875 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.148215 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-host-run-netns\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:54.148875 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.148222 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3a773f3a-800c-4c84-b18f-7a48cdd92692-run\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:54.148875 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.148222 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-run-systemd\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:54.148875 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.148238 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c407642e-2f4a-4c47-906e-56137857d162-host-slash\") pod \"iptables-alerter-z7jnf\" (UID: \"c407642e-2f4a-4c47-906e-56137857d162\") " pod="openshift-network-operator/iptables-alerter-z7jnf" Oct 30 16:22:54.148875 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.148253 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-etc-openvswitch\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:54.148875 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.148267 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3a773f3a-800c-4c84-b18f-7a48cdd92692-etc-systemd\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:54.148875 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.148273 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c407642e-2f4a-4c47-906e-56137857d162-host-slash\") pod \"iptables-alerter-z7jnf\" (UID: \"c407642e-2f4a-4c47-906e-56137857d162\") " pod="openshift-network-operator/iptables-alerter-z7jnf" Oct 30 16:22:54.148875 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.148267 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-multus-socket-dir-parent\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:54.148875 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.148297 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-host-run-netns\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:54.148875 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.148293 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-hostroot\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:54.148875 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.148331 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/394d816a-1e33-433f-abb1-586089620763-registration-dir\") pod \"aws-ebs-csi-driver-node-z96s5\" (UID: \"394d816a-1e33-433f-abb1-586089620763\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z96s5" Oct 30 16:22:54.148875 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.148338 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3a773f3a-800c-4c84-b18f-7a48cdd92692-etc-systemd\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:54.148875 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.148334 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-hostroot\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:54.148875 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.148355 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-run-ovn\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:54.148875 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.148370 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/394d816a-1e33-433f-abb1-586089620763-registration-dir\") pod \"aws-ebs-csi-driver-node-z96s5\" (UID: \"394d816a-1e33-433f-abb1-586089620763\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z96s5" Oct 30 16:22:54.148875 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.148386 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-run-ovn\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:54.148875 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.148391 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-host-run-k8s-cni-cncf-io\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:54.148875 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.148429 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-host-var-lib-cni-multus\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:54.148875 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.148452 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-etc-kubernetes\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:54.148875 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.148476 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-host-cni-netd\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:54.148875 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.148479 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-host-run-k8s-cni-cncf-io\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:54.148875 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.148492 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-host-var-lib-cni-multus\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:54.148875 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.148508 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-host-var-lib-cni-bin\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:54.148875 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.148519 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d3a1c417-2698-4462-b7ff-1d2e66d7a10f-host-cni-netd\") pod \"ovnkube-node-56v9s\" (UID: \"d3a1c417-2698-4462-b7ff-1d2e66d7a10f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:54.148875 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.148563 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-host-var-lib-cni-bin\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:54.148875 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.148537 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe9cbcfe-25a7-4da2-bd0d-cb313273c846-etc-kubernetes\") pod \"multus-9nfhv\" (UID: \"fe9cbcfe-25a7-4da2-bd0d-cb313273c846\") " pod="openshift-multus/multus-9nfhv" Oct 30 16:22:54.148875 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.148563 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c24kh\" (UID: \"8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c\") " pod="openshift-insights/insights-runtime-extractor-c24kh" Oct 30 16:22:54.148875 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.148533 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3a773f3a-800c-4c84-b18f-7a48cdd92692-lib-modules\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:54.148875 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.148613 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3a773f3a-800c-4c84-b18f-7a48cdd92692-lib-modules\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:54.148875 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.148614 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a773f3a-800c-4c84-b18f-7a48cdd92692-var-lib-kubelet\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:54.148875 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.148651 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a773f3a-800c-4c84-b18f-7a48cdd92692-var-lib-kubelet\") pod \"tuned-xbbxh\" (UID: \"3a773f3a-800c-4c84-b18f-7a48cdd92692\") " pod="openshift-cluster-node-tuning-operator/tuned-xbbxh" Oct 30 16:22:54.151515 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.149971 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b804fc37-0b07-4edf-8720-b31b5698b935-metrics-certs\") pod \"network-metrics-daemon-mbqwf\" (UID: \"b804fc37-0b07-4edf-8720-b31b5698b935\") " pod="openshift-multus/network-metrics-daemon-mbqwf" Oct 30 16:22:54.151515 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.150186 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc-metrics-tls\") pod \"dns-default-jvqzq\" (UID: \"7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc\") " pod="openshift-dns/dns-default-jvqzq" Oct 30 16:22:54.217174 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.217147 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mbqwf" Oct 30 16:22:54.222060 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.222031 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jvqzq" Oct 30 16:22:54.561398 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.561329 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mbqwf"] Oct 30 16:22:54.565760 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:54.565735 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb804fc37_0b07_4edf_8720_b31b5698b935.slice/crio-51b3b6a72d454d86ffbac8995f52f3ca0fe1b28568a2d0915d408557d81bb0af WatchSource:0}: Error finding container 51b3b6a72d454d86ffbac8995f52f3ca0fe1b28568a2d0915d408557d81bb0af: Status 404 returned error can't find the container with id 51b3b6a72d454d86ffbac8995f52f3ca0fe1b28568a2d0915d408557d81bb0af Oct 30 16:22:54.567829 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.567808 4493 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 16:22:54.593475 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.593452 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jvqzq"] Oct 30 16:22:54.597103 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:22:54.597082 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c4a0bb4_3e7f_4ea1_91e2_4ae2b79f0ccc.slice/crio-cd4d01736d853b050dac55248892e2166f07d05491a836785b33b7bfc61aadff WatchSource:0}: Error finding container cd4d01736d853b050dac55248892e2166f07d05491a836785b33b7bfc61aadff: Status 404 returned error can't find the container with id cd4d01736d853b050dac55248892e2166f07d05491a836785b33b7bfc61aadff Oct 30 16:22:54.651310 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.651284 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c24kh\" (UID: \"8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c\") " pod="openshift-insights/insights-runtime-extractor-c24kh" Oct 30 16:22:54.651443 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:22:54.651427 4493 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 16:22:54.651506 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:22:54.651495 4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c-insights-runtime-extractor-tls podName:8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c nodeName:}" failed. No retries permitted until 2026-10-30 16:22:55.651470531 +0000 UTC m=+2.878645626 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c24kh" (UID: "8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c") : secret "insights-runtime-extractor-tls" not found Oct 30 16:22:54.969705 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.969668 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jvqzq" event={"ID":"7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc","Type":"ContainerStarted","Data":"cd4d01736d853b050dac55248892e2166f07d05491a836785b33b7bfc61aadff"} Oct 30 16:22:54.971072 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:54.971047 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mbqwf" event={"ID":"b804fc37-0b07-4edf-8720-b31b5698b935","Type":"ContainerStarted","Data":"51b3b6a72d454d86ffbac8995f52f3ca0fe1b28568a2d0915d408557d81bb0af"} Oct 30 16:22:55.464975 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:55.464943 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-r5ppb_0e649944-a09d-44b1-91d2-fda00b30292d/dns-node-resolver/0.log" Oct 30 16:22:55.669518 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:55.669233 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c24kh\" (UID: \"8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c\") " pod="openshift-insights/insights-runtime-extractor-c24kh" Oct 30 16:22:55.669518 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:22:55.669367 4493 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 16:22:55.669518 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:22:55.669437 4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c-insights-runtime-extractor-tls podName:8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c nodeName:}" failed. No retries permitted until 2026-10-30 16:22:57.669417535 +0000 UTC m=+4.896592628 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c24kh" (UID: "8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c") : secret "insights-runtime-extractor-tls" not found Oct 30 16:22:55.976673 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:55.976635 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mbqwf" event={"ID":"b804fc37-0b07-4edf-8720-b31b5698b935","Type":"ContainerStarted","Data":"584cab5f5b9520c33d48f5440d92c575ddc80e5f5593302160b15d816df5bbfa"} Oct 30 16:22:55.977263 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:55.976735 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mbqwf" event={"ID":"b804fc37-0b07-4edf-8720-b31b5698b935","Type":"ContainerStarted","Data":"3e38944a15f97170d28d44b931d5da38f26a8cdf59fbe7d4df647b83f0de5d7c"} Oct 30 16:22:56.068828 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:56.063961 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jlr9c_17bb82ba-5262-47f7-abbc-99f1329500fa/node-ca/0.log" Oct 30 16:22:56.980967 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:56.980932 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jvqzq" event={"ID":"7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc","Type":"ContainerStarted","Data":"1a3c5413aef4042c76483e93cbd2da35cb3aae7abea313948a7c9e044545c5d2"} Oct 30 16:22:56.981352 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:56.980988 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jvqzq" event={"ID":"7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc","Type":"ContainerStarted","Data":"6a0cd36f9be7a2792df27f2765c022646dd5eacdc669ceb6cc17795fc56a5909"} Oct 30 16:22:57.064491 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:57.064468 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hlfhx_8a7b4cd2-b289-42b0-a54b-d868de87d7d3/serve-healthcheck-canary/0.log" Oct 30 16:22:57.491697 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:57.491672 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-fftpj_21fd140d-96ea-462e-9f8b-ef6d480f8407/migrator/0.log" Oct 30 16:22:57.682381 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:57.682355 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c24kh\" (UID: \"8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c\") " pod="openshift-insights/insights-runtime-extractor-c24kh" Oct 30 16:22:57.682526 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:22:57.682450 4493 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 16:22:57.682526 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:22:57.682497 4493 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c-insights-runtime-extractor-tls podName:8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c nodeName:}" failed. No retries permitted until 2026-10-30 16:23:01.682485673 +0000 UTC m=+8.909660751 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c24kh" (UID: "8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c") : secret "insights-runtime-extractor-tls" not found Oct 30 16:22:57.832858 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:57.832768 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:57.984779 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:57.984748 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-jvqzq" Oct 30 16:22:59.478514 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:59.478486 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:22:59.478918 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:22:59.478682 4493 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 16:23:00.492641 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:00.492552 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-426gj" Oct 30 16:23:01.710170 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:01.710128 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c24kh\" (UID: \"8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c\") " pod="openshift-insights/insights-runtime-extractor-c24kh" Oct 30 16:23:01.712511 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:01.712492 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c24kh\" (UID: \"8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c\") " pod="openshift-insights/insights-runtime-extractor-c24kh" Oct 30 16:23:01.724498 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:01.724476 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-c24kh" Oct 30 16:23:02.027940 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:02.027856 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-c24kh"] Oct 30 16:23:02.031666 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:23:02.031636 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b48ec5c_748a_4793_aa6d_4e4e4f8b3a0c.slice/crio-87b58d96eb5b4a8ab965c1a7ffe906e3a128c330643f153c9fc2b73fedd296c8 WatchSource:0}: Error finding container 87b58d96eb5b4a8ab965c1a7ffe906e3a128c330643f153c9fc2b73fedd296c8: Status 404 returned error can't find the container with id 87b58d96eb5b4a8ab965c1a7ffe906e3a128c330643f153c9fc2b73fedd296c8 Oct 30 16:23:02.194569 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:02.194540 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-56v9s" Oct 30 16:23:03.006589 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:03.006547 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c24kh" event={"ID":"8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c","Type":"ContainerStarted","Data":"25796b23952a18502b5e23023216b3f999a25786d41d514b9c6621d5e9ed031c"} Oct 30 16:23:03.007064 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:03.006638 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c24kh" event={"ID":"8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c","Type":"ContainerStarted","Data":"a0860e179ec386118304fbf35baad3d92dabe69d0cc153e11d4325317d519084"} Oct 30 16:23:03.007064 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:03.006659 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c24kh" event={"ID":"8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c","Type":"ContainerStarted","Data":"87b58d96eb5b4a8ab965c1a7ffe906e3a128c330643f153c9fc2b73fedd296c8"} Oct 30 16:23:05.014815 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:05.014767 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c24kh" event={"ID":"8b48ec5c-748a-4793-aa6d-4e4e4f8b3a0c","Type":"ContainerStarted","Data":"205aa88e2a2cbcf7fb0f36faf6601fe85e376b07c65e404519f98e4454ca4a80"} Oct 30 16:23:05.926341 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:05.926301 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-8b5bbf7-s2s69"] Oct 30 16:23:05.929539 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:05.929518 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8b5bbf7-s2s69" Oct 30 16:23:05.969611 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:05.969313 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 16:23:05.969611 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:05.969451 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 16:23:05.980078 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:05.980044 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 16:23:05.980078 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:05.980079 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 16:23:05.980439 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:05.980419 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-r5btp\"" Oct 30 16:23:05.980836 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:05.980819 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8b5bbf7-s2s69"] Oct 30 16:23:06.038609 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:06.038576 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2a272af5-e22c-4de5-8af9-28a810fe7035-trusted-ca\") pod \"image-registry-8b5bbf7-s2s69\" (UID: \"2a272af5-e22c-4de5-8af9-28a810fe7035\") " pod="openshift-image-registry/image-registry-8b5bbf7-s2s69" Oct 30 16:23:06.038609 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:06.038605 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njvgq\" (UniqueName: \"kubernetes.io/projected/2a272af5-e22c-4de5-8af9-28a810fe7035-kube-api-access-njvgq\") pod \"image-registry-8b5bbf7-s2s69\" (UID: \"2a272af5-e22c-4de5-8af9-28a810fe7035\") " pod="openshift-image-registry/image-registry-8b5bbf7-s2s69" Oct 30 16:23:06.039075 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:06.038631 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2a272af5-e22c-4de5-8af9-28a810fe7035-image-registry-private-configuration\") pod \"image-registry-8b5bbf7-s2s69\" (UID: \"2a272af5-e22c-4de5-8af9-28a810fe7035\") " pod="openshift-image-registry/image-registry-8b5bbf7-s2s69" Oct 30 16:23:06.039075 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:06.038685 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2a272af5-e22c-4de5-8af9-28a810fe7035-registry-tls\") pod \"image-registry-8b5bbf7-s2s69\" (UID: \"2a272af5-e22c-4de5-8af9-28a810fe7035\") " pod="openshift-image-registry/image-registry-8b5bbf7-s2s69" Oct 30 16:23:06.039075 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:06.038806 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2a272af5-e22c-4de5-8af9-28a810fe7035-ca-trust-extracted\") pod \"image-registry-8b5bbf7-s2s69\" (UID: \"2a272af5-e22c-4de5-8af9-28a810fe7035\") " pod="openshift-image-registry/image-registry-8b5bbf7-s2s69" Oct 30 16:23:06.039075 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:06.038858 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2a272af5-e22c-4de5-8af9-28a810fe7035-registry-certificates\") pod \"image-registry-8b5bbf7-s2s69\" (UID: \"2a272af5-e22c-4de5-8af9-28a810fe7035\") " pod="openshift-image-registry/image-registry-8b5bbf7-s2s69" Oct 30 16:23:06.039075 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:06.038929 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2a272af5-e22c-4de5-8af9-28a810fe7035-installation-pull-secrets\") pod \"image-registry-8b5bbf7-s2s69\" (UID: \"2a272af5-e22c-4de5-8af9-28a810fe7035\") " pod="openshift-image-registry/image-registry-8b5bbf7-s2s69" Oct 30 16:23:06.039075 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:06.038958 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2a272af5-e22c-4de5-8af9-28a810fe7035-bound-sa-token\") pod \"image-registry-8b5bbf7-s2s69\" (UID: \"2a272af5-e22c-4de5-8af9-28a810fe7035\") " pod="openshift-image-registry/image-registry-8b5bbf7-s2s69" Oct 30 16:23:06.139636 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:06.139603 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2a272af5-e22c-4de5-8af9-28a810fe7035-trusted-ca\") pod \"image-registry-8b5bbf7-s2s69\" (UID: \"2a272af5-e22c-4de5-8af9-28a810fe7035\") " pod="openshift-image-registry/image-registry-8b5bbf7-s2s69" Oct 30 16:23:06.139636 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:06.139635 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njvgq\" (UniqueName: \"kubernetes.io/projected/2a272af5-e22c-4de5-8af9-28a810fe7035-kube-api-access-njvgq\") pod \"image-registry-8b5bbf7-s2s69\" (UID: \"2a272af5-e22c-4de5-8af9-28a810fe7035\") " pod="openshift-image-registry/image-registry-8b5bbf7-s2s69" Oct 30 16:23:06.139886 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:06.139657 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2a272af5-e22c-4de5-8af9-28a810fe7035-image-registry-private-configuration\") pod \"image-registry-8b5bbf7-s2s69\" (UID: \"2a272af5-e22c-4de5-8af9-28a810fe7035\") " pod="openshift-image-registry/image-registry-8b5bbf7-s2s69" Oct 30 16:23:06.139886 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:06.139820 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2a272af5-e22c-4de5-8af9-28a810fe7035-registry-tls\") pod \"image-registry-8b5bbf7-s2s69\" (UID: \"2a272af5-e22c-4de5-8af9-28a810fe7035\") " pod="openshift-image-registry/image-registry-8b5bbf7-s2s69" Oct 30 16:23:06.139886 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:06.139858 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2a272af5-e22c-4de5-8af9-28a810fe7035-ca-trust-extracted\") pod \"image-registry-8b5bbf7-s2s69\" (UID: \"2a272af5-e22c-4de5-8af9-28a810fe7035\") " pod="openshift-image-registry/image-registry-8b5bbf7-s2s69" Oct 30 16:23:06.139886 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:06.139884 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2a272af5-e22c-4de5-8af9-28a810fe7035-registry-certificates\") pod \"image-registry-8b5bbf7-s2s69\" (UID: \"2a272af5-e22c-4de5-8af9-28a810fe7035\") " pod="openshift-image-registry/image-registry-8b5bbf7-s2s69" Oct 30 16:23:06.140078 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:06.139942 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2a272af5-e22c-4de5-8af9-28a810fe7035-installation-pull-secrets\") pod \"image-registry-8b5bbf7-s2s69\" (UID: \"2a272af5-e22c-4de5-8af9-28a810fe7035\") " pod="openshift-image-registry/image-registry-8b5bbf7-s2s69" Oct 30 16:23:06.140078 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:06.140016 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2a272af5-e22c-4de5-8af9-28a810fe7035-bound-sa-token\") pod \"image-registry-8b5bbf7-s2s69\" (UID: \"2a272af5-e22c-4de5-8af9-28a810fe7035\") " pod="openshift-image-registry/image-registry-8b5bbf7-s2s69" Oct 30 16:23:06.140344 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:06.140294 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2a272af5-e22c-4de5-8af9-28a810fe7035-ca-trust-extracted\") pod \"image-registry-8b5bbf7-s2s69\" (UID: \"2a272af5-e22c-4de5-8af9-28a810fe7035\") " pod="openshift-image-registry/image-registry-8b5bbf7-s2s69" Oct 30 16:23:06.140817 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:06.140777 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2a272af5-e22c-4de5-8af9-28a810fe7035-trusted-ca\") pod \"image-registry-8b5bbf7-s2s69\" (UID: \"2a272af5-e22c-4de5-8af9-28a810fe7035\") " pod="openshift-image-registry/image-registry-8b5bbf7-s2s69" Oct 30 16:23:06.140919 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:06.140897 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2a272af5-e22c-4de5-8af9-28a810fe7035-registry-certificates\") pod \"image-registry-8b5bbf7-s2s69\" (UID: \"2a272af5-e22c-4de5-8af9-28a810fe7035\") " pod="openshift-image-registry/image-registry-8b5bbf7-s2s69" Oct 30 16:23:06.142386 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:06.142367 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2a272af5-e22c-4de5-8af9-28a810fe7035-registry-tls\") pod \"image-registry-8b5bbf7-s2s69\" (UID: \"2a272af5-e22c-4de5-8af9-28a810fe7035\") " pod="openshift-image-registry/image-registry-8b5bbf7-s2s69" Oct 30 16:23:06.142917 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:06.142897 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2a272af5-e22c-4de5-8af9-28a810fe7035-image-registry-private-configuration\") pod \"image-registry-8b5bbf7-s2s69\" (UID: \"2a272af5-e22c-4de5-8af9-28a810fe7035\") " pod="openshift-image-registry/image-registry-8b5bbf7-s2s69" Oct 30 16:23:06.142987 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:06.142897 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2a272af5-e22c-4de5-8af9-28a810fe7035-installation-pull-secrets\") pod \"image-registry-8b5bbf7-s2s69\" (UID: \"2a272af5-e22c-4de5-8af9-28a810fe7035\") " pod="openshift-image-registry/image-registry-8b5bbf7-s2s69" Oct 30 16:23:06.164265 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:06.164240 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2a272af5-e22c-4de5-8af9-28a810fe7035-bound-sa-token\") pod \"image-registry-8b5bbf7-s2s69\" (UID: \"2a272af5-e22c-4de5-8af9-28a810fe7035\") " pod="openshift-image-registry/image-registry-8b5bbf7-s2s69" Oct 30 16:23:06.164476 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:06.164458 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njvgq\" (UniqueName: \"kubernetes.io/projected/2a272af5-e22c-4de5-8af9-28a810fe7035-kube-api-access-njvgq\") pod \"image-registry-8b5bbf7-s2s69\" (UID: \"2a272af5-e22c-4de5-8af9-28a810fe7035\") " pod="openshift-image-registry/image-registry-8b5bbf7-s2s69" Oct 30 16:23:06.240651 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:06.240570 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8b5bbf7-s2s69" Oct 30 16:23:06.562802 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:06.562683 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8b5bbf7-s2s69"] Oct 30 16:23:06.566350 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:23:06.566324 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a272af5_e22c_4de5_8af9_28a810fe7035.slice/crio-53e8ce30d0eb130b3bd3520ba07ba9cb765aa63a5acd8bbd8785f3b503f43c24 WatchSource:0}: Error finding container 53e8ce30d0eb130b3bd3520ba07ba9cb765aa63a5acd8bbd8785f3b503f43c24: Status 404 returned error can't find the container with id 53e8ce30d0eb130b3bd3520ba07ba9cb765aa63a5acd8bbd8785f3b503f43c24 Oct 30 16:23:07.022804 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:07.022761 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8b5bbf7-s2s69" event={"ID":"2a272af5-e22c-4de5-8af9-28a810fe7035","Type":"ContainerStarted","Data":"f579fea837a791ffb631e977369e7bc639642d78410c6f8f23c6b9306558ef8d"} Oct 30 16:23:07.023000 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:07.022823 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8b5bbf7-s2s69" event={"ID":"2a272af5-e22c-4de5-8af9-28a810fe7035","Type":"ContainerStarted","Data":"53e8ce30d0eb130b3bd3520ba07ba9cb765aa63a5acd8bbd8785f3b503f43c24"} Oct 30 16:23:07.023000 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:07.022904 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-8b5bbf7-s2s69" Oct 30 16:23:07.067550 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:07.067487 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-8b5bbf7-s2s69" podStartSLOduration=2.067470426 podStartE2EDuration="2.067470426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:23:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:23:07.067089705 +0000 UTC m=+14.294264816" watchObservedRunningTime="2026-10-30 16:23:07.067470426 +0000 UTC m=+14.294645517" Oct 30 16:23:07.989497 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:07.989468 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-jvqzq" Oct 30 16:23:08.737424 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:08.737392 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-pg4kq" Oct 30 16:23:09.614499 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:09.614464 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8bnmd"] Oct 30 16:23:09.618765 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:09.618750 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8bnmd" Oct 30 16:23:09.635098 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:09.635075 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Oct 30 16:23:09.647034 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:09.647002 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-rk9zh\"" Oct 30 16:23:09.647474 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:09.647454 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8bnmd"] Oct 30 16:23:09.766005 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:09.765970 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a3b1f16c-b2cc-4f04-92a9-b424da398fe7-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-8bnmd\" (UID: \"a3b1f16c-b2cc-4f04-92a9-b424da398fe7\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8bnmd" Oct 30 16:23:09.866368 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:09.866294 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a3b1f16c-b2cc-4f04-92a9-b424da398fe7-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-8bnmd\" (UID: \"a3b1f16c-b2cc-4f04-92a9-b424da398fe7\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8bnmd" Oct 30 16:23:09.868898 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:09.868863 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a3b1f16c-b2cc-4f04-92a9-b424da398fe7-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-8bnmd\" (UID: \"a3b1f16c-b2cc-4f04-92a9-b424da398fe7\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8bnmd" Oct 30 16:23:09.930223 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:09.930195 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8bnmd" Oct 30 16:23:10.223851 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:10.223819 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8bnmd"] Oct 30 16:23:10.227075 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:23:10.227048 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3b1f16c_b2cc_4f04_92a9_b424da398fe7.slice/crio-67333e1030bf669870dc21bc4ca29b9b622ae2e983bd32962ffab18f39fe9a2f WatchSource:0}: Error finding container 67333e1030bf669870dc21bc4ca29b9b622ae2e983bd32962ffab18f39fe9a2f: Status 404 returned error can't find the container with id 67333e1030bf669870dc21bc4ca29b9b622ae2e983bd32962ffab18f39fe9a2f Oct 30 16:23:11.036619 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:11.036576 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8bnmd" event={"ID":"a3b1f16c-b2cc-4f04-92a9-b424da398fe7","Type":"ContainerStarted","Data":"67333e1030bf669870dc21bc4ca29b9b622ae2e983bd32962ffab18f39fe9a2f"} Oct 30 16:23:12.041008 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:12.040977 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8bnmd" event={"ID":"a3b1f16c-b2cc-4f04-92a9-b424da398fe7","Type":"ContainerStarted","Data":"289b8c0b25e4c15e92982631d3531f5bc9a3aba4cafc410ec6da8d3ad2480f15"} Oct 30 16:23:12.041394 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:12.041160 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8bnmd" Oct 30 16:23:12.045557 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:12.045535 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8bnmd" Oct 30 16:23:12.081092 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:12.080974 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-8bnmd" podStartSLOduration=1.975196094 podStartE2EDuration="3.080961436s" totalImagesPullingTime="1.105765342s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:23:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:23:12.08029256 +0000 UTC m=+19.307467659" watchObservedRunningTime="2026-10-30 16:23:12.080961436 +0000 UTC m=+19.308136537" Oct 30 16:23:17.442020 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.441984 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-2hmkq"] Oct 30 16:23:17.450799 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.450757 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2hmkq" Oct 30 16:23:17.482551 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.482291 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 16:23:17.482551 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.482397 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 16:23:17.482551 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.482423 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 16:23:17.482551 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.482397 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 16:23:17.482866 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.482608 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 16:23:17.482866 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.482704 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-mk66n\"" Oct 30 16:23:17.483550 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.483525 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-2hmkq"] Oct 30 16:23:17.495030 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.495009 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-d5phv"] Oct 30 16:23:17.505983 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.505965 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-v27z6"] Oct 30 16:23:17.506098 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.506081 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-d5phv" Oct 30 16:23:17.509296 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.509277 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-v27z6" Oct 30 16:23:17.515215 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.515197 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-d5phv"] Oct 30 16:23:17.549856 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.549833 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 16:23:17.550006 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.549846 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 16:23:17.550080 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.549891 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 16:23:17.550080 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.549914 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 16:23:17.550080 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.549939 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-cmzmt\"" Oct 30 16:23:17.550218 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.550115 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-grp8r\"" Oct 30 16:23:17.570713 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.570696 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 16:23:17.570830 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.570811 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 16:23:17.616657 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.616637 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4471fff4-f324-4b99-96bd-8b4feeb518a2-node-exporter-accelerators-collector-config\") pod \"node-exporter-v27z6\" (UID: \"4471fff4-f324-4b99-96bd-8b4feeb518a2\") " pod="openshift-monitoring/node-exporter-v27z6" Oct 30 16:23:17.616765 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.616666 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7cdc9574-e634-408a-8084-b7263520724d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-2hmkq\" (UID: \"7cdc9574-e634-408a-8084-b7263520724d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2hmkq" Oct 30 16:23:17.616765 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.616686 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cdc9574-e634-408a-8084-b7263520724d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-2hmkq\" (UID: \"7cdc9574-e634-408a-8084-b7263520724d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2hmkq" Oct 30 16:23:17.616765 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.616702 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4a801701-a1db-4feb-a49e-018c3fd03038-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-d5phv\" (UID: \"4a801701-a1db-4feb-a49e-018c3fd03038\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-d5phv" Oct 30 16:23:17.616765 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.616752 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4471fff4-f324-4b99-96bd-8b4feeb518a2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-v27z6\" (UID: \"4471fff4-f324-4b99-96bd-8b4feeb518a2\") " pod="openshift-monitoring/node-exporter-v27z6" Oct 30 16:23:17.616949 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.616822 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwmvr\" (UniqueName: \"kubernetes.io/projected/4a801701-a1db-4feb-a49e-018c3fd03038-kube-api-access-hwmvr\") pod \"kube-state-metrics-5dc8d7d89b-d5phv\" (UID: \"4a801701-a1db-4feb-a49e-018c3fd03038\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-d5phv" Oct 30 16:23:17.616949 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.616867 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4a801701-a1db-4feb-a49e-018c3fd03038-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-d5phv\" (UID: \"4a801701-a1db-4feb-a49e-018c3fd03038\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-d5phv" Oct 30 16:23:17.616949 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.616942 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7cdc9574-e634-408a-8084-b7263520724d-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-2hmkq\" (UID: \"7cdc9574-e634-408a-8084-b7263520724d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2hmkq" Oct 30 16:23:17.617058 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.616959 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a801701-a1db-4feb-a49e-018c3fd03038-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-d5phv\" (UID: \"4a801701-a1db-4feb-a49e-018c3fd03038\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-d5phv" Oct 30 16:23:17.617058 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.616977 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4a801701-a1db-4feb-a49e-018c3fd03038-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-d5phv\" (UID: \"4a801701-a1db-4feb-a49e-018c3fd03038\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-d5phv" Oct 30 16:23:17.617058 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.616992 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4471fff4-f324-4b99-96bd-8b4feeb518a2-node-exporter-wtmp\") pod \"node-exporter-v27z6\" (UID: \"4471fff4-f324-4b99-96bd-8b4feeb518a2\") " pod="openshift-monitoring/node-exporter-v27z6" Oct 30 16:23:17.617058 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.617010 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4471fff4-f324-4b99-96bd-8b4feeb518a2-metrics-client-ca\") pod \"node-exporter-v27z6\" (UID: \"4471fff4-f324-4b99-96bd-8b4feeb518a2\") " pod="openshift-monitoring/node-exporter-v27z6" Oct 30 16:23:17.617205 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.617051 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4a801701-a1db-4feb-a49e-018c3fd03038-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-d5phv\" (UID: \"4a801701-a1db-4feb-a49e-018c3fd03038\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-d5phv" Oct 30 16:23:17.617205 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.617110 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rlbk\" (UniqueName: \"kubernetes.io/projected/4471fff4-f324-4b99-96bd-8b4feeb518a2-kube-api-access-8rlbk\") pod \"node-exporter-v27z6\" (UID: \"4471fff4-f324-4b99-96bd-8b4feeb518a2\") " pod="openshift-monitoring/node-exporter-v27z6" Oct 30 16:23:17.617205 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.617128 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4471fff4-f324-4b99-96bd-8b4feeb518a2-sys\") pod \"node-exporter-v27z6\" (UID: \"4471fff4-f324-4b99-96bd-8b4feeb518a2\") " pod="openshift-monitoring/node-exporter-v27z6" Oct 30 16:23:17.617205 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.617142 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4471fff4-f324-4b99-96bd-8b4feeb518a2-root\") pod \"node-exporter-v27z6\" (UID: \"4471fff4-f324-4b99-96bd-8b4feeb518a2\") " pod="openshift-monitoring/node-exporter-v27z6" Oct 30 16:23:17.617205 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.617159 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4471fff4-f324-4b99-96bd-8b4feeb518a2-node-exporter-tls\") pod \"node-exporter-v27z6\" (UID: \"4471fff4-f324-4b99-96bd-8b4feeb518a2\") " pod="openshift-monitoring/node-exporter-v27z6" Oct 30 16:23:17.617205 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.617175 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mnxl\" (UniqueName: \"kubernetes.io/projected/7cdc9574-e634-408a-8084-b7263520724d-kube-api-access-7mnxl\") pod \"openshift-state-metrics-7bd7489957-2hmkq\" (UID: \"7cdc9574-e634-408a-8084-b7263520724d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2hmkq" Oct 30 16:23:17.617450 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.617266 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4471fff4-f324-4b99-96bd-8b4feeb518a2-node-exporter-textfile\") pod \"node-exporter-v27z6\" (UID: \"4471fff4-f324-4b99-96bd-8b4feeb518a2\") " pod="openshift-monitoring/node-exporter-v27z6" Oct 30 16:23:17.718537 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.718476 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4471fff4-f324-4b99-96bd-8b4feeb518a2-sys\") pod \"node-exporter-v27z6\" (UID: \"4471fff4-f324-4b99-96bd-8b4feeb518a2\") " pod="openshift-monitoring/node-exporter-v27z6" Oct 30 16:23:17.718537 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.718507 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4471fff4-f324-4b99-96bd-8b4feeb518a2-root\") pod \"node-exporter-v27z6\" (UID: \"4471fff4-f324-4b99-96bd-8b4feeb518a2\") " pod="openshift-monitoring/node-exporter-v27z6" Oct 30 16:23:17.718537 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.718534 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4471fff4-f324-4b99-96bd-8b4feeb518a2-node-exporter-tls\") pod \"node-exporter-v27z6\" (UID: \"4471fff4-f324-4b99-96bd-8b4feeb518a2\") " pod="openshift-monitoring/node-exporter-v27z6" Oct 30 16:23:17.718764 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.718559 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mnxl\" (UniqueName: \"kubernetes.io/projected/7cdc9574-e634-408a-8084-b7263520724d-kube-api-access-7mnxl\") pod \"openshift-state-metrics-7bd7489957-2hmkq\" (UID: \"7cdc9574-e634-408a-8084-b7263520724d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2hmkq" Oct 30 16:23:17.718764 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.718593 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4471fff4-f324-4b99-96bd-8b4feeb518a2-node-exporter-textfile\") pod \"node-exporter-v27z6\" (UID: \"4471fff4-f324-4b99-96bd-8b4feeb518a2\") " pod="openshift-monitoring/node-exporter-v27z6" Oct 30 16:23:17.718764 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.718593 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4471fff4-f324-4b99-96bd-8b4feeb518a2-sys\") pod \"node-exporter-v27z6\" (UID: \"4471fff4-f324-4b99-96bd-8b4feeb518a2\") " pod="openshift-monitoring/node-exporter-v27z6" Oct 30 16:23:17.718764 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.718607 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4471fff4-f324-4b99-96bd-8b4feeb518a2-root\") pod \"node-exporter-v27z6\" (UID: \"4471fff4-f324-4b99-96bd-8b4feeb518a2\") " pod="openshift-monitoring/node-exporter-v27z6" Oct 30 16:23:17.718764 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.718621 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4471fff4-f324-4b99-96bd-8b4feeb518a2-node-exporter-accelerators-collector-config\") pod \"node-exporter-v27z6\" (UID: \"4471fff4-f324-4b99-96bd-8b4feeb518a2\") " pod="openshift-monitoring/node-exporter-v27z6" Oct 30 16:23:17.718764 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.718646 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7cdc9574-e634-408a-8084-b7263520724d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-2hmkq\" (UID: \"7cdc9574-e634-408a-8084-b7263520724d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2hmkq" Oct 30 16:23:17.718764 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.718669 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cdc9574-e634-408a-8084-b7263520724d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-2hmkq\" (UID: \"7cdc9574-e634-408a-8084-b7263520724d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2hmkq" Oct 30 16:23:17.718764 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.718692 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4a801701-a1db-4feb-a49e-018c3fd03038-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-d5phv\" (UID: \"4a801701-a1db-4feb-a49e-018c3fd03038\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-d5phv" Oct 30 16:23:17.718764 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.718716 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4471fff4-f324-4b99-96bd-8b4feeb518a2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-v27z6\" (UID: \"4471fff4-f324-4b99-96bd-8b4feeb518a2\") " pod="openshift-monitoring/node-exporter-v27z6" Oct 30 16:23:17.718764 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.718741 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwmvr\" (UniqueName: \"kubernetes.io/projected/4a801701-a1db-4feb-a49e-018c3fd03038-kube-api-access-hwmvr\") pod \"kube-state-metrics-5dc8d7d89b-d5phv\" (UID: \"4a801701-a1db-4feb-a49e-018c3fd03038\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-d5phv" Oct 30 16:23:17.719392 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.718773 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4a801701-a1db-4feb-a49e-018c3fd03038-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-d5phv\" (UID: \"4a801701-a1db-4feb-a49e-018c3fd03038\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-d5phv" Oct 30 16:23:17.719392 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.718861 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7cdc9574-e634-408a-8084-b7263520724d-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-2hmkq\" (UID: \"7cdc9574-e634-408a-8084-b7263520724d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2hmkq" Oct 30 16:23:17.719392 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.718890 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a801701-a1db-4feb-a49e-018c3fd03038-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-d5phv\" (UID: \"4a801701-a1db-4feb-a49e-018c3fd03038\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-d5phv" Oct 30 16:23:17.719392 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.718921 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4a801701-a1db-4feb-a49e-018c3fd03038-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-d5phv\" (UID: \"4a801701-a1db-4feb-a49e-018c3fd03038\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-d5phv" Oct 30 16:23:17.719392 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.718957 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4471fff4-f324-4b99-96bd-8b4feeb518a2-node-exporter-wtmp\") pod \"node-exporter-v27z6\" (UID: \"4471fff4-f324-4b99-96bd-8b4feeb518a2\") " pod="openshift-monitoring/node-exporter-v27z6" Oct 30 16:23:17.719392 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.718988 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4471fff4-f324-4b99-96bd-8b4feeb518a2-metrics-client-ca\") pod \"node-exporter-v27z6\" (UID: \"4471fff4-f324-4b99-96bd-8b4feeb518a2\") " pod="openshift-monitoring/node-exporter-v27z6" Oct 30 16:23:17.719392 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.718994 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4471fff4-f324-4b99-96bd-8b4feeb518a2-node-exporter-textfile\") pod \"node-exporter-v27z6\" (UID: \"4471fff4-f324-4b99-96bd-8b4feeb518a2\") " pod="openshift-monitoring/node-exporter-v27z6" Oct 30 16:23:17.719392 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.719024 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4a801701-a1db-4feb-a49e-018c3fd03038-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-d5phv\" (UID: \"4a801701-a1db-4feb-a49e-018c3fd03038\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-d5phv" Oct 30 16:23:17.719392 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.719073 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rlbk\" (UniqueName: \"kubernetes.io/projected/4471fff4-f324-4b99-96bd-8b4feeb518a2-kube-api-access-8rlbk\") pod \"node-exporter-v27z6\" (UID: \"4471fff4-f324-4b99-96bd-8b4feeb518a2\") " pod="openshift-monitoring/node-exporter-v27z6" Oct 30 16:23:17.719392 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.719261 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4a801701-a1db-4feb-a49e-018c3fd03038-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-d5phv\" (UID: \"4a801701-a1db-4feb-a49e-018c3fd03038\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-d5phv" Oct 30 16:23:17.720036 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.719478 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4471fff4-f324-4b99-96bd-8b4feeb518a2-node-exporter-accelerators-collector-config\") pod \"node-exporter-v27z6\" (UID: \"4471fff4-f324-4b99-96bd-8b4feeb518a2\") " pod="openshift-monitoring/node-exporter-v27z6" Oct 30 16:23:17.720036 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.719806 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4471fff4-f324-4b99-96bd-8b4feeb518a2-node-exporter-wtmp\") pod \"node-exporter-v27z6\" (UID: \"4471fff4-f324-4b99-96bd-8b4feeb518a2\") " pod="openshift-monitoring/node-exporter-v27z6" Oct 30 16:23:17.720140 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.720070 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4471fff4-f324-4b99-96bd-8b4feeb518a2-metrics-client-ca\") pod \"node-exporter-v27z6\" (UID: \"4471fff4-f324-4b99-96bd-8b4feeb518a2\") " pod="openshift-monitoring/node-exporter-v27z6" Oct 30 16:23:17.720140 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.720101 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7cdc9574-e634-408a-8084-b7263520724d-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-2hmkq\" (UID: \"7cdc9574-e634-408a-8084-b7263520724d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2hmkq" Oct 30 16:23:17.720239 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.720215 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a801701-a1db-4feb-a49e-018c3fd03038-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-d5phv\" (UID: \"4a801701-a1db-4feb-a49e-018c3fd03038\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-d5phv" Oct 30 16:23:17.720486 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.720442 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4a801701-a1db-4feb-a49e-018c3fd03038-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-d5phv\" (UID: \"4a801701-a1db-4feb-a49e-018c3fd03038\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-d5phv" Oct 30 16:23:17.721986 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.721578 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4a801701-a1db-4feb-a49e-018c3fd03038-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-d5phv\" (UID: \"4a801701-a1db-4feb-a49e-018c3fd03038\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-d5phv" Oct 30 16:23:17.722121 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.722099 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cdc9574-e634-408a-8084-b7263520724d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-2hmkq\" (UID: \"7cdc9574-e634-408a-8084-b7263520724d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2hmkq" Oct 30 16:23:17.722533 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.722513 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4471fff4-f324-4b99-96bd-8b4feeb518a2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-v27z6\" (UID: \"4471fff4-f324-4b99-96bd-8b4feeb518a2\") " pod="openshift-monitoring/node-exporter-v27z6" Oct 30 16:23:17.722769 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.722748 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4471fff4-f324-4b99-96bd-8b4feeb518a2-node-exporter-tls\") pod \"node-exporter-v27z6\" (UID: \"4471fff4-f324-4b99-96bd-8b4feeb518a2\") " pod="openshift-monitoring/node-exporter-v27z6" Oct 30 16:23:17.722965 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.722946 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7cdc9574-e634-408a-8084-b7263520724d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-2hmkq\" (UID: \"7cdc9574-e634-408a-8084-b7263520724d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2hmkq" Oct 30 16:23:17.723031 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.722974 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4a801701-a1db-4feb-a49e-018c3fd03038-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-d5phv\" (UID: \"4a801701-a1db-4feb-a49e-018c3fd03038\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-d5phv" Oct 30 16:23:17.741323 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.741302 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mnxl\" (UniqueName: \"kubernetes.io/projected/7cdc9574-e634-408a-8084-b7263520724d-kube-api-access-7mnxl\") pod \"openshift-state-metrics-7bd7489957-2hmkq\" (UID: \"7cdc9574-e634-408a-8084-b7263520724d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2hmkq" Oct 30 16:23:17.741840 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.741823 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rlbk\" (UniqueName: \"kubernetes.io/projected/4471fff4-f324-4b99-96bd-8b4feeb518a2-kube-api-access-8rlbk\") pod \"node-exporter-v27z6\" (UID: \"4471fff4-f324-4b99-96bd-8b4feeb518a2\") " pod="openshift-monitoring/node-exporter-v27z6" Oct 30 16:23:17.742013 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.741997 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwmvr\" (UniqueName: \"kubernetes.io/projected/4a801701-a1db-4feb-a49e-018c3fd03038-kube-api-access-hwmvr\") pod \"kube-state-metrics-5dc8d7d89b-d5phv\" (UID: \"4a801701-a1db-4feb-a49e-018c3fd03038\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-d5phv" Oct 30 16:23:17.765850 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.765829 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2hmkq" Oct 30 16:23:17.823718 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.823687 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-d5phv" Oct 30 16:23:17.830071 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:17.830046 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-v27z6" Oct 30 16:23:17.844508 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:23:17.844482 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4471fff4_f324_4b99_96bd_8b4feeb518a2.slice/crio-e69ffd3ae65b4f9f659cb69657a890c27a854419fb11e39e8fad0cd45b1f0766 WatchSource:0}: Error finding container e69ffd3ae65b4f9f659cb69657a890c27a854419fb11e39e8fad0cd45b1f0766: Status 404 returned error can't find the container with id e69ffd3ae65b4f9f659cb69657a890c27a854419fb11e39e8fad0cd45b1f0766 Oct 30 16:23:18.060610 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:18.060538 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v27z6" event={"ID":"4471fff4-f324-4b99-96bd-8b4feeb518a2","Type":"ContainerStarted","Data":"e69ffd3ae65b4f9f659cb69657a890c27a854419fb11e39e8fad0cd45b1f0766"} Oct 30 16:23:18.071129 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:18.071057 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-2hmkq"] Oct 30 16:23:18.074379 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:23:18.074344 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cdc9574_e634_408a_8084_b7263520724d.slice/crio-eadebfd32edbdb2406ff617e97bbd2733bf6de4854b4a6e82d69dbbd94c08147 WatchSource:0}: Error finding container eadebfd32edbdb2406ff617e97bbd2733bf6de4854b4a6e82d69dbbd94c08147: Status 404 returned error can't find the container with id eadebfd32edbdb2406ff617e97bbd2733bf6de4854b4a6e82d69dbbd94c08147 Oct 30 16:23:18.133232 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:18.133210 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-d5phv"] Oct 30 16:23:18.141105 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:23:18.141076 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a801701_a1db_4feb_a49e_018c3fd03038.slice/crio-197eec510d0d1e3c4446dd2b7904fef04ac7d1066cf916c4792540f24692078c WatchSource:0}: Error finding container 197eec510d0d1e3c4446dd2b7904fef04ac7d1066cf916c4792540f24692078c: Status 404 returned error can't find the container with id 197eec510d0d1e3c4446dd2b7904fef04ac7d1066cf916c4792540f24692078c Oct 30 16:23:19.066293 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:19.066213 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-d5phv" event={"ID":"4a801701-a1db-4feb-a49e-018c3fd03038","Type":"ContainerStarted","Data":"197eec510d0d1e3c4446dd2b7904fef04ac7d1066cf916c4792540f24692078c"} Oct 30 16:23:19.068201 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:19.068171 4493 generic.go:356] "Generic (PLEG): container finished" podID="4471fff4-f324-4b99-96bd-8b4feeb518a2" containerID="f56c01074b97d537b1b11bf1e6d5545b4e3aeb10c513093a1ebdd855265da0f1" exitCode=0 Oct 30 16:23:19.068321 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:19.068247 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v27z6" event={"ID":"4471fff4-f324-4b99-96bd-8b4feeb518a2","Type":"ContainerDied","Data":"f56c01074b97d537b1b11bf1e6d5545b4e3aeb10c513093a1ebdd855265da0f1"} Oct 30 16:23:19.070999 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:19.070963 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2hmkq" event={"ID":"7cdc9574-e634-408a-8084-b7263520724d","Type":"ContainerStarted","Data":"1ad50c36ec2314f96009901c49c3c2448ea78494c91c58a7140fb4ad66743bfd"} Oct 30 16:23:19.071097 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:19.071026 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2hmkq" event={"ID":"7cdc9574-e634-408a-8084-b7263520724d","Type":"ContainerStarted","Data":"e6e9d520c79e0c99903ec85f1ae174ac9c249b686397394daa2861d761c95858"} Oct 30 16:23:19.071097 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:19.071056 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2hmkq" event={"ID":"7cdc9574-e634-408a-8084-b7263520724d","Type":"ContainerStarted","Data":"eadebfd32edbdb2406ff617e97bbd2733bf6de4854b4a6e82d69dbbd94c08147"} Oct 30 16:23:20.075945 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:20.075909 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v27z6" event={"ID":"4471fff4-f324-4b99-96bd-8b4feeb518a2","Type":"ContainerStarted","Data":"f085087077a3a6b22fb429c63110d1dcda0459dc77269717eb02378cd8d9fcbd"} Oct 30 16:23:20.076394 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:20.075971 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v27z6" event={"ID":"4471fff4-f324-4b99-96bd-8b4feeb518a2","Type":"ContainerStarted","Data":"77d4e626ddc9a63ee019d22f9320fdba76d8935a456ccb400a5aaf598970b312"} Oct 30 16:23:20.077974 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:20.077946 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2hmkq" event={"ID":"7cdc9574-e634-408a-8084-b7263520724d","Type":"ContainerStarted","Data":"6336b4de2a5edc71616e8a0284edb88834c07ed759cb5e889a663253e159ac89"} Oct 30 16:23:20.079885 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:20.079863 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-d5phv" event={"ID":"4a801701-a1db-4feb-a49e-018c3fd03038","Type":"ContainerStarted","Data":"31a36535242667a04cabdbbceef3ef5645341f73c31422878bc77f7ceba3e8ec"} Oct 30 16:23:20.079996 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:20.079911 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-d5phv" event={"ID":"4a801701-a1db-4feb-a49e-018c3fd03038","Type":"ContainerStarted","Data":"76a94f62884b22bac9c39299eaf79370c663fb83c4e0e5578ce6c2dfa19f8fa5"} Oct 30 16:23:20.079996 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:20.079941 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-d5phv" event={"ID":"4a801701-a1db-4feb-a49e-018c3fd03038","Type":"ContainerStarted","Data":"bb612f78c6ff9e587360dcd3efc8260aad41980d6db56895a10058ee3ba1cb32"} Oct 30 16:23:20.120075 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:20.120038 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-v27z6" podStartSLOduration=2.184835666 podStartE2EDuration="3.120026026s" totalImagesPullingTime="844.858495ms" totalInitContainerRuntime="90.331865ms" isStatefulPod=true podCreationTimestamp="2026-10-30 16:23:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:23:20.119055177 +0000 UTC m=+27.346230300" watchObservedRunningTime="2026-10-30 16:23:20.120026026 +0000 UTC m=+27.347201122" Oct 30 16:23:20.161947 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:20.161877 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-d5phv" podStartSLOduration=1.808260497 podStartE2EDuration="3.161861346s" totalImagesPullingTime="1.353600849s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:23:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:23:20.16116829 +0000 UTC m=+27.388343390" watchObservedRunningTime="2026-10-30 16:23:20.161861346 +0000 UTC m=+27.389036447" Oct 30 16:23:20.204533 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:20.204483 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-2hmkq" podStartSLOduration=2.108102027 podStartE2EDuration="3.204469984s" totalImagesPullingTime="1.096367957s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:23:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:23:20.203766293 +0000 UTC m=+27.430941394" watchObservedRunningTime="2026-10-30 16:23:20.204469984 +0000 UTC m=+27.431645084" Oct 30 16:23:22.741236 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:22.741201 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-64fdfd6547-66n2g"] Oct 30 16:23:22.768971 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:22.768937 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-64fdfd6547-66n2g"] Oct 30 16:23:22.769110 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:22.769076 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-64fdfd6547-66n2g" Oct 30 16:23:22.785681 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:22.785658 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 30 16:23:22.809554 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:22.809531 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 30 16:23:22.809675 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:22.809557 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-pqwjr\"" Oct 30 16:23:22.809675 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:22.809574 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 30 16:23:22.809675 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:22.809596 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 30 16:23:22.809675 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:22.809641 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 30 16:23:22.809960 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:22.809932 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 30 16:23:22.962346 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:22.962317 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4709930f-fc73-4b82-951f-a428ad89313a-metrics-client-ca\") pod \"telemeter-client-64fdfd6547-66n2g\" (UID: \"4709930f-fc73-4b82-951f-a428ad89313a\") " pod="openshift-monitoring/telemeter-client-64fdfd6547-66n2g" Oct 30 16:23:22.962496 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:22.962353 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/4709930f-fc73-4b82-951f-a428ad89313a-telemeter-client-tls\") pod \"telemeter-client-64fdfd6547-66n2g\" (UID: \"4709930f-fc73-4b82-951f-a428ad89313a\") " pod="openshift-monitoring/telemeter-client-64fdfd6547-66n2g" Oct 30 16:23:22.962496 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:22.962381 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zvqm\" (UniqueName: \"kubernetes.io/projected/4709930f-fc73-4b82-951f-a428ad89313a-kube-api-access-5zvqm\") pod \"telemeter-client-64fdfd6547-66n2g\" (UID: \"4709930f-fc73-4b82-951f-a428ad89313a\") " pod="openshift-monitoring/telemeter-client-64fdfd6547-66n2g" Oct 30 16:23:22.962496 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:22.962480 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4709930f-fc73-4b82-951f-a428ad89313a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-64fdfd6547-66n2g\" (UID: \"4709930f-fc73-4b82-951f-a428ad89313a\") " pod="openshift-monitoring/telemeter-client-64fdfd6547-66n2g" Oct 30 16:23:22.962667 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:22.962562 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/4709930f-fc73-4b82-951f-a428ad89313a-federate-client-tls\") pod \"telemeter-client-64fdfd6547-66n2g\" (UID: \"4709930f-fc73-4b82-951f-a428ad89313a\") " pod="openshift-monitoring/telemeter-client-64fdfd6547-66n2g" Oct 30 16:23:22.962667 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:22.962590 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4709930f-fc73-4b82-951f-a428ad89313a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-64fdfd6547-66n2g\" (UID: \"4709930f-fc73-4b82-951f-a428ad89313a\") " pod="openshift-monitoring/telemeter-client-64fdfd6547-66n2g" Oct 30 16:23:22.962667 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:22.962621 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4709930f-fc73-4b82-951f-a428ad89313a-serving-certs-ca-bundle\") pod \"telemeter-client-64fdfd6547-66n2g\" (UID: \"4709930f-fc73-4b82-951f-a428ad89313a\") " pod="openshift-monitoring/telemeter-client-64fdfd6547-66n2g" Oct 30 16:23:22.962764 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:22.962720 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/4709930f-fc73-4b82-951f-a428ad89313a-secret-telemeter-client\") pod \"telemeter-client-64fdfd6547-66n2g\" (UID: \"4709930f-fc73-4b82-951f-a428ad89313a\") " pod="openshift-monitoring/telemeter-client-64fdfd6547-66n2g" Oct 30 16:23:23.063699 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:23.063631 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4709930f-fc73-4b82-951f-a428ad89313a-metrics-client-ca\") pod \"telemeter-client-64fdfd6547-66n2g\" (UID: \"4709930f-fc73-4b82-951f-a428ad89313a\") " pod="openshift-monitoring/telemeter-client-64fdfd6547-66n2g" Oct 30 16:23:23.063699 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:23.063661 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/4709930f-fc73-4b82-951f-a428ad89313a-telemeter-client-tls\") pod \"telemeter-client-64fdfd6547-66n2g\" (UID: \"4709930f-fc73-4b82-951f-a428ad89313a\") " pod="openshift-monitoring/telemeter-client-64fdfd6547-66n2g" Oct 30 16:23:23.063699 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:23.063677 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zvqm\" (UniqueName: \"kubernetes.io/projected/4709930f-fc73-4b82-951f-a428ad89313a-kube-api-access-5zvqm\") pod \"telemeter-client-64fdfd6547-66n2g\" (UID: \"4709930f-fc73-4b82-951f-a428ad89313a\") " pod="openshift-monitoring/telemeter-client-64fdfd6547-66n2g" Oct 30 16:23:23.063994 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:23.063700 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4709930f-fc73-4b82-951f-a428ad89313a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-64fdfd6547-66n2g\" (UID: \"4709930f-fc73-4b82-951f-a428ad89313a\") " pod="openshift-monitoring/telemeter-client-64fdfd6547-66n2g" Oct 30 16:23:23.063994 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:23.063736 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/4709930f-fc73-4b82-951f-a428ad89313a-federate-client-tls\") pod \"telemeter-client-64fdfd6547-66n2g\" (UID: \"4709930f-fc73-4b82-951f-a428ad89313a\") " pod="openshift-monitoring/telemeter-client-64fdfd6547-66n2g" Oct 30 16:23:23.063994 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:23.063758 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4709930f-fc73-4b82-951f-a428ad89313a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-64fdfd6547-66n2g\" (UID: \"4709930f-fc73-4b82-951f-a428ad89313a\") " pod="openshift-monitoring/telemeter-client-64fdfd6547-66n2g" Oct 30 16:23:23.063994 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:23.063807 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4709930f-fc73-4b82-951f-a428ad89313a-serving-certs-ca-bundle\") pod \"telemeter-client-64fdfd6547-66n2g\" (UID: \"4709930f-fc73-4b82-951f-a428ad89313a\") " pod="openshift-monitoring/telemeter-client-64fdfd6547-66n2g" Oct 30 16:23:23.063994 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:23.063854 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/4709930f-fc73-4b82-951f-a428ad89313a-secret-telemeter-client\") pod \"telemeter-client-64fdfd6547-66n2g\" (UID: \"4709930f-fc73-4b82-951f-a428ad89313a\") " pod="openshift-monitoring/telemeter-client-64fdfd6547-66n2g" Oct 30 16:23:23.064634 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:23.064453 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4709930f-fc73-4b82-951f-a428ad89313a-metrics-client-ca\") pod \"telemeter-client-64fdfd6547-66n2g\" (UID: \"4709930f-fc73-4b82-951f-a428ad89313a\") " pod="openshift-monitoring/telemeter-client-64fdfd6547-66n2g" Oct 30 16:23:23.064865 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:23.064558 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4709930f-fc73-4b82-951f-a428ad89313a-serving-certs-ca-bundle\") pod \"telemeter-client-64fdfd6547-66n2g\" (UID: \"4709930f-fc73-4b82-951f-a428ad89313a\") " pod="openshift-monitoring/telemeter-client-64fdfd6547-66n2g" Oct 30 16:23:23.065143 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:23.065120 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4709930f-fc73-4b82-951f-a428ad89313a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-64fdfd6547-66n2g\" (UID: \"4709930f-fc73-4b82-951f-a428ad89313a\") " pod="openshift-monitoring/telemeter-client-64fdfd6547-66n2g" Oct 30 16:23:23.066403 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:23.066385 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/4709930f-fc73-4b82-951f-a428ad89313a-federate-client-tls\") pod \"telemeter-client-64fdfd6547-66n2g\" (UID: \"4709930f-fc73-4b82-951f-a428ad89313a\") " pod="openshift-monitoring/telemeter-client-64fdfd6547-66n2g" Oct 30 16:23:23.066526 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:23.066441 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4709930f-fc73-4b82-951f-a428ad89313a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-64fdfd6547-66n2g\" (UID: \"4709930f-fc73-4b82-951f-a428ad89313a\") " pod="openshift-monitoring/telemeter-client-64fdfd6547-66n2g" Oct 30 16:23:23.066986 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:23.066964 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/4709930f-fc73-4b82-951f-a428ad89313a-telemeter-client-tls\") pod \"telemeter-client-64fdfd6547-66n2g\" (UID: \"4709930f-fc73-4b82-951f-a428ad89313a\") " pod="openshift-monitoring/telemeter-client-64fdfd6547-66n2g" Oct 30 16:23:23.067270 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:23.067251 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/4709930f-fc73-4b82-951f-a428ad89313a-secret-telemeter-client\") pod \"telemeter-client-64fdfd6547-66n2g\" (UID: \"4709930f-fc73-4b82-951f-a428ad89313a\") " pod="openshift-monitoring/telemeter-client-64fdfd6547-66n2g" Oct 30 16:23:23.086319 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:23.086296 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zvqm\" (UniqueName: \"kubernetes.io/projected/4709930f-fc73-4b82-951f-a428ad89313a-kube-api-access-5zvqm\") pod \"telemeter-client-64fdfd6547-66n2g\" (UID: \"4709930f-fc73-4b82-951f-a428ad89313a\") " pod="openshift-monitoring/telemeter-client-64fdfd6547-66n2g" Oct 30 16:23:23.119151 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:23.119129 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-64fdfd6547-66n2g" Oct 30 16:23:23.475247 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:23.475217 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-64fdfd6547-66n2g"] Oct 30 16:23:23.478297 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:23:23.478266 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4709930f_fc73_4b82_951f_a428ad89313a.slice/crio-3bf385594d50db8471a52441663cd6d7f194aa3e58fe2c91436fb702829bcafe WatchSource:0}: Error finding container 3bf385594d50db8471a52441663cd6d7f194aa3e58fe2c91436fb702829bcafe: Status 404 returned error can't find the container with id 3bf385594d50db8471a52441663cd6d7f194aa3e58fe2c91436fb702829bcafe Oct 30 16:23:24.094876 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:24.094838 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-64fdfd6547-66n2g" event={"ID":"4709930f-fc73-4b82-951f-a428ad89313a","Type":"ContainerStarted","Data":"3bf385594d50db8471a52441663cd6d7f194aa3e58fe2c91436fb702829bcafe"} Oct 30 16:23:26.106850 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:26.106817 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-64fdfd6547-66n2g" event={"ID":"4709930f-fc73-4b82-951f-a428ad89313a","Type":"ContainerStarted","Data":"b4152b9ec7f8619493796da5f1017e073b850ed785a5807691a3283de2ce0b8e"} Oct 30 16:23:28.030298 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:28.030273 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-8b5bbf7-s2s69" Oct 30 16:23:28.115312 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:28.115276 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-64fdfd6547-66n2g" event={"ID":"4709930f-fc73-4b82-951f-a428ad89313a","Type":"ContainerStarted","Data":"ac4e6b4c6a1dc5f22986b83c4c83aa43e084d6b0fe2c671231b417b2b489f8ce"} Oct 30 16:23:28.115603 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:28.115370 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-64fdfd6547-66n2g" event={"ID":"4709930f-fc73-4b82-951f-a428ad89313a","Type":"ContainerStarted","Data":"52cf34997452721c8616a21fa61bd2c9e01155be64ffd7cd1373bb82aa455b5f"} Oct 30 16:23:28.163628 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:28.163582 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-64fdfd6547-66n2g" podStartSLOduration=2.393124611 podStartE2EDuration="6.163568668s" totalImagesPullingTime="3.770444057s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:23:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:23:28.162645802 +0000 UTC m=+35.389820902" watchObservedRunningTime="2026-10-30 16:23:28.163568668 +0000 UTC m=+35.390743767" Oct 30 16:23:30.339192 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:23:30.339157 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 16:23:30.495753 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:23:30.495722 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-426gj" Oct 30 16:24:00.338827 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:24:00.338769 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 16:24:21.001312 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:24:21.001283 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 16:24:21.005190 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:24:21.005174 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:24:21.021967 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:24:21.021938 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 16:24:21.022092 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:24:21.022047 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 16:24:21.022092 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:24:21.022048 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 16:24:21.022092 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:24:21.022048 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 16:24:21.022092 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:24:21.022051 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 16:24:21.033645 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:24:21.033626 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 16:24:21.033774 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:24:21.033656 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 16:24:21.033774 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:24:21.033730 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-fhclw\"" Oct 30 16:24:21.046277 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:24:21.046251 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 16:24:21.046370 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:24:21.046283 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 16:24:21.047278 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:24:21.047261 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 16:24:21.063447 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:24:21.063425 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4779dd76-1dee-475b-8276-5823961467dc-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4779dd76-1dee-475b-8276-5823961467dc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:24:21.063502 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:24:21.063453 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4779dd76-1dee-475b-8276-5823961467dc-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4779dd76-1dee-475b-8276-5823961467dc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:24:21.063502 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:24:21.063479 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4779dd76-1dee-475b-8276-5823961467dc-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4779dd76-1dee-475b-8276-5823961467dc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:24:21.063697 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:24:21.063680 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q5s8\" (UniqueName: \"kubernetes.io/projected/4779dd76-1dee-475b-8276-5823961467dc-kube-api-access-5q5s8\") pod \"alertmanager-main-0\" (UID: \"4779dd76-1dee-475b-8276-5823961467dc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:24:21.063745 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:24:21.063718 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4779dd76-1dee-475b-8276-5823961467dc-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4779dd76-1dee-475b-8276-5823961467dc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:24:21.063816 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:24:21.063746 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4779dd76-1dee-475b-8276-5823961467dc-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4779dd76-1dee-475b-8276-5823961467dc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:24:21.063874 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:24:21.063813 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4779dd76-1dee-475b-8276-5823961467dc-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4779dd76-1dee-475b-8276-5823961467dc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:24:21.063925 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:24:21.063885 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4779dd76-1dee-475b-8276-5823961467dc-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4779dd76-1dee-475b-8276-5823961467dc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:24:21.063974 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:24:21.063961 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4779dd76-1dee-475b-8276-5823961467dc-web-config\") pod \"alertmanager-main-0\" (UID: \"4779dd76-1dee-475b-8276-5823961467dc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:24:21.064011 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:24:21.063987 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4779dd76-1dee-475b-8276-5823961467dc-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4779dd76-1dee-475b-8276-5823961467dc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:24:21.064046 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:24:21.064009 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4779dd76-1dee-475b-8276-5823961467dc-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4779dd76-1dee-475b-8276-5823961467dc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:24:21.064078 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:24:21.064050 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4779dd76-1dee-475b-8276-5823961467dc-config-out\") pod \"alertmanager-main-0\" (UID: \"4779dd76-1dee-475b-8276-5823961467dc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:24:21.064078 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:24:21.064069 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4779dd76-1dee-475b-8276-5823961467dc-config-volume\") pod \"alertmanager-main-0\" (UID: \"4779dd76-1dee-475b-8276-5823961467dc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:24:21.165042 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:24:21.165018 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4779dd76-1dee-475b-8276-5823961467dc-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4779dd76-1dee-475b-8276-5823961467dc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:24:21.165191 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:24:21.165051 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q5s8\" (UniqueName: \"kubernetes.io/projected/4779dd76-1dee-475b-8276-5823961467dc-kube-api-access-5q5s8\") pod \"alertmanager-main-0\" (UID: \"4779dd76-1dee-475b-8276-5823961467dc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:24:21.165191 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:24:21.165072 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4779dd76-1dee-475b-8276-5823961467dc-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4779dd76-1dee-475b-8276-5823961467dc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:24:21.165299 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:24:21.165186 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4779dd76-1dee-475b-8276-5823961467dc-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4779dd76-1dee-475b-8276-5823961467dc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:24:21.165299 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:24:21.165284 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4779dd76-1dee-475b-8276-5823961467dc-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4779dd76-1dee-475b-8276-5823961467dc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:24:21.165401 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:24:21.165333 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4779dd76-1dee-475b-8276-5823961467dc-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4779dd76-1dee-475b-8276-5823961467dc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:24:21.165401 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:24:21.165374 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4779dd76-1dee-475b-8276-5823961467dc-web-config\") pod \"alertmanager-main-0\" (UID: \"4779dd76-1dee-475b-8276-5823961467dc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:24:21.165506 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:24:21.165405 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4779dd76-1dee-475b-8276-5823961467dc-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4779dd76-1dee-475b-8276-5823961467dc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:24:21.165506 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:24:21.165437 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4779dd76-1dee-475b-8276-5823961467dc-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4779dd76-1dee-475b-8276-5823961467dc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:24:21.165506 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:24:21.165474 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4779dd76-1dee-475b-8276-5823961467dc-config-out\") pod \"alertmanager-main-0\" (UID: \"4779dd76-1dee-475b-8276-5823961467dc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:24:21.165647 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:24:21.165503 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4779dd76-1dee-475b-8276-5823961467dc-config-volume\") pod \"alertmanager-main-0\" (UID: \"4779dd76-1dee-475b-8276-5823961467dc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:24:21.165647 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:24:21.165537 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4779dd76-1dee-475b-8276-5823961467dc-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4779dd76-1dee-475b-8276-5823961467dc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:24:21.165647 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:24:21.165560 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4779dd76-1dee-475b-8276-5823961467dc-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4779dd76-1dee-475b-8276-5823961467dc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:24:21.166460 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:24:21.166100 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4779dd76-1dee-475b-8276-5823961467dc-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4779dd76-1dee-475b-8276-5823961467dc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:24:21.166575 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:24:21.166499 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4779dd76-1dee-475b-8276-5823961467dc-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4779dd76-1dee-475b-8276-5823961467dc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:24:21.167500 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:24:21.167472 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4779dd76-1dee-475b-8276-5823961467dc-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4779dd76-1dee-475b-8276-5823961467dc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:24:21.168326 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:24:21.168301 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4779dd76-1dee-475b-8276-5823961467dc-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4779dd76-1dee-475b-8276-5823961467dc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:24:21.168772 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:24:21.168749 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4779dd76-1dee-475b-8276-5823961467dc-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4779dd76-1dee-475b-8276-5823961467dc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:24:21.169173 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:24:21.169148 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4779dd76-1dee-475b-8276-5823961467dc-config-volume\") pod \"alertmanager-main-0\" (UID: \"4779dd76-1dee-475b-8276-5823961467dc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:24:21.169259 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:24:21.169235 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4779dd76-1dee-475b-8276-5823961467dc-web-config\") pod \"alertmanager-main-0\" (UID: \"4779dd76-1dee-475b-8276-5823961467dc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:24:21.169379 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:24:21.169353 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4779dd76-1dee-475b-8276-5823961467dc-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4779dd76-1dee-475b-8276-5823961467dc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:24:21.169447 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:24:21.169408 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4779dd76-1dee-475b-8276-5823961467dc-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4779dd76-1dee-475b-8276-5823961467dc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:24:21.169502 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:24:21.169486 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4779dd76-1dee-475b-8276-5823961467dc-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4779dd76-1dee-475b-8276-5823961467dc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:24:21.169887 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:24:21.169872 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4779dd76-1dee-475b-8276-5823961467dc-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4779dd76-1dee-475b-8276-5823961467dc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:24:21.169932 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:24:21.169877 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4779dd76-1dee-475b-8276-5823961467dc-config-out\") pod \"alertmanager-main-0\" (UID: \"4779dd76-1dee-475b-8276-5823961467dc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:24:21.188160 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:24:21.188141 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q5s8\" (UniqueName: \"kubernetes.io/projected/4779dd76-1dee-475b-8276-5823961467dc-kube-api-access-5q5s8\") pod \"alertmanager-main-0\" (UID: \"4779dd76-1dee-475b-8276-5823961467dc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:24:21.328472 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:24:21.328427 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:24:21.657877 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:24:21.657842 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 16:24:21.660926 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:24:21.660900 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4779dd76_1dee_475b_8276_5823961467dc.slice/crio-9f3fc677d186eebdfb55eb0d20a646e9ad133be1b5f57832897addb1a818d45b WatchSource:0}: Error finding container 9f3fc677d186eebdfb55eb0d20a646e9ad133be1b5f57832897addb1a818d45b: Status 404 returned error can't find the container with id 9f3fc677d186eebdfb55eb0d20a646e9ad133be1b5f57832897addb1a818d45b Oct 30 16:24:22.290258 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:24:22.290227 4493 generic.go:356] "Generic (PLEG): container finished" podID="4779dd76-1dee-475b-8276-5823961467dc" containerID="254c7115f138a028ebc66a936fcaff1c16f352fbf6e9cebbfe5cdf505129ab66" exitCode=0 Oct 30 16:24:22.290646 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:24:22.290310 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4779dd76-1dee-475b-8276-5823961467dc","Type":"ContainerDied","Data":"254c7115f138a028ebc66a936fcaff1c16f352fbf6e9cebbfe5cdf505129ab66"} Oct 30 16:24:22.290646 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:24:22.290376 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4779dd76-1dee-475b-8276-5823961467dc","Type":"ContainerStarted","Data":"9f3fc677d186eebdfb55eb0d20a646e9ad133be1b5f57832897addb1a818d45b"} Oct 30 16:24:24.301772 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:24:24.301683 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4779dd76-1dee-475b-8276-5823961467dc","Type":"ContainerStarted","Data":"ce622aa76d958da6b5fe23cd3fe36498dceed0c56c2a9664a9613083b61f5d58"} Oct 30 16:24:24.302148 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:24:24.301798 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4779dd76-1dee-475b-8276-5823961467dc","Type":"ContainerStarted","Data":"ab55ef49e3a3f74dcf8a432080910416475da13f72ee5f5a6dc493fa76ba921b"} Oct 30 16:24:24.302148 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:24:24.301828 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4779dd76-1dee-475b-8276-5823961467dc","Type":"ContainerStarted","Data":"ae87ab5d7d8e1bf0353cf06ad86e1b56ae2ad58d313ef38d018aedfeec830e01"} Oct 30 16:24:24.302148 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:24:24.301852 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4779dd76-1dee-475b-8276-5823961467dc","Type":"ContainerStarted","Data":"81309b53ecd037bdbbeeafb790d0c4e00c10d49eecf37abe46a0d8259de601a8"} Oct 30 16:24:24.302148 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:24:24.301873 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4779dd76-1dee-475b-8276-5823961467dc","Type":"ContainerStarted","Data":"40d28b1a59b7d5ffcd8235fa99780c58040b10d128d8934e9c3ee3eec7aa34e4"} Oct 30 16:24:25.307979 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:24:25.307947 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4779dd76-1dee-475b-8276-5823961467dc","Type":"ContainerStarted","Data":"ad6d09b10222cbe5d1a03d3213d4a34b167f1af12b4d628f55eff6e038e5cdcd"} Oct 30 16:24:25.395875 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:24:25.395828 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.216066012 podStartE2EDuration="5.395816442s" totalImagesPullingTime="2.166268748s" totalInitContainerRuntime="13.481682ms" isStatefulPod=false podCreationTimestamp="2026-10-30 16:24:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:24:25.383984516 +0000 UTC m=+92.611159615" watchObservedRunningTime="2026-10-30 16:24:25.395816442 +0000 UTC m=+92.622991539" Oct 30 16:24:30.339928 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:24:30.339893 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 16:25:00.339597 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:25:00.339555 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 16:25:30.340097 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:25:30.340059 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 16:26:00.338874 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:26:00.338838 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 16:26:30.342430 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:26:30.340581 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 16:26:37.780440 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:37.780348 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7157s48"] Oct 30 16:26:37.783761 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:37.783745 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7157s48" Oct 30 16:26:37.809565 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:37.809534 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 16:26:37.809565 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:37.809559 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-p4mrv\"" Oct 30 16:26:37.809763 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:37.809577 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 16:26:37.810620 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:37.810602 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7157s48"] Oct 30 16:26:37.880480 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:37.880454 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9c4ac6e2-9eb6-4a3e-a536-300d213dae4a-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7157s48\" (UID: \"9c4ac6e2-9eb6-4a3e-a536-300d213dae4a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7157s48" Oct 30 16:26:37.880599 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:37.880504 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9c4ac6e2-9eb6-4a3e-a536-300d213dae4a-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7157s48\" (UID: \"9c4ac6e2-9eb6-4a3e-a536-300d213dae4a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7157s48" Oct 30 16:26:37.880599 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:37.880557 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5vfl\" (UniqueName: \"kubernetes.io/projected/9c4ac6e2-9eb6-4a3e-a536-300d213dae4a-kube-api-access-h5vfl\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7157s48\" (UID: \"9c4ac6e2-9eb6-4a3e-a536-300d213dae4a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7157s48" Oct 30 16:26:37.981856 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:37.981821 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9c4ac6e2-9eb6-4a3e-a536-300d213dae4a-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7157s48\" (UID: \"9c4ac6e2-9eb6-4a3e-a536-300d213dae4a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7157s48" Oct 30 16:26:37.981938 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:37.981863 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5vfl\" (UniqueName: \"kubernetes.io/projected/9c4ac6e2-9eb6-4a3e-a536-300d213dae4a-kube-api-access-h5vfl\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7157s48\" (UID: \"9c4ac6e2-9eb6-4a3e-a536-300d213dae4a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7157s48" Oct 30 16:26:37.981938 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:37.981907 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9c4ac6e2-9eb6-4a3e-a536-300d213dae4a-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7157s48\" (UID: \"9c4ac6e2-9eb6-4a3e-a536-300d213dae4a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7157s48" Oct 30 16:26:37.982193 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:37.982176 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9c4ac6e2-9eb6-4a3e-a536-300d213dae4a-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7157s48\" (UID: \"9c4ac6e2-9eb6-4a3e-a536-300d213dae4a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7157s48" Oct 30 16:26:37.982236 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:37.982209 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9c4ac6e2-9eb6-4a3e-a536-300d213dae4a-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7157s48\" (UID: \"9c4ac6e2-9eb6-4a3e-a536-300d213dae4a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7157s48" Oct 30 16:26:38.005151 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:38.005125 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5vfl\" (UniqueName: \"kubernetes.io/projected/9c4ac6e2-9eb6-4a3e-a536-300d213dae4a-kube-api-access-h5vfl\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7157s48\" (UID: \"9c4ac6e2-9eb6-4a3e-a536-300d213dae4a\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7157s48" Oct 30 16:26:38.098765 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:38.098697 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7157s48" Oct 30 16:26:38.392821 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:38.392793 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7157s48"] Oct 30 16:26:38.396690 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:26:38.396664 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c4ac6e2_9eb6_4a3e_a536_300d213dae4a.slice/crio-ddc09d26ae807f10c7b5218423e1c3c027e4d2e528a5b8d8351bae755cdb1c10 WatchSource:0}: Error finding container ddc09d26ae807f10c7b5218423e1c3c027e4d2e528a5b8d8351bae755cdb1c10: Status 404 returned error can't find the container with id ddc09d26ae807f10c7b5218423e1c3c027e4d2e528a5b8d8351bae755cdb1c10 Oct 30 16:26:38.759284 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:38.759209 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7157s48" event={"ID":"9c4ac6e2-9eb6-4a3e-a536-300d213dae4a","Type":"ContainerStarted","Data":"ddc09d26ae807f10c7b5218423e1c3c027e4d2e528a5b8d8351bae755cdb1c10"} Oct 30 16:26:39.535665 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:39.535619 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awt54c"] Oct 30 16:26:39.540427 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:39.540400 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awt54c" Oct 30 16:26:39.566506 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:39.566471 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-hrhhc\"" Oct 30 16:26:39.567859 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:39.567838 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awt54c"] Oct 30 16:26:39.594745 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:39.594706 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2eb25641-1607-4558-85ed-a51033782b02-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awt54c\" (UID: \"2eb25641-1607-4558-85ed-a51033782b02\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awt54c" Oct 30 16:26:39.594934 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:39.594833 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2eb25641-1607-4558-85ed-a51033782b02-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awt54c\" (UID: \"2eb25641-1607-4558-85ed-a51033782b02\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awt54c" Oct 30 16:26:39.594934 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:39.594919 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mnnc\" (UniqueName: \"kubernetes.io/projected/2eb25641-1607-4558-85ed-a51033782b02-kube-api-access-4mnnc\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awt54c\" (UID: \"2eb25641-1607-4558-85ed-a51033782b02\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awt54c" Oct 30 16:26:39.633175 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:39.633140 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vjtnw"] Oct 30 16:26:39.636918 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:39.636895 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vjtnw" Oct 30 16:26:39.651048 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:39.651023 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-lhsrj\"" Oct 30 16:26:39.659075 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:39.659049 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vjtnw"] Oct 30 16:26:39.696062 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:39.696024 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2eb25641-1607-4558-85ed-a51033782b02-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awt54c\" (UID: \"2eb25641-1607-4558-85ed-a51033782b02\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awt54c" Oct 30 16:26:39.696260 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:39.696080 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/40060367-43c3-40cb-a2c2-dc587e7bae98-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vjtnw\" (UID: \"40060367-43c3-40cb-a2c2-dc587e7bae98\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vjtnw" Oct 30 16:26:39.696260 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:39.696124 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkxpd\" (UniqueName: \"kubernetes.io/projected/40060367-43c3-40cb-a2c2-dc587e7bae98-kube-api-access-lkxpd\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vjtnw\" (UID: \"40060367-43c3-40cb-a2c2-dc587e7bae98\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vjtnw" Oct 30 16:26:39.696260 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:39.696177 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mnnc\" (UniqueName: \"kubernetes.io/projected/2eb25641-1607-4558-85ed-a51033782b02-kube-api-access-4mnnc\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awt54c\" (UID: \"2eb25641-1607-4558-85ed-a51033782b02\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awt54c" Oct 30 16:26:39.696462 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:39.696327 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2eb25641-1607-4558-85ed-a51033782b02-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awt54c\" (UID: \"2eb25641-1607-4558-85ed-a51033782b02\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awt54c" Oct 30 16:26:39.696462 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:39.696364 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/40060367-43c3-40cb-a2c2-dc587e7bae98-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vjtnw\" (UID: \"40060367-43c3-40cb-a2c2-dc587e7bae98\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vjtnw" Oct 30 16:26:39.696572 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:39.696502 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2eb25641-1607-4558-85ed-a51033782b02-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awt54c\" (UID: \"2eb25641-1607-4558-85ed-a51033782b02\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awt54c" Oct 30 16:26:39.696655 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:39.696637 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2eb25641-1607-4558-85ed-a51033782b02-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awt54c\" (UID: \"2eb25641-1607-4558-85ed-a51033782b02\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awt54c" Oct 30 16:26:39.721118 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:39.721078 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mnnc\" (UniqueName: \"kubernetes.io/projected/2eb25641-1607-4558-85ed-a51033782b02-kube-api-access-4mnnc\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awt54c\" (UID: \"2eb25641-1607-4558-85ed-a51033782b02\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awt54c" Oct 30 16:26:39.736584 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:39.736551 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zcbk"] Oct 30 16:26:39.740304 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:39.740212 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zcbk" Oct 30 16:26:39.754301 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:39.754279 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-s5tj9\"" Oct 30 16:26:39.764885 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:39.764857 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zcbk"] Oct 30 16:26:39.797397 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:39.797309 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ac42d273-7025-4cdc-88b2-dfc39ed12a2a-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zcbk\" (UID: \"ac42d273-7025-4cdc-88b2-dfc39ed12a2a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zcbk" Oct 30 16:26:39.797397 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:39.797358 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkxpd\" (UniqueName: \"kubernetes.io/projected/40060367-43c3-40cb-a2c2-dc587e7bae98-kube-api-access-lkxpd\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vjtnw\" (UID: \"40060367-43c3-40cb-a2c2-dc587e7bae98\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vjtnw" Oct 30 16:26:39.797644 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:39.797453 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ac42d273-7025-4cdc-88b2-dfc39ed12a2a-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zcbk\" (UID: \"ac42d273-7025-4cdc-88b2-dfc39ed12a2a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zcbk" Oct 30 16:26:39.797644 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:39.797494 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88r82\" (UniqueName: \"kubernetes.io/projected/ac42d273-7025-4cdc-88b2-dfc39ed12a2a-kube-api-access-88r82\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zcbk\" (UID: \"ac42d273-7025-4cdc-88b2-dfc39ed12a2a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zcbk" Oct 30 16:26:39.797644 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:39.797530 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/40060367-43c3-40cb-a2c2-dc587e7bae98-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vjtnw\" (UID: \"40060367-43c3-40cb-a2c2-dc587e7bae98\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vjtnw" Oct 30 16:26:39.797644 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:39.797587 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/40060367-43c3-40cb-a2c2-dc587e7bae98-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vjtnw\" (UID: \"40060367-43c3-40cb-a2c2-dc587e7bae98\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vjtnw" Oct 30 16:26:39.798030 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:39.798008 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/40060367-43c3-40cb-a2c2-dc587e7bae98-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vjtnw\" (UID: \"40060367-43c3-40cb-a2c2-dc587e7bae98\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vjtnw" Oct 30 16:26:39.798583 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:39.798564 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/40060367-43c3-40cb-a2c2-dc587e7bae98-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vjtnw\" (UID: \"40060367-43c3-40cb-a2c2-dc587e7bae98\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vjtnw" Oct 30 16:26:39.821639 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:39.821611 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkxpd\" (UniqueName: \"kubernetes.io/projected/40060367-43c3-40cb-a2c2-dc587e7bae98-kube-api-access-lkxpd\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vjtnw\" (UID: \"40060367-43c3-40cb-a2c2-dc587e7bae98\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vjtnw" Oct 30 16:26:39.831970 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:39.831942 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jd2lp"] Oct 30 16:26:39.835814 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:39.835733 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jd2lp" Oct 30 16:26:39.849755 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:39.849732 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-rtg4l\"" Oct 30 16:26:39.856757 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:39.856734 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jd2lp"] Oct 30 16:26:39.858530 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:39.858511 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awt54c" Oct 30 16:26:39.898184 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:39.898145 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a1933525-3f60-4e41-aad1-157f7b74c2c5-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jd2lp\" (UID: \"a1933525-3f60-4e41-aad1-157f7b74c2c5\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jd2lp" Oct 30 16:26:39.898320 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:39.898190 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ac42d273-7025-4cdc-88b2-dfc39ed12a2a-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zcbk\" (UID: \"ac42d273-7025-4cdc-88b2-dfc39ed12a2a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zcbk" Oct 30 16:26:39.898320 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:39.898224 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ltdk\" (UniqueName: \"kubernetes.io/projected/a1933525-3f60-4e41-aad1-157f7b74c2c5-kube-api-access-4ltdk\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jd2lp\" (UID: \"a1933525-3f60-4e41-aad1-157f7b74c2c5\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jd2lp" Oct 30 16:26:39.898320 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:39.898311 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88r82\" (UniqueName: \"kubernetes.io/projected/ac42d273-7025-4cdc-88b2-dfc39ed12a2a-kube-api-access-88r82\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zcbk\" (UID: \"ac42d273-7025-4cdc-88b2-dfc39ed12a2a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zcbk" Oct 30 16:26:39.898448 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:39.898389 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a1933525-3f60-4e41-aad1-157f7b74c2c5-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jd2lp\" (UID: \"a1933525-3f60-4e41-aad1-157f7b74c2c5\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jd2lp" Oct 30 16:26:39.898448 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:39.898418 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ac42d273-7025-4cdc-88b2-dfc39ed12a2a-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zcbk\" (UID: \"ac42d273-7025-4cdc-88b2-dfc39ed12a2a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zcbk" Oct 30 16:26:39.898612 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:39.898591 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ac42d273-7025-4cdc-88b2-dfc39ed12a2a-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zcbk\" (UID: \"ac42d273-7025-4cdc-88b2-dfc39ed12a2a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zcbk" Oct 30 16:26:39.899031 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:39.899004 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ac42d273-7025-4cdc-88b2-dfc39ed12a2a-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zcbk\" (UID: \"ac42d273-7025-4cdc-88b2-dfc39ed12a2a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zcbk" Oct 30 16:26:39.926144 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:39.926118 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88r82\" (UniqueName: \"kubernetes.io/projected/ac42d273-7025-4cdc-88b2-dfc39ed12a2a-kube-api-access-88r82\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zcbk\" (UID: \"ac42d273-7025-4cdc-88b2-dfc39ed12a2a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zcbk" Oct 30 16:26:39.955380 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:39.955345 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vjtnw" Oct 30 16:26:40.000987 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:39.999532 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a1933525-3f60-4e41-aad1-157f7b74c2c5-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jd2lp\" (UID: \"a1933525-3f60-4e41-aad1-157f7b74c2c5\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jd2lp" Oct 30 16:26:40.000987 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:39.999615 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a1933525-3f60-4e41-aad1-157f7b74c2c5-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jd2lp\" (UID: \"a1933525-3f60-4e41-aad1-157f7b74c2c5\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jd2lp" Oct 30 16:26:40.000987 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:39.999652 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ltdk\" (UniqueName: \"kubernetes.io/projected/a1933525-3f60-4e41-aad1-157f7b74c2c5-kube-api-access-4ltdk\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jd2lp\" (UID: \"a1933525-3f60-4e41-aad1-157f7b74c2c5\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jd2lp" Oct 30 16:26:40.000987 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.000011 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a1933525-3f60-4e41-aad1-157f7b74c2c5-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jd2lp\" (UID: \"a1933525-3f60-4e41-aad1-157f7b74c2c5\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jd2lp" Oct 30 16:26:40.000987 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.000035 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a1933525-3f60-4e41-aad1-157f7b74c2c5-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jd2lp\" (UID: \"a1933525-3f60-4e41-aad1-157f7b74c2c5\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jd2lp" Oct 30 16:26:40.026859 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.026830 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ltdk\" (UniqueName: \"kubernetes.io/projected/a1933525-3f60-4e41-aad1-157f7b74c2c5-kube-api-access-4ltdk\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jd2lp\" (UID: \"a1933525-3f60-4e41-aad1-157f7b74c2c5\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jd2lp" Oct 30 16:26:40.058219 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.058072 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zcbk" Oct 30 16:26:40.155467 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.155433 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jd2lp" Oct 30 16:26:40.171534 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.171504 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awt54c"] Oct 30 16:26:40.177886 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:26:40.177841 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eb25641_1607_4558_85ed_a51033782b02.slice/crio-dbecaa2e0a217b7db408567a99a762a0e938200c39b285ea7d2f2a0f96631e42 WatchSource:0}: Error finding container dbecaa2e0a217b7db408567a99a762a0e938200c39b285ea7d2f2a0f96631e42: Status 404 returned error can't find the container with id dbecaa2e0a217b7db408567a99a762a0e938200c39b285ea7d2f2a0f96631e42 Oct 30 16:26:40.258869 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.258836 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vjtnw"] Oct 30 16:26:40.262179 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:26:40.262150 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40060367_43c3_40cb_a2c2_dc587e7bae98.slice/crio-25f775b458be01ed7549392e466301ab39af0b84e393e95fc46f45428dc029ac WatchSource:0}: Error finding container 25f775b458be01ed7549392e466301ab39af0b84e393e95fc46f45428dc029ac: Status 404 returned error can't find the container with id 25f775b458be01ed7549392e466301ab39af0b84e393e95fc46f45428dc029ac Oct 30 16:26:40.362399 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.362330 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zcbk"] Oct 30 16:26:40.365688 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:26:40.365660 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac42d273_7025_4cdc_88b2_dfc39ed12a2a.slice/crio-44264949079d01a83d0fda5e472a81612909ab708ab147fe175910fd05f2c512 WatchSource:0}: Error finding container 44264949079d01a83d0fda5e472a81612909ab708ab147fe175910fd05f2c512: Status 404 returned error can't find the container with id 44264949079d01a83d0fda5e472a81612909ab708ab147fe175910fd05f2c512 Oct 30 16:26:40.438818 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.437891 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6n8m6"] Oct 30 16:26:40.442862 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.442744 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6n8m6" Oct 30 16:26:40.457680 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.457656 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-xtm6x\"" Oct 30 16:26:40.460732 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.460637 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 16:26:40.460732 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.460696 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 16:26:40.461206 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.461190 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 16:26:40.474989 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.474774 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6n8m6"] Oct 30 16:26:40.477032 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.476003 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jd2lp"] Oct 30 16:26:40.481926 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:26:40.481897 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1933525_3f60_4e41_aad1_157f7b74c2c5.slice/crio-1635b6846cd8c10e6cbf128a62931d5ce56286be27e366770543c91ce97ea6a0 WatchSource:0}: Error finding container 1635b6846cd8c10e6cbf128a62931d5ce56286be27e366770543c91ce97ea6a0: Status 404 returned error can't find the container with id 1635b6846cd8c10e6cbf128a62931d5ce56286be27e366770543c91ce97ea6a0 Oct 30 16:26:40.503771 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.503745 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5903f4b2-9604-4eab-adf5-e0f7452ffbba-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-6n8m6\" (UID: \"5903f4b2-9604-4eab-adf5-e0f7452ffbba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6n8m6" Oct 30 16:26:40.503917 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.503808 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5903f4b2-9604-4eab-adf5-e0f7452ffbba-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-6n8m6\" (UID: \"5903f4b2-9604-4eab-adf5-e0f7452ffbba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6n8m6" Oct 30 16:26:40.503917 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.503842 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5903f4b2-9604-4eab-adf5-e0f7452ffbba-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-6n8m6\" (UID: \"5903f4b2-9604-4eab-adf5-e0f7452ffbba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6n8m6" Oct 30 16:26:40.504038 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.503926 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5903f4b2-9604-4eab-adf5-e0f7452ffbba-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-6n8m6\" (UID: \"5903f4b2-9604-4eab-adf5-e0f7452ffbba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6n8m6" Oct 30 16:26:40.504038 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.503945 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5903f4b2-9604-4eab-adf5-e0f7452ffbba-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-6n8m6\" (UID: \"5903f4b2-9604-4eab-adf5-e0f7452ffbba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6n8m6" Oct 30 16:26:40.504038 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.504026 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8c7h\" (UniqueName: \"kubernetes.io/projected/5903f4b2-9604-4eab-adf5-e0f7452ffbba-kube-api-access-j8c7h\") pod \"maas-default-gateway-openshift-default-7854949c9-6n8m6\" (UID: \"5903f4b2-9604-4eab-adf5-e0f7452ffbba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6n8m6" Oct 30 16:26:40.504135 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.504079 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5903f4b2-9604-4eab-adf5-e0f7452ffbba-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-6n8m6\" (UID: \"5903f4b2-9604-4eab-adf5-e0f7452ffbba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6n8m6" Oct 30 16:26:40.504173 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.504131 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5903f4b2-9604-4eab-adf5-e0f7452ffbba-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-6n8m6\" (UID: \"5903f4b2-9604-4eab-adf5-e0f7452ffbba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6n8m6" Oct 30 16:26:40.504173 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.504169 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5903f4b2-9604-4eab-adf5-e0f7452ffbba-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-6n8m6\" (UID: \"5903f4b2-9604-4eab-adf5-e0f7452ffbba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6n8m6" Oct 30 16:26:40.576423 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.576387 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-fx429"] Oct 30 16:26:40.580316 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.580292 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-fx429" Oct 30 16:26:40.600227 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.599899 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-ckw59\"" Oct 30 16:26:40.605510 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.605425 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d9779f9-732c-4971-b52b-d26ce92abf2f-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-fx429\" (UID: \"5d9779f9-732c-4971-b52b-d26ce92abf2f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-fx429" Oct 30 16:26:40.605510 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.605463 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d9779f9-732c-4971-b52b-d26ce92abf2f-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-fx429\" (UID: \"5d9779f9-732c-4971-b52b-d26ce92abf2f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-fx429" Oct 30 16:26:40.605510 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.605502 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5903f4b2-9604-4eab-adf5-e0f7452ffbba-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-6n8m6\" (UID: \"5903f4b2-9604-4eab-adf5-e0f7452ffbba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6n8m6" Oct 30 16:26:40.605770 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.605548 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5903f4b2-9604-4eab-adf5-e0f7452ffbba-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-6n8m6\" (UID: \"5903f4b2-9604-4eab-adf5-e0f7452ffbba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6n8m6" Oct 30 16:26:40.605770 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.605593 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5903f4b2-9604-4eab-adf5-e0f7452ffbba-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-6n8m6\" (UID: \"5903f4b2-9604-4eab-adf5-e0f7452ffbba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6n8m6" Oct 30 16:26:40.605770 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.605616 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d9779f9-732c-4971-b52b-d26ce92abf2f-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-fx429\" (UID: \"5d9779f9-732c-4971-b52b-d26ce92abf2f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-fx429" Oct 30 16:26:40.605770 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.605650 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5903f4b2-9604-4eab-adf5-e0f7452ffbba-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-6n8m6\" (UID: \"5903f4b2-9604-4eab-adf5-e0f7452ffbba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6n8m6" Oct 30 16:26:40.605770 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.605757 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d9779f9-732c-4971-b52b-d26ce92abf2f-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-fx429\" (UID: \"5d9779f9-732c-4971-b52b-d26ce92abf2f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-fx429" Oct 30 16:26:40.606607 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.606585 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5903f4b2-9604-4eab-adf5-e0f7452ffbba-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-6n8m6\" (UID: \"5903f4b2-9604-4eab-adf5-e0f7452ffbba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6n8m6" Oct 30 16:26:40.606936 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.606913 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-fx429"] Oct 30 16:26:40.606936 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.606913 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5903f4b2-9604-4eab-adf5-e0f7452ffbba-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-6n8m6\" (UID: \"5903f4b2-9604-4eab-adf5-e0f7452ffbba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6n8m6" Oct 30 16:26:40.607168 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.606993 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5903f4b2-9604-4eab-adf5-e0f7452ffbba-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-6n8m6\" (UID: \"5903f4b2-9604-4eab-adf5-e0f7452ffbba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6n8m6" Oct 30 16:26:40.607168 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.607060 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5903f4b2-9604-4eab-adf5-e0f7452ffbba-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-6n8m6\" (UID: \"5903f4b2-9604-4eab-adf5-e0f7452ffbba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6n8m6" Oct 30 16:26:40.607168 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.607118 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d9779f9-732c-4971-b52b-d26ce92abf2f-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-fx429\" (UID: \"5d9779f9-732c-4971-b52b-d26ce92abf2f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-fx429" Oct 30 16:26:40.607168 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.607159 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7prgz\" (UniqueName: \"kubernetes.io/projected/5d9779f9-732c-4971-b52b-d26ce92abf2f-kube-api-access-7prgz\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-fx429\" (UID: \"5d9779f9-732c-4971-b52b-d26ce92abf2f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-fx429" Oct 30 16:26:40.607427 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.607187 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d9779f9-732c-4971-b52b-d26ce92abf2f-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-fx429\" (UID: \"5d9779f9-732c-4971-b52b-d26ce92abf2f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-fx429" Oct 30 16:26:40.607427 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.607222 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5903f4b2-9604-4eab-adf5-e0f7452ffbba-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-6n8m6\" (UID: \"5903f4b2-9604-4eab-adf5-e0f7452ffbba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6n8m6" Oct 30 16:26:40.607427 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.607244 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5903f4b2-9604-4eab-adf5-e0f7452ffbba-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-6n8m6\" (UID: \"5903f4b2-9604-4eab-adf5-e0f7452ffbba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6n8m6" Oct 30 16:26:40.607427 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.607295 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8c7h\" (UniqueName: \"kubernetes.io/projected/5903f4b2-9604-4eab-adf5-e0f7452ffbba-kube-api-access-j8c7h\") pod \"maas-default-gateway-openshift-default-7854949c9-6n8m6\" (UID: \"5903f4b2-9604-4eab-adf5-e0f7452ffbba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6n8m6" Oct 30 16:26:40.607427 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.607325 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d9779f9-732c-4971-b52b-d26ce92abf2f-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-fx429\" (UID: \"5d9779f9-732c-4971-b52b-d26ce92abf2f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-fx429" Oct 30 16:26:40.607427 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.607343 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d9779f9-732c-4971-b52b-d26ce92abf2f-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-fx429\" (UID: \"5d9779f9-732c-4971-b52b-d26ce92abf2f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-fx429" Oct 30 16:26:40.607726 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.607607 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5903f4b2-9604-4eab-adf5-e0f7452ffbba-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-6n8m6\" (UID: \"5903f4b2-9604-4eab-adf5-e0f7452ffbba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6n8m6" Oct 30 16:26:40.607895 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.607861 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5903f4b2-9604-4eab-adf5-e0f7452ffbba-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-6n8m6\" (UID: \"5903f4b2-9604-4eab-adf5-e0f7452ffbba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6n8m6" Oct 30 16:26:40.608407 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.608382 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5903f4b2-9604-4eab-adf5-e0f7452ffbba-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-6n8m6\" (UID: \"5903f4b2-9604-4eab-adf5-e0f7452ffbba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6n8m6" Oct 30 16:26:40.610647 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.610598 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5903f4b2-9604-4eab-adf5-e0f7452ffbba-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-6n8m6\" (UID: \"5903f4b2-9604-4eab-adf5-e0f7452ffbba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6n8m6" Oct 30 16:26:40.611880 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.611859 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5903f4b2-9604-4eab-adf5-e0f7452ffbba-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-6n8m6\" (UID: \"5903f4b2-9604-4eab-adf5-e0f7452ffbba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6n8m6" Oct 30 16:26:40.630233 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.630208 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5903f4b2-9604-4eab-adf5-e0f7452ffbba-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-6n8m6\" (UID: \"5903f4b2-9604-4eab-adf5-e0f7452ffbba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6n8m6" Oct 30 16:26:40.631947 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.631904 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8c7h\" (UniqueName: \"kubernetes.io/projected/5903f4b2-9604-4eab-adf5-e0f7452ffbba-kube-api-access-j8c7h\") pod \"maas-default-gateway-openshift-default-7854949c9-6n8m6\" (UID: \"5903f4b2-9604-4eab-adf5-e0f7452ffbba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6n8m6" Oct 30 16:26:40.708745 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.708701 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d9779f9-732c-4971-b52b-d26ce92abf2f-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-fx429\" (UID: \"5d9779f9-732c-4971-b52b-d26ce92abf2f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-fx429" Oct 30 16:26:40.708965 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.708769 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d9779f9-732c-4971-b52b-d26ce92abf2f-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-fx429\" (UID: \"5d9779f9-732c-4971-b52b-d26ce92abf2f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-fx429" Oct 30 16:26:40.708965 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.708809 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7prgz\" (UniqueName: \"kubernetes.io/projected/5d9779f9-732c-4971-b52b-d26ce92abf2f-kube-api-access-7prgz\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-fx429\" (UID: \"5d9779f9-732c-4971-b52b-d26ce92abf2f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-fx429" Oct 30 16:26:40.708965 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.708832 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d9779f9-732c-4971-b52b-d26ce92abf2f-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-fx429\" (UID: \"5d9779f9-732c-4971-b52b-d26ce92abf2f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-fx429" Oct 30 16:26:40.708965 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.708865 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d9779f9-732c-4971-b52b-d26ce92abf2f-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-fx429\" (UID: \"5d9779f9-732c-4971-b52b-d26ce92abf2f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-fx429" Oct 30 16:26:40.708965 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.708890 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d9779f9-732c-4971-b52b-d26ce92abf2f-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-fx429\" (UID: \"5d9779f9-732c-4971-b52b-d26ce92abf2f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-fx429" Oct 30 16:26:40.708965 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.708917 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d9779f9-732c-4971-b52b-d26ce92abf2f-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-fx429\" (UID: \"5d9779f9-732c-4971-b52b-d26ce92abf2f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-fx429" Oct 30 16:26:40.708965 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.708931 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d9779f9-732c-4971-b52b-d26ce92abf2f-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-fx429\" (UID: \"5d9779f9-732c-4971-b52b-d26ce92abf2f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-fx429" Oct 30 16:26:40.708965 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.708962 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d9779f9-732c-4971-b52b-d26ce92abf2f-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-fx429\" (UID: \"5d9779f9-732c-4971-b52b-d26ce92abf2f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-fx429" Oct 30 16:26:40.710454 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.709619 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d9779f9-732c-4971-b52b-d26ce92abf2f-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-fx429\" (UID: \"5d9779f9-732c-4971-b52b-d26ce92abf2f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-fx429" Oct 30 16:26:40.710454 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.710043 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d9779f9-732c-4971-b52b-d26ce92abf2f-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-fx429\" (UID: \"5d9779f9-732c-4971-b52b-d26ce92abf2f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-fx429" Oct 30 16:26:40.710454 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.710221 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d9779f9-732c-4971-b52b-d26ce92abf2f-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-fx429\" (UID: \"5d9779f9-732c-4971-b52b-d26ce92abf2f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-fx429" Oct 30 16:26:40.710454 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.710451 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d9779f9-732c-4971-b52b-d26ce92abf2f-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-fx429\" (UID: \"5d9779f9-732c-4971-b52b-d26ce92abf2f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-fx429" Oct 30 16:26:40.710970 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.710933 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d9779f9-732c-4971-b52b-d26ce92abf2f-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-fx429\" (UID: \"5d9779f9-732c-4971-b52b-d26ce92abf2f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-fx429" Oct 30 16:26:40.712939 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.712913 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d9779f9-732c-4971-b52b-d26ce92abf2f-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-fx429\" (UID: \"5d9779f9-732c-4971-b52b-d26ce92abf2f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-fx429" Oct 30 16:26:40.713127 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.713106 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d9779f9-732c-4971-b52b-d26ce92abf2f-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-fx429\" (UID: \"5d9779f9-732c-4971-b52b-d26ce92abf2f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-fx429" Oct 30 16:26:40.735355 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.735327 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d9779f9-732c-4971-b52b-d26ce92abf2f-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-fx429\" (UID: \"5d9779f9-732c-4971-b52b-d26ce92abf2f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-fx429" Oct 30 16:26:40.737614 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.737592 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7prgz\" (UniqueName: \"kubernetes.io/projected/5d9779f9-732c-4971-b52b-d26ce92abf2f-kube-api-access-7prgz\") pod \"openshift-ai-inference-openshift-ai-inference-6f8685f4cf-fx429\" (UID: \"5d9779f9-732c-4971-b52b-d26ce92abf2f\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-fx429" Oct 30 16:26:40.758450 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.758342 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6n8m6" Oct 30 16:26:40.774863 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.774551 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jd2lp" event={"ID":"a1933525-3f60-4e41-aad1-157f7b74c2c5","Type":"ContainerStarted","Data":"1635b6846cd8c10e6cbf128a62931d5ce56286be27e366770543c91ce97ea6a0"} Oct 30 16:26:40.778969 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.778488 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vjtnw" event={"ID":"40060367-43c3-40cb-a2c2-dc587e7bae98","Type":"ContainerStarted","Data":"25f775b458be01ed7549392e466301ab39af0b84e393e95fc46f45428dc029ac"} Oct 30 16:26:40.781101 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.781074 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zcbk" event={"ID":"ac42d273-7025-4cdc-88b2-dfc39ed12a2a","Type":"ContainerStarted","Data":"44264949079d01a83d0fda5e472a81612909ab708ab147fe175910fd05f2c512"} Oct 30 16:26:40.785728 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.785702 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awt54c" event={"ID":"2eb25641-1607-4558-85ed-a51033782b02","Type":"ContainerStarted","Data":"dbecaa2e0a217b7db408567a99a762a0e938200c39b285ea7d2f2a0f96631e42"} Oct 30 16:26:40.835829 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.835609 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czd6zb"] Oct 30 16:26:40.839947 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.839425 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czd6zb" Oct 30 16:26:40.855410 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.855358 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-cjsww\"" Oct 30 16:26:40.862038 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.861970 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czd6zb"] Oct 30 16:26:40.895818 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.895627 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-fx429" Oct 30 16:26:40.910713 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.910681 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qf22\" (UniqueName: \"kubernetes.io/projected/11087140-99ef-44c4-b7fb-3320684b5aa7-kube-api-access-4qf22\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czd6zb\" (UID: \"11087140-99ef-44c4-b7fb-3320684b5aa7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czd6zb" Oct 30 16:26:40.910887 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.910863 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/11087140-99ef-44c4-b7fb-3320684b5aa7-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czd6zb\" (UID: \"11087140-99ef-44c4-b7fb-3320684b5aa7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czd6zb" Oct 30 16:26:40.910963 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.910925 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/11087140-99ef-44c4-b7fb-3320684b5aa7-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czd6zb\" (UID: \"11087140-99ef-44c4-b7fb-3320684b5aa7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czd6zb" Oct 30 16:26:40.999399 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:40.997649 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qsslj"] Oct 30 16:26:41.005535 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:41.005493 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qsslj" Oct 30 16:26:41.013707 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:41.012230 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/11087140-99ef-44c4-b7fb-3320684b5aa7-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czd6zb\" (UID: \"11087140-99ef-44c4-b7fb-3320684b5aa7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czd6zb" Oct 30 16:26:41.013707 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:41.012305 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/11087140-99ef-44c4-b7fb-3320684b5aa7-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czd6zb\" (UID: \"11087140-99ef-44c4-b7fb-3320684b5aa7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czd6zb" Oct 30 16:26:41.013707 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:41.012367 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qf22\" (UniqueName: \"kubernetes.io/projected/11087140-99ef-44c4-b7fb-3320684b5aa7-kube-api-access-4qf22\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czd6zb\" (UID: \"11087140-99ef-44c4-b7fb-3320684b5aa7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czd6zb" Oct 30 16:26:41.013707 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:41.013050 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/11087140-99ef-44c4-b7fb-3320684b5aa7-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czd6zb\" (UID: \"11087140-99ef-44c4-b7fb-3320684b5aa7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czd6zb" Oct 30 16:26:41.013707 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:41.013435 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/11087140-99ef-44c4-b7fb-3320684b5aa7-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czd6zb\" (UID: \"11087140-99ef-44c4-b7fb-3320684b5aa7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czd6zb" Oct 30 16:26:41.024651 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:41.024523 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qsslj"] Oct 30 16:26:41.040891 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:41.040859 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qf22\" (UniqueName: \"kubernetes.io/projected/11087140-99ef-44c4-b7fb-3320684b5aa7-kube-api-access-4qf22\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czd6zb\" (UID: \"11087140-99ef-44c4-b7fb-3320684b5aa7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czd6zb" Oct 30 16:26:41.071102 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:41.071068 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-bhhng\"" Oct 30 16:26:41.075155 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:41.075065 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6n8m6"] Oct 30 16:26:41.113384 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:41.113096 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slbkf\" (UniqueName: \"kubernetes.io/projected/49dbf366-6e7b-4dcd-b357-b2460c81db05-kube-api-access-slbkf\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qsslj\" (UID: \"49dbf366-6e7b-4dcd-b357-b2460c81db05\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qsslj" Oct 30 16:26:41.113384 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:41.113182 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/49dbf366-6e7b-4dcd-b357-b2460c81db05-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qsslj\" (UID: \"49dbf366-6e7b-4dcd-b357-b2460c81db05\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qsslj" Oct 30 16:26:41.113384 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:41.113220 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/49dbf366-6e7b-4dcd-b357-b2460c81db05-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qsslj\" (UID: \"49dbf366-6e7b-4dcd-b357-b2460c81db05\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qsslj" Oct 30 16:26:41.162160 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:41.162017 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czd6zb" Oct 30 16:26:41.213843 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:41.213800 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slbkf\" (UniqueName: \"kubernetes.io/projected/49dbf366-6e7b-4dcd-b357-b2460c81db05-kube-api-access-slbkf\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qsslj\" (UID: \"49dbf366-6e7b-4dcd-b357-b2460c81db05\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qsslj" Oct 30 16:26:41.213973 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:41.213890 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/49dbf366-6e7b-4dcd-b357-b2460c81db05-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qsslj\" (UID: \"49dbf366-6e7b-4dcd-b357-b2460c81db05\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qsslj" Oct 30 16:26:41.213973 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:41.213928 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/49dbf366-6e7b-4dcd-b357-b2460c81db05-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qsslj\" (UID: \"49dbf366-6e7b-4dcd-b357-b2460c81db05\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qsslj" Oct 30 16:26:41.214338 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:41.214318 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/49dbf366-6e7b-4dcd-b357-b2460c81db05-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qsslj\" (UID: \"49dbf366-6e7b-4dcd-b357-b2460c81db05\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qsslj" Oct 30 16:26:41.214691 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:41.214673 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/49dbf366-6e7b-4dcd-b357-b2460c81db05-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qsslj\" (UID: \"49dbf366-6e7b-4dcd-b357-b2460c81db05\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qsslj" Oct 30 16:26:41.227673 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:41.227633 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-fx429"] Oct 30 16:26:41.239133 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:41.238951 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slbkf\" (UniqueName: \"kubernetes.io/projected/49dbf366-6e7b-4dcd-b357-b2460c81db05-kube-api-access-slbkf\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qsslj\" (UID: \"49dbf366-6e7b-4dcd-b357-b2460c81db05\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qsslj" Oct 30 16:26:41.337234 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:41.336982 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qsslj" Oct 30 16:26:41.487535 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:41.487506 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czd6zb"] Oct 30 16:26:41.490423 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:26:41.490386 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11087140_99ef_44c4_b7fb_3320684b5aa7.slice/crio-3865f23b0cf35f5e13da71d34459bbc50593ec1d14326ecc67d65183ff81d3a2 WatchSource:0}: Error finding container 3865f23b0cf35f5e13da71d34459bbc50593ec1d14326ecc67d65183ff81d3a2: Status 404 returned error can't find the container with id 3865f23b0cf35f5e13da71d34459bbc50593ec1d14326ecc67d65183ff81d3a2 Oct 30 16:26:41.705839 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:41.705707 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qsslj"] Oct 30 16:26:41.801747 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:41.801713 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6n8m6" event={"ID":"5903f4b2-9604-4eab-adf5-e0f7452ffbba","Type":"ContainerStarted","Data":"da0f9ee7da7bd7ff4b7662b6203787976a4eb8fe04e772c9426081cbf57161b7"} Oct 30 16:26:41.803422 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:41.803392 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qsslj" event={"ID":"49dbf366-6e7b-4dcd-b357-b2460c81db05","Type":"ContainerStarted","Data":"c51bb4a5141295baf4fbb8b11b14ce876d6793cafdabd4e997f90626a6a6751c"} Oct 30 16:26:41.816881 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:41.816844 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czd6zb" event={"ID":"11087140-99ef-44c4-b7fb-3320684b5aa7","Type":"ContainerStarted","Data":"3865f23b0cf35f5e13da71d34459bbc50593ec1d14326ecc67d65183ff81d3a2"} Oct 30 16:26:41.818598 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:41.818543 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-fx429" event={"ID":"5d9779f9-732c-4971-b52b-d26ce92abf2f","Type":"ContainerStarted","Data":"e655f06aa8568a69b93c47a80bcb9349971db14a391a11c968dd32b549a4a339"} Oct 30 16:26:49.075350 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:49.074537 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:26:49.075350 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:49.074625 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:26:49.075350 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:49.074671 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:26:49.394564 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:49.393811 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:26:49.394564 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:49.393920 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:26:49.394564 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:49.393963 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:26:49.907380 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:49.906827 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6n8m6" event={"ID":"5903f4b2-9604-4eab-adf5-e0f7452ffbba","Type":"ContainerStarted","Data":"0106d84270cd88a079caf167aba8d8c7db014d9244c9dc23c718710171feaf7f"} Oct 30 16:26:49.911146 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:49.911114 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-fx429" event={"ID":"5d9779f9-732c-4971-b52b-d26ce92abf2f","Type":"ContainerStarted","Data":"a5ad123a2c0dcf61e768356f19e325c287b0a9860a2b387dedb5d3891748a15a"} Oct 30 16:26:49.980826 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:49.980028 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6n8m6" podStartSLOduration=1.9914251200000002 podStartE2EDuration="9.980008445s" totalImagesPullingTime="7.988583325s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:26:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:26:49.967846345 +0000 UTC m=+237.195021447" watchObservedRunningTime="2026-10-30 16:26:49.980008445 +0000 UTC m=+237.207183524" Oct 30 16:26:50.013399 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:50.012917 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-fx429" podStartSLOduration=1.8616404659999999 podStartE2EDuration="10.012895571s" totalImagesPullingTime="8.151255105s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:26:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:26:50.011983116 +0000 UTC m=+237.239158241" watchObservedRunningTime="2026-10-30 16:26:50.012895571 +0000 UTC m=+237.240070674" Oct 30 16:26:50.759218 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:50.759128 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6n8m6" Oct 30 16:26:50.765073 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:50.765043 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6n8m6" Oct 30 16:26:50.896454 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:50.896372 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-fx429" Oct 30 16:26:50.902385 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:50.902174 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-fx429" Oct 30 16:26:50.923089 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:50.922976 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-fx429" Oct 30 16:26:50.923089 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:50.923027 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6n8m6" Oct 30 16:26:50.923970 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:50.923924 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6f8685f4cf-fx429" Oct 30 16:26:50.924264 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:50.924230 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6n8m6" Oct 30 16:26:51.928339 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:51.928302 4493 generic.go:356] "Generic (PLEG): container finished" podID="11087140-99ef-44c4-b7fb-3320684b5aa7" containerID="dd7439dd1586b8fcb54ec5592c1acfdc568cc858961c495a63718ea020271913" exitCode=0 Oct 30 16:26:51.928835 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:51.928422 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czd6zb" event={"ID":"11087140-99ef-44c4-b7fb-3320684b5aa7","Type":"ContainerDied","Data":"dd7439dd1586b8fcb54ec5592c1acfdc568cc858961c495a63718ea020271913"} Oct 30 16:26:51.933622 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:51.931442 4493 generic.go:356] "Generic (PLEG): container finished" podID="9c4ac6e2-9eb6-4a3e-a536-300d213dae4a" containerID="47d969a7c0901152d03e3eac2c1796a821a9c580e24600f579b3ed091b70ae8a" exitCode=0 Oct 30 16:26:51.933622 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:51.933142 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7157s48" event={"ID":"9c4ac6e2-9eb6-4a3e-a536-300d213dae4a","Type":"ContainerDied","Data":"47d969a7c0901152d03e3eac2c1796a821a9c580e24600f579b3ed091b70ae8a"} Oct 30 16:26:52.941644 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:52.940726 4493 generic.go:356] "Generic (PLEG): container finished" podID="9c4ac6e2-9eb6-4a3e-a536-300d213dae4a" containerID="a684ff9fb5791db0cb45a921cc6b1f8e644c7b8dd314fd545667b3771941d6bd" exitCode=0 Oct 30 16:26:52.941644 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:52.940841 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7157s48" event={"ID":"9c4ac6e2-9eb6-4a3e-a536-300d213dae4a","Type":"ContainerDied","Data":"a684ff9fb5791db0cb45a921cc6b1f8e644c7b8dd314fd545667b3771941d6bd"} Oct 30 16:26:55.958838 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:55.958441 4493 generic.go:356] "Generic (PLEG): container finished" podID="40060367-43c3-40cb-a2c2-dc587e7bae98" containerID="80dd6769af65af14dbe58bbcc59538314f8273c36d572409bc01d78b5134a516" exitCode=0 Oct 30 16:26:55.958838 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:55.958575 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vjtnw" event={"ID":"40060367-43c3-40cb-a2c2-dc587e7bae98","Type":"ContainerDied","Data":"80dd6769af65af14dbe58bbcc59538314f8273c36d572409bc01d78b5134a516"} Oct 30 16:26:55.961833 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:55.961771 4493 generic.go:356] "Generic (PLEG): container finished" podID="ac42d273-7025-4cdc-88b2-dfc39ed12a2a" containerID="d0322978ef12b5585a0dc37246c0c2e89a6409741490865ccf8022ed17fc5380" exitCode=0 Oct 30 16:26:55.961931 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:55.961894 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zcbk" event={"ID":"ac42d273-7025-4cdc-88b2-dfc39ed12a2a","Type":"ContainerDied","Data":"d0322978ef12b5585a0dc37246c0c2e89a6409741490865ccf8022ed17fc5380"} Oct 30 16:26:55.966039 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:55.965283 4493 generic.go:356] "Generic (PLEG): container finished" podID="2eb25641-1607-4558-85ed-a51033782b02" containerID="e125358c093d6a8ff599d6f0cb821a008a2ddb623607ea7b6cef80021a3e7a8e" exitCode=0 Oct 30 16:26:55.966039 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:55.965376 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awt54c" event={"ID":"2eb25641-1607-4558-85ed-a51033782b02","Type":"ContainerDied","Data":"e125358c093d6a8ff599d6f0cb821a008a2ddb623607ea7b6cef80021a3e7a8e"} Oct 30 16:26:55.968468 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:55.967885 4493 generic.go:356] "Generic (PLEG): container finished" podID="a1933525-3f60-4e41-aad1-157f7b74c2c5" containerID="9565fc2a00cf854b24627e709809d929e2fe152e1f697d51d9b1f44e8fb3c03b" exitCode=0 Oct 30 16:26:55.968468 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:55.967913 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jd2lp" event={"ID":"a1933525-3f60-4e41-aad1-157f7b74c2c5","Type":"ContainerDied","Data":"9565fc2a00cf854b24627e709809d929e2fe152e1f697d51d9b1f44e8fb3c03b"} Oct 30 16:26:56.974755 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:56.974720 4493 generic.go:356] "Generic (PLEG): container finished" podID="ac42d273-7025-4cdc-88b2-dfc39ed12a2a" containerID="c8979fde0e2cb3bf5556dccd6dbc9632f86067d80b28b5b930593e1449f1d6cf" exitCode=0 Oct 30 16:26:56.975203 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:56.974827 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zcbk" event={"ID":"ac42d273-7025-4cdc-88b2-dfc39ed12a2a","Type":"ContainerDied","Data":"c8979fde0e2cb3bf5556dccd6dbc9632f86067d80b28b5b930593e1449f1d6cf"} Oct 30 16:26:56.978176 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:56.978154 4493 generic.go:356] "Generic (PLEG): container finished" podID="2eb25641-1607-4558-85ed-a51033782b02" containerID="f11776a080885f6ce905ba4ff55da1bc94e49d207314b07a6975e1ca2225d3fd" exitCode=0 Oct 30 16:26:56.978289 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:56.978256 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awt54c" event={"ID":"2eb25641-1607-4558-85ed-a51033782b02","Type":"ContainerDied","Data":"f11776a080885f6ce905ba4ff55da1bc94e49d207314b07a6975e1ca2225d3fd"} Oct 30 16:26:56.980758 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:56.980733 4493 generic.go:356] "Generic (PLEG): container finished" podID="a1933525-3f60-4e41-aad1-157f7b74c2c5" containerID="f932d208d5f5b30fbf68950a97efeb475ba1ddc46128caea30f6dd308bb4fbdb" exitCode=0 Oct 30 16:26:56.980899 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:56.980859 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jd2lp" event={"ID":"a1933525-3f60-4e41-aad1-157f7b74c2c5","Type":"ContainerDied","Data":"f932d208d5f5b30fbf68950a97efeb475ba1ddc46128caea30f6dd308bb4fbdb"} Oct 30 16:26:56.983594 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:56.983572 4493 generic.go:356] "Generic (PLEG): container finished" podID="49dbf366-6e7b-4dcd-b357-b2460c81db05" containerID="d8e99be1a9464d712d559957f9ee60d9ced2dbd93ac2ecae4de0789a96e09f59" exitCode=0 Oct 30 16:26:56.983707 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:56.983650 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qsslj" event={"ID":"49dbf366-6e7b-4dcd-b357-b2460c81db05","Type":"ContainerDied","Data":"d8e99be1a9464d712d559957f9ee60d9ced2dbd93ac2ecae4de0789a96e09f59"} Oct 30 16:26:56.986064 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:56.985863 4493 generic.go:356] "Generic (PLEG): container finished" podID="11087140-99ef-44c4-b7fb-3320684b5aa7" containerID="e46c95f96a5aa6e9201c0a24bf43151ef3a74080c40dff3bb76fc1dc65bb5a77" exitCode=0 Oct 30 16:26:56.986064 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:56.985925 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czd6zb" event={"ID":"11087140-99ef-44c4-b7fb-3320684b5aa7","Type":"ContainerDied","Data":"e46c95f96a5aa6e9201c0a24bf43151ef3a74080c40dff3bb76fc1dc65bb5a77"} Oct 30 16:26:57.994325 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:57.994287 4493 generic.go:356] "Generic (PLEG): container finished" podID="40060367-43c3-40cb-a2c2-dc587e7bae98" containerID="4594b88e80848ed236f161c8067110608c13a6d8f9f4ed535d7759d3eb6a2ee3" exitCode=0 Oct 30 16:26:57.994811 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:26:57.994412 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vjtnw" event={"ID":"40060367-43c3-40cb-a2c2-dc587e7bae98","Type":"ContainerDied","Data":"4594b88e80848ed236f161c8067110608c13a6d8f9f4ed535d7759d3eb6a2ee3"} Oct 30 16:27:00.373245 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:27:00.373122 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 16:27:02.038604 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:02.038570 4493 generic.go:356] "Generic (PLEG): container finished" podID="49dbf366-6e7b-4dcd-b357-b2460c81db05" containerID="ee75093fc60fc6cb85e6a3bbbae28baec93f843e98ca58031b63e53cdee30ce7" exitCode=0 Oct 30 16:27:02.039079 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:02.038710 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qsslj" event={"ID":"49dbf366-6e7b-4dcd-b357-b2460c81db05","Type":"ContainerDied","Data":"ee75093fc60fc6cb85e6a3bbbae28baec93f843e98ca58031b63e53cdee30ce7"} Oct 30 16:27:07.087634 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:07.087544 4493 generic.go:356] "Generic (PLEG): container finished" podID="40060367-43c3-40cb-a2c2-dc587e7bae98" containerID="b0b71506305943f867063d7c12d9895f93b34b124355c2a92ebb1e68ff604e9f" exitCode=0 Oct 30 16:27:07.088167 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:07.087648 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vjtnw" event={"ID":"40060367-43c3-40cb-a2c2-dc587e7bae98","Type":"ContainerDied","Data":"b0b71506305943f867063d7c12d9895f93b34b124355c2a92ebb1e68ff604e9f"} Oct 30 16:27:07.091643 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:07.091611 4493 generic.go:356] "Generic (PLEG): container finished" podID="9c4ac6e2-9eb6-4a3e-a536-300d213dae4a" containerID="24696b269725ee16207b751da18443bcbc7d82d2afcf98c728376d46bf9e9d5a" exitCode=0 Oct 30 16:27:07.091813 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:07.091721 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7157s48" event={"ID":"9c4ac6e2-9eb6-4a3e-a536-300d213dae4a","Type":"ContainerDied","Data":"24696b269725ee16207b751da18443bcbc7d82d2afcf98c728376d46bf9e9d5a"} Oct 30 16:27:08.320950 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:08.320920 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7157s48" Oct 30 16:27:08.328021 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:08.327935 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vjtnw" Oct 30 16:27:08.436122 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:08.436091 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40060367-43c3-40cb-a2c2-dc587e7bae98-kube-api-access-lkxpd\" (UniqueName: \"kubernetes.io/projected/40060367-43c3-40cb-a2c2-dc587e7bae98-kube-api-access-lkxpd\") pod \"40060367-43c3-40cb-a2c2-dc587e7bae98\" (UID: \"40060367-43c3-40cb-a2c2-dc587e7bae98\") " Oct 30 16:27:08.436309 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:08.436147 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c4ac6e2-9eb6-4a3e-a536-300d213dae4a-util\" (UniqueName: \"kubernetes.io/empty-dir/9c4ac6e2-9eb6-4a3e-a536-300d213dae4a-util\") pod \"9c4ac6e2-9eb6-4a3e-a536-300d213dae4a\" (UID: \"9c4ac6e2-9eb6-4a3e-a536-300d213dae4a\") " Oct 30 16:27:08.436309 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:08.436194 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c4ac6e2-9eb6-4a3e-a536-300d213dae4a-bundle\" (UniqueName: \"kubernetes.io/empty-dir/9c4ac6e2-9eb6-4a3e-a536-300d213dae4a-bundle\") pod \"9c4ac6e2-9eb6-4a3e-a536-300d213dae4a\" (UID: \"9c4ac6e2-9eb6-4a3e-a536-300d213dae4a\") " Oct 30 16:27:08.436309 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:08.436240 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c4ac6e2-9eb6-4a3e-a536-300d213dae4a-kube-api-access-h5vfl\" (UniqueName: \"kubernetes.io/projected/9c4ac6e2-9eb6-4a3e-a536-300d213dae4a-kube-api-access-h5vfl\") pod \"9c4ac6e2-9eb6-4a3e-a536-300d213dae4a\" (UID: \"9c4ac6e2-9eb6-4a3e-a536-300d213dae4a\") " Oct 30 16:27:08.436478 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:08.436324 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40060367-43c3-40cb-a2c2-dc587e7bae98-bundle\" (UniqueName: \"kubernetes.io/empty-dir/40060367-43c3-40cb-a2c2-dc587e7bae98-bundle\") pod \"40060367-43c3-40cb-a2c2-dc587e7bae98\" (UID: \"40060367-43c3-40cb-a2c2-dc587e7bae98\") " Oct 30 16:27:08.436478 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:08.436340 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40060367-43c3-40cb-a2c2-dc587e7bae98-util\" (UniqueName: \"kubernetes.io/empty-dir/40060367-43c3-40cb-a2c2-dc587e7bae98-util\") pod \"40060367-43c3-40cb-a2c2-dc587e7bae98\" (UID: \"40060367-43c3-40cb-a2c2-dc587e7bae98\") " Oct 30 16:27:08.438076 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:08.438031 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40060367-43c3-40cb-a2c2-dc587e7bae98-bundle" pod "40060367-43c3-40cb-a2c2-dc587e7bae98" (UID: "40060367-43c3-40cb-a2c2-dc587e7bae98"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:27:08.441232 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:08.441199 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c4ac6e2-9eb6-4a3e-a536-300d213dae4a-kube-api-access-h5vfl" pod "9c4ac6e2-9eb6-4a3e-a536-300d213dae4a" (UID: "9c4ac6e2-9eb6-4a3e-a536-300d213dae4a"). InnerVolumeSpecName "kube-api-access-h5vfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:27:08.441361 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:08.441290 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c4ac6e2-9eb6-4a3e-a536-300d213dae4a-bundle" pod "9c4ac6e2-9eb6-4a3e-a536-300d213dae4a" (UID: "9c4ac6e2-9eb6-4a3e-a536-300d213dae4a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:27:08.445095 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:08.445048 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40060367-43c3-40cb-a2c2-dc587e7bae98-kube-api-access-lkxpd" pod "40060367-43c3-40cb-a2c2-dc587e7bae98" (UID: "40060367-43c3-40cb-a2c2-dc587e7bae98"). InnerVolumeSpecName "kube-api-access-lkxpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:27:08.447687 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:08.447655 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40060367-43c3-40cb-a2c2-dc587e7bae98-util" pod "40060367-43c3-40cb-a2c2-dc587e7bae98" (UID: "40060367-43c3-40cb-a2c2-dc587e7bae98"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:27:08.447952 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:08.447927 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c4ac6e2-9eb6-4a3e-a536-300d213dae4a-util" pod "9c4ac6e2-9eb6-4a3e-a536-300d213dae4a" (UID: "9c4ac6e2-9eb6-4a3e-a536-300d213dae4a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:27:08.537184 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:08.537148 4493 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/40060367-43c3-40cb-a2c2-dc587e7bae98-bundle\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:27:08.537184 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:08.537177 4493 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/40060367-43c3-40cb-a2c2-dc587e7bae98-util\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:27:08.537409 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:08.537191 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkxpd\" (UniqueName: \"kubernetes.io/projected/40060367-43c3-40cb-a2c2-dc587e7bae98-kube-api-access-lkxpd\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:27:08.537409 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:08.537209 4493 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9c4ac6e2-9eb6-4a3e-a536-300d213dae4a-util\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:27:08.537409 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:08.537217 4493 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9c4ac6e2-9eb6-4a3e-a536-300d213dae4a-bundle\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:27:08.537409 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:08.537230 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5vfl\" (UniqueName: \"kubernetes.io/projected/9c4ac6e2-9eb6-4a3e-a536-300d213dae4a-kube-api-access-h5vfl\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:27:09.109811 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:09.109041 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vjtnw" event={"ID":"40060367-43c3-40cb-a2c2-dc587e7bae98","Type":"ContainerDied","Data":"25f775b458be01ed7549392e466301ab39af0b84e393e95fc46f45428dc029ac"} Oct 30 16:27:09.109811 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:09.109102 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25f775b458be01ed7549392e466301ab39af0b84e393e95fc46f45428dc029ac" Oct 30 16:27:09.109811 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:09.109173 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6vjtnw" Oct 30 16:27:09.113941 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:09.113916 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7157s48" event={"ID":"9c4ac6e2-9eb6-4a3e-a536-300d213dae4a","Type":"ContainerDied","Data":"ddc09d26ae807f10c7b5218423e1c3c027e4d2e528a5b8d8351bae755cdb1c10"} Oct 30 16:27:09.113941 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:09.113934 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be7157s48" Oct 30 16:27:09.114161 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:09.113957 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ddc09d26ae807f10c7b5218423e1c3c027e4d2e528a5b8d8351bae755cdb1c10" Oct 30 16:27:11.126939 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:11.126906 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jd2lp" event={"ID":"a1933525-3f60-4e41-aad1-157f7b74c2c5","Type":"ContainerStarted","Data":"cc30f7c163529a8291cb95339a2965ef15cc163d4a9b4965d31ea3e0d4713d8e"} Oct 30 16:27:11.129078 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:11.129050 4493 generic.go:356] "Generic (PLEG): container finished" podID="49dbf366-6e7b-4dcd-b357-b2460c81db05" containerID="136faff488c30c08fdbe90142aec891787433fda5d12c1b9d21fc7e3a3ab32fa" exitCode=0 Oct 30 16:27:11.129206 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:11.129131 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qsslj" event={"ID":"49dbf366-6e7b-4dcd-b357-b2460c81db05","Type":"ContainerDied","Data":"136faff488c30c08fdbe90142aec891787433fda5d12c1b9d21fc7e3a3ab32fa"} Oct 30 16:27:11.131130 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:11.131108 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zcbk" event={"ID":"ac42d273-7025-4cdc-88b2-dfc39ed12a2a","Type":"ContainerStarted","Data":"27d6b838bdc3d835c5cb38ed8ed8de8349b597e5d122a247889262095fda7e03"} Oct 30 16:27:11.171057 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:11.171013 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jd2lp" podStartSLOduration=2.079054018 podStartE2EDuration="32.171000324s" totalImagesPullingTime="30.091946306s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:26:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:27:11.169085811 +0000 UTC m=+258.396260910" watchObservedRunningTime="2026-10-30 16:27:11.171000324 +0000 UTC m=+258.398175424" Oct 30 16:27:11.200380 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:11.200349 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-hvbkm"] Oct 30 16:27:11.200695 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:11.200683 4493 cpu_manager.go:414] "containerMap: removing container" podUID="40060367-43c3-40cb-a2c2-dc587e7bae98" containerName="pull" Oct 30 16:27:11.200749 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:11.200697 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40060367-43c3-40cb-a2c2-dc587e7bae98" containerName="pull" Oct 30 16:27:11.200749 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:11.200705 4493 cpu_manager.go:414] "containerMap: removing container" podUID="9c4ac6e2-9eb6-4a3e-a536-300d213dae4a" containerName="extract" Oct 30 16:27:11.200749 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:11.200710 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c4ac6e2-9eb6-4a3e-a536-300d213dae4a" containerName="extract" Oct 30 16:27:11.200749 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:11.200727 4493 cpu_manager.go:414] "containerMap: removing container" podUID="40060367-43c3-40cb-a2c2-dc587e7bae98" containerName="extract" Oct 30 16:27:11.200749 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:11.200732 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40060367-43c3-40cb-a2c2-dc587e7bae98" containerName="extract" Oct 30 16:27:11.200749 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:11.200747 4493 cpu_manager.go:414] "containerMap: removing container" podUID="9c4ac6e2-9eb6-4a3e-a536-300d213dae4a" containerName="pull" Oct 30 16:27:11.200947 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:11.200752 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c4ac6e2-9eb6-4a3e-a536-300d213dae4a" containerName="pull" Oct 30 16:27:11.200947 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:11.200765 4493 cpu_manager.go:414] "containerMap: removing container" podUID="9c4ac6e2-9eb6-4a3e-a536-300d213dae4a" containerName="util" Oct 30 16:27:11.200947 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:11.200769 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c4ac6e2-9eb6-4a3e-a536-300d213dae4a" containerName="util" Oct 30 16:27:11.200947 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:11.200776 4493 cpu_manager.go:414] "containerMap: removing container" podUID="40060367-43c3-40cb-a2c2-dc587e7bae98" containerName="util" Oct 30 16:27:11.200947 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:11.200800 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40060367-43c3-40cb-a2c2-dc587e7bae98" containerName="util" Oct 30 16:27:11.200947 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:11.200854 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c4ac6e2-9eb6-4a3e-a536-300d213dae4a" containerName="extract" Oct 30 16:27:11.200947 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:11.200865 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="40060367-43c3-40cb-a2c2-dc587e7bae98" containerName="extract" Oct 30 16:27:11.221409 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:11.221388 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-hvbkm" Oct 30 16:27:11.250979 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:11.250957 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 16:27:11.251131 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:11.251105 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 16:27:11.251231 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:11.251156 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-jwzjl\"" Oct 30 16:27:11.251231 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:11.251164 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 16:27:11.251231 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:11.251171 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 16:27:11.251231 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:11.251202 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 16:27:11.264372 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:11.264349 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/97327913-e240-41af-8e5a-dee9155bf11e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6cbf9dc6c8-hvbkm\" (UID: \"97327913-e240-41af-8e5a-dee9155bf11e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-hvbkm" Oct 30 16:27:11.264473 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:11.264383 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/97327913-e240-41af-8e5a-dee9155bf11e-webhook-cert\") pod \"opendatahub-operator-controller-manager-6cbf9dc6c8-hvbkm\" (UID: \"97327913-e240-41af-8e5a-dee9155bf11e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-hvbkm" Oct 30 16:27:11.264473 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:11.264418 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcq44\" (UniqueName: \"kubernetes.io/projected/97327913-e240-41af-8e5a-dee9155bf11e-kube-api-access-fcq44\") pod \"opendatahub-operator-controller-manager-6cbf9dc6c8-hvbkm\" (UID: \"97327913-e240-41af-8e5a-dee9155bf11e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-hvbkm" Oct 30 16:27:11.264568 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:11.264489 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97327913-e240-41af-8e5a-dee9155bf11e-tmp\") pod \"opendatahub-operator-controller-manager-6cbf9dc6c8-hvbkm\" (UID: \"97327913-e240-41af-8e5a-dee9155bf11e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-hvbkm" Oct 30 16:27:11.264568 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:11.264530 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/97327913-e240-41af-8e5a-dee9155bf11e-manifests\") pod \"opendatahub-operator-controller-manager-6cbf9dc6c8-hvbkm\" (UID: \"97327913-e240-41af-8e5a-dee9155bf11e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-hvbkm" Oct 30 16:27:11.264667 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:11.264604 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/97327913-e240-41af-8e5a-dee9155bf11e-metrics-cert\") pod \"opendatahub-operator-controller-manager-6cbf9dc6c8-hvbkm\" (UID: \"97327913-e240-41af-8e5a-dee9155bf11e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-hvbkm" Oct 30 16:27:11.273233 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:11.273179 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-hvbkm"] Oct 30 16:27:11.275261 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:11.275214 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zcbk" podStartSLOduration=2.1419819860000002 podStartE2EDuration="32.275199095s" totalImagesPullingTime="30.133217109s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:26:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:27:11.274668839 +0000 UTC m=+258.501843939" watchObservedRunningTime="2026-10-30 16:27:11.275199095 +0000 UTC m=+258.502374196" Oct 30 16:27:11.365246 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:11.365220 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcq44\" (UniqueName: \"kubernetes.io/projected/97327913-e240-41af-8e5a-dee9155bf11e-kube-api-access-fcq44\") pod \"opendatahub-operator-controller-manager-6cbf9dc6c8-hvbkm\" (UID: \"97327913-e240-41af-8e5a-dee9155bf11e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-hvbkm" Oct 30 16:27:11.365367 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:11.365259 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97327913-e240-41af-8e5a-dee9155bf11e-tmp\") pod \"opendatahub-operator-controller-manager-6cbf9dc6c8-hvbkm\" (UID: \"97327913-e240-41af-8e5a-dee9155bf11e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-hvbkm" Oct 30 16:27:11.365367 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:11.365305 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/97327913-e240-41af-8e5a-dee9155bf11e-manifests\") pod \"opendatahub-operator-controller-manager-6cbf9dc6c8-hvbkm\" (UID: \"97327913-e240-41af-8e5a-dee9155bf11e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-hvbkm" Oct 30 16:27:11.365489 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:11.365397 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/97327913-e240-41af-8e5a-dee9155bf11e-metrics-cert\") pod \"opendatahub-operator-controller-manager-6cbf9dc6c8-hvbkm\" (UID: \"97327913-e240-41af-8e5a-dee9155bf11e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-hvbkm" Oct 30 16:27:11.365489 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:11.365427 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/97327913-e240-41af-8e5a-dee9155bf11e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6cbf9dc6c8-hvbkm\" (UID: \"97327913-e240-41af-8e5a-dee9155bf11e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-hvbkm" Oct 30 16:27:11.365489 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:11.365462 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/97327913-e240-41af-8e5a-dee9155bf11e-webhook-cert\") pod \"opendatahub-operator-controller-manager-6cbf9dc6c8-hvbkm\" (UID: \"97327913-e240-41af-8e5a-dee9155bf11e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-hvbkm" Oct 30 16:27:11.365824 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:11.365773 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/97327913-e240-41af-8e5a-dee9155bf11e-manifests\") pod \"opendatahub-operator-controller-manager-6cbf9dc6c8-hvbkm\" (UID: \"97327913-e240-41af-8e5a-dee9155bf11e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-hvbkm" Oct 30 16:27:11.367894 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:11.367872 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/97327913-e240-41af-8e5a-dee9155bf11e-webhook-cert\") pod \"opendatahub-operator-controller-manager-6cbf9dc6c8-hvbkm\" (UID: \"97327913-e240-41af-8e5a-dee9155bf11e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-hvbkm" Oct 30 16:27:11.367985 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:11.367975 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/97327913-e240-41af-8e5a-dee9155bf11e-metrics-cert\") pod \"opendatahub-operator-controller-manager-6cbf9dc6c8-hvbkm\" (UID: \"97327913-e240-41af-8e5a-dee9155bf11e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-hvbkm" Oct 30 16:27:11.368160 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:11.368144 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97327913-e240-41af-8e5a-dee9155bf11e-tmp\") pod \"opendatahub-operator-controller-manager-6cbf9dc6c8-hvbkm\" (UID: \"97327913-e240-41af-8e5a-dee9155bf11e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-hvbkm" Oct 30 16:27:11.368352 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:11.368336 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/97327913-e240-41af-8e5a-dee9155bf11e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6cbf9dc6c8-hvbkm\" (UID: \"97327913-e240-41af-8e5a-dee9155bf11e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-hvbkm" Oct 30 16:27:11.389217 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:11.389196 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcq44\" (UniqueName: \"kubernetes.io/projected/97327913-e240-41af-8e5a-dee9155bf11e-kube-api-access-fcq44\") pod \"opendatahub-operator-controller-manager-6cbf9dc6c8-hvbkm\" (UID: \"97327913-e240-41af-8e5a-dee9155bf11e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-hvbkm" Oct 30 16:27:11.591034 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:11.591004 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-hvbkm" Oct 30 16:27:11.963082 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:11.963058 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-hvbkm"] Oct 30 16:27:11.965084 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:27:11.965057 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97327913_e240_41af_8e5a_dee9155bf11e.slice/crio-6d99a3dd5e4dc0d82486c75bac38e76d0dc8657f2cc6c30995372515a9da7d7c WatchSource:0}: Error finding container 6d99a3dd5e4dc0d82486c75bac38e76d0dc8657f2cc6c30995372515a9da7d7c: Status 404 returned error can't find the container with id 6d99a3dd5e4dc0d82486c75bac38e76d0dc8657f2cc6c30995372515a9da7d7c Oct 30 16:27:12.138010 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:12.137912 4493 generic.go:356] "Generic (PLEG): container finished" podID="11087140-99ef-44c4-b7fb-3320684b5aa7" containerID="8d05ac17b4a1732d46c182c9eefd7545866e799581633e6edc3b7aa139da2bc2" exitCode=0 Oct 30 16:27:12.138010 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:12.137992 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czd6zb" event={"ID":"11087140-99ef-44c4-b7fb-3320684b5aa7","Type":"ContainerDied","Data":"8d05ac17b4a1732d46c182c9eefd7545866e799581633e6edc3b7aa139da2bc2"} Oct 30 16:27:12.140171 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:12.140150 4493 generic.go:356] "Generic (PLEG): container finished" podID="ac42d273-7025-4cdc-88b2-dfc39ed12a2a" containerID="27d6b838bdc3d835c5cb38ed8ed8de8349b597e5d122a247889262095fda7e03" exitCode=0 Oct 30 16:27:12.140258 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:12.140235 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zcbk" event={"ID":"ac42d273-7025-4cdc-88b2-dfc39ed12a2a","Type":"ContainerDied","Data":"27d6b838bdc3d835c5cb38ed8ed8de8349b597e5d122a247889262095fda7e03"} Oct 30 16:27:12.142281 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:12.142257 4493 generic.go:356] "Generic (PLEG): container finished" podID="2eb25641-1607-4558-85ed-a51033782b02" containerID="49c7ba2d0dca258b57e039f7a41e2fdeef96c8a4359e4a5d5dca815b78541d02" exitCode=0 Oct 30 16:27:12.142383 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:12.142283 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awt54c" event={"ID":"2eb25641-1607-4558-85ed-a51033782b02","Type":"ContainerDied","Data":"49c7ba2d0dca258b57e039f7a41e2fdeef96c8a4359e4a5d5dca815b78541d02"} Oct 30 16:27:12.143514 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:12.143488 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-hvbkm" event={"ID":"97327913-e240-41af-8e5a-dee9155bf11e","Type":"ContainerStarted","Data":"6d99a3dd5e4dc0d82486c75bac38e76d0dc8657f2cc6c30995372515a9da7d7c"} Oct 30 16:27:12.145413 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:12.145392 4493 generic.go:356] "Generic (PLEG): container finished" podID="a1933525-3f60-4e41-aad1-157f7b74c2c5" containerID="cc30f7c163529a8291cb95339a2965ef15cc163d4a9b4965d31ea3e0d4713d8e" exitCode=0 Oct 30 16:27:12.145497 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:12.145466 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jd2lp" event={"ID":"a1933525-3f60-4e41-aad1-157f7b74c2c5","Type":"ContainerDied","Data":"cc30f7c163529a8291cb95339a2965ef15cc163d4a9b4965d31ea3e0d4713d8e"} Oct 30 16:27:12.279821 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:12.279798 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qsslj" Oct 30 16:27:12.375086 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:12.375059 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49dbf366-6e7b-4dcd-b357-b2460c81db05-util\" (UniqueName: \"kubernetes.io/empty-dir/49dbf366-6e7b-4dcd-b357-b2460c81db05-util\") pod \"49dbf366-6e7b-4dcd-b357-b2460c81db05\" (UID: \"49dbf366-6e7b-4dcd-b357-b2460c81db05\") " Oct 30 16:27:12.375233 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:12.375094 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49dbf366-6e7b-4dcd-b357-b2460c81db05-bundle\" (UniqueName: \"kubernetes.io/empty-dir/49dbf366-6e7b-4dcd-b357-b2460c81db05-bundle\") pod \"49dbf366-6e7b-4dcd-b357-b2460c81db05\" (UID: \"49dbf366-6e7b-4dcd-b357-b2460c81db05\") " Oct 30 16:27:12.375233 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:12.375180 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49dbf366-6e7b-4dcd-b357-b2460c81db05-kube-api-access-slbkf\" (UniqueName: \"kubernetes.io/projected/49dbf366-6e7b-4dcd-b357-b2460c81db05-kube-api-access-slbkf\") pod \"49dbf366-6e7b-4dcd-b357-b2460c81db05\" (UID: \"49dbf366-6e7b-4dcd-b357-b2460c81db05\") " Oct 30 16:27:12.375547 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:12.375521 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49dbf366-6e7b-4dcd-b357-b2460c81db05-bundle" pod "49dbf366-6e7b-4dcd-b357-b2460c81db05" (UID: "49dbf366-6e7b-4dcd-b357-b2460c81db05"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:27:12.377354 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:12.377334 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49dbf366-6e7b-4dcd-b357-b2460c81db05-kube-api-access-slbkf" pod "49dbf366-6e7b-4dcd-b357-b2460c81db05" (UID: "49dbf366-6e7b-4dcd-b357-b2460c81db05"). InnerVolumeSpecName "kube-api-access-slbkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:27:12.380749 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:12.380719 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49dbf366-6e7b-4dcd-b357-b2460c81db05-util" pod "49dbf366-6e7b-4dcd-b357-b2460c81db05" (UID: "49dbf366-6e7b-4dcd-b357-b2460c81db05"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:27:12.475948 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:12.475922 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slbkf\" (UniqueName: \"kubernetes.io/projected/49dbf366-6e7b-4dcd-b357-b2460c81db05-kube-api-access-slbkf\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:27:12.475948 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:12.475945 4493 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/49dbf366-6e7b-4dcd-b357-b2460c81db05-util\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:27:12.476093 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:12.475956 4493 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/49dbf366-6e7b-4dcd-b357-b2460c81db05-bundle\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:27:13.271637 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:13.152532 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qsslj" Oct 30 16:27:13.271637 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:13.152579 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644qsslj" event={"ID":"49dbf366-6e7b-4dcd-b357-b2460c81db05","Type":"ContainerDied","Data":"c51bb4a5141295baf4fbb8b11b14ce876d6793cafdabd4e997f90626a6a6751c"} Oct 30 16:27:13.271637 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:13.152633 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c51bb4a5141295baf4fbb8b11b14ce876d6793cafdabd4e997f90626a6a6751c" Oct 30 16:27:13.350711 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:13.350676 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zcbk" Oct 30 16:27:13.384020 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:13.383960 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac42d273-7025-4cdc-88b2-dfc39ed12a2a-util\" (UniqueName: \"kubernetes.io/empty-dir/ac42d273-7025-4cdc-88b2-dfc39ed12a2a-util\") pod \"ac42d273-7025-4cdc-88b2-dfc39ed12a2a\" (UID: \"ac42d273-7025-4cdc-88b2-dfc39ed12a2a\") " Oct 30 16:27:13.391186 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:13.391155 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac42d273-7025-4cdc-88b2-dfc39ed12a2a-util" pod "ac42d273-7025-4cdc-88b2-dfc39ed12a2a" (UID: "ac42d273-7025-4cdc-88b2-dfc39ed12a2a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:27:13.391437 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:13.391414 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac42d273-7025-4cdc-88b2-dfc39ed12a2a-bundle\" (UniqueName: \"kubernetes.io/empty-dir/ac42d273-7025-4cdc-88b2-dfc39ed12a2a-bundle\") pod \"ac42d273-7025-4cdc-88b2-dfc39ed12a2a\" (UID: \"ac42d273-7025-4cdc-88b2-dfc39ed12a2a\") " Oct 30 16:27:13.391518 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:13.391462 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac42d273-7025-4cdc-88b2-dfc39ed12a2a-kube-api-access-88r82\" (UniqueName: \"kubernetes.io/projected/ac42d273-7025-4cdc-88b2-dfc39ed12a2a-kube-api-access-88r82\") pod \"ac42d273-7025-4cdc-88b2-dfc39ed12a2a\" (UID: \"ac42d273-7025-4cdc-88b2-dfc39ed12a2a\") " Oct 30 16:27:13.392181 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:13.392157 4493 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ac42d273-7025-4cdc-88b2-dfc39ed12a2a-util\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:27:13.394273 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:13.394242 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac42d273-7025-4cdc-88b2-dfc39ed12a2a-bundle" pod "ac42d273-7025-4cdc-88b2-dfc39ed12a2a" (UID: "ac42d273-7025-4cdc-88b2-dfc39ed12a2a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:27:13.394515 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:13.394479 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac42d273-7025-4cdc-88b2-dfc39ed12a2a-kube-api-access-88r82" pod "ac42d273-7025-4cdc-88b2-dfc39ed12a2a" (UID: "ac42d273-7025-4cdc-88b2-dfc39ed12a2a"). InnerVolumeSpecName "kube-api-access-88r82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:27:13.422493 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:13.422224 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jd2lp" Oct 30 16:27:13.447466 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:13.447437 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awt54c" Oct 30 16:27:13.461273 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:13.461232 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czd6zb" Oct 30 16:27:13.493250 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:13.493221 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eb25641-1607-4558-85ed-a51033782b02-util\" (UniqueName: \"kubernetes.io/empty-dir/2eb25641-1607-4558-85ed-a51033782b02-util\") pod \"2eb25641-1607-4558-85ed-a51033782b02\" (UID: \"2eb25641-1607-4558-85ed-a51033782b02\") " Oct 30 16:27:13.493386 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:13.493288 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eb25641-1607-4558-85ed-a51033782b02-bundle\" (UniqueName: \"kubernetes.io/empty-dir/2eb25641-1607-4558-85ed-a51033782b02-bundle\") pod \"2eb25641-1607-4558-85ed-a51033782b02\" (UID: \"2eb25641-1607-4558-85ed-a51033782b02\") " Oct 30 16:27:13.493386 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:13.493317 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1933525-3f60-4e41-aad1-157f7b74c2c5-util\" (UniqueName: \"kubernetes.io/empty-dir/a1933525-3f60-4e41-aad1-157f7b74c2c5-util\") pod \"a1933525-3f60-4e41-aad1-157f7b74c2c5\" (UID: \"a1933525-3f60-4e41-aad1-157f7b74c2c5\") " Oct 30 16:27:13.493386 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:13.493369 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1933525-3f60-4e41-aad1-157f7b74c2c5-kube-api-access-4ltdk\" (UniqueName: \"kubernetes.io/projected/a1933525-3f60-4e41-aad1-157f7b74c2c5-kube-api-access-4ltdk\") pod \"a1933525-3f60-4e41-aad1-157f7b74c2c5\" (UID: \"a1933525-3f60-4e41-aad1-157f7b74c2c5\") " Oct 30 16:27:13.493559 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:13.493453 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1933525-3f60-4e41-aad1-157f7b74c2c5-bundle\" (UniqueName: \"kubernetes.io/empty-dir/a1933525-3f60-4e41-aad1-157f7b74c2c5-bundle\") pod \"a1933525-3f60-4e41-aad1-157f7b74c2c5\" (UID: \"a1933525-3f60-4e41-aad1-157f7b74c2c5\") " Oct 30 16:27:13.493559 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:13.493493 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2eb25641-1607-4558-85ed-a51033782b02-kube-api-access-4mnnc\" (UniqueName: \"kubernetes.io/projected/2eb25641-1607-4558-85ed-a51033782b02-kube-api-access-4mnnc\") pod \"2eb25641-1607-4558-85ed-a51033782b02\" (UID: \"2eb25641-1607-4558-85ed-a51033782b02\") " Oct 30 16:27:13.493819 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:13.493779 4493 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ac42d273-7025-4cdc-88b2-dfc39ed12a2a-bundle\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:27:13.493902 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:13.493828 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88r82\" (UniqueName: \"kubernetes.io/projected/ac42d273-7025-4cdc-88b2-dfc39ed12a2a-kube-api-access-88r82\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:27:13.494400 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:13.494368 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eb25641-1607-4558-85ed-a51033782b02-bundle" pod "2eb25641-1607-4558-85ed-a51033782b02" (UID: "2eb25641-1607-4558-85ed-a51033782b02"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:27:13.496163 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:13.496136 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1933525-3f60-4e41-aad1-157f7b74c2c5-bundle" pod "a1933525-3f60-4e41-aad1-157f7b74c2c5" (UID: "a1933525-3f60-4e41-aad1-157f7b74c2c5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:27:13.503033 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:13.503006 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eb25641-1607-4558-85ed-a51033782b02-util" pod "2eb25641-1607-4558-85ed-a51033782b02" (UID: "2eb25641-1607-4558-85ed-a51033782b02"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:27:13.503148 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:13.503103 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1933525-3f60-4e41-aad1-157f7b74c2c5-util" pod "a1933525-3f60-4e41-aad1-157f7b74c2c5" (UID: "a1933525-3f60-4e41-aad1-157f7b74c2c5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:27:13.503564 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:13.503540 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1933525-3f60-4e41-aad1-157f7b74c2c5-kube-api-access-4ltdk" pod "a1933525-3f60-4e41-aad1-157f7b74c2c5" (UID: "a1933525-3f60-4e41-aad1-157f7b74c2c5"). InnerVolumeSpecName "kube-api-access-4ltdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:27:13.503647 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:13.503596 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eb25641-1607-4558-85ed-a51033782b02-kube-api-access-4mnnc" pod "2eb25641-1607-4558-85ed-a51033782b02" (UID: "2eb25641-1607-4558-85ed-a51033782b02"). InnerVolumeSpecName "kube-api-access-4mnnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:27:13.594876 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:13.594777 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11087140-99ef-44c4-b7fb-3320684b5aa7-kube-api-access-4qf22\" (UniqueName: \"kubernetes.io/projected/11087140-99ef-44c4-b7fb-3320684b5aa7-kube-api-access-4qf22\") pod \"11087140-99ef-44c4-b7fb-3320684b5aa7\" (UID: \"11087140-99ef-44c4-b7fb-3320684b5aa7\") " Oct 30 16:27:13.595044 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:13.594904 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11087140-99ef-44c4-b7fb-3320684b5aa7-util\" (UniqueName: \"kubernetes.io/empty-dir/11087140-99ef-44c4-b7fb-3320684b5aa7-util\") pod \"11087140-99ef-44c4-b7fb-3320684b5aa7\" (UID: \"11087140-99ef-44c4-b7fb-3320684b5aa7\") " Oct 30 16:27:13.595044 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:13.594986 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11087140-99ef-44c4-b7fb-3320684b5aa7-bundle\" (UniqueName: \"kubernetes.io/empty-dir/11087140-99ef-44c4-b7fb-3320684b5aa7-bundle\") pod \"11087140-99ef-44c4-b7fb-3320684b5aa7\" (UID: \"11087140-99ef-44c4-b7fb-3320684b5aa7\") " Oct 30 16:27:13.595294 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:13.595248 4493 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2eb25641-1607-4558-85ed-a51033782b02-util\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:27:13.595418 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:13.595310 4493 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2eb25641-1607-4558-85ed-a51033782b02-bundle\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:27:13.595418 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:13.595327 4493 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a1933525-3f60-4e41-aad1-157f7b74c2c5-util\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:27:13.595418 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:13.595342 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ltdk\" (UniqueName: \"kubernetes.io/projected/a1933525-3f60-4e41-aad1-157f7b74c2c5-kube-api-access-4ltdk\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:27:13.595418 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:13.595357 4493 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a1933525-3f60-4e41-aad1-157f7b74c2c5-bundle\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:27:13.595418 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:13.595366 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mnnc\" (UniqueName: \"kubernetes.io/projected/2eb25641-1607-4558-85ed-a51033782b02-kube-api-access-4mnnc\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:27:13.595845 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:13.595817 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11087140-99ef-44c4-b7fb-3320684b5aa7-bundle" pod "11087140-99ef-44c4-b7fb-3320684b5aa7" (UID: "11087140-99ef-44c4-b7fb-3320684b5aa7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:27:13.597474 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:13.597446 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11087140-99ef-44c4-b7fb-3320684b5aa7-kube-api-access-4qf22" pod "11087140-99ef-44c4-b7fb-3320684b5aa7" (UID: "11087140-99ef-44c4-b7fb-3320684b5aa7"). InnerVolumeSpecName "kube-api-access-4qf22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:27:13.602479 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:13.602455 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11087140-99ef-44c4-b7fb-3320684b5aa7-util" pod "11087140-99ef-44c4-b7fb-3320684b5aa7" (UID: "11087140-99ef-44c4-b7fb-3320684b5aa7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:27:13.696728 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:13.696646 4493 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/11087140-99ef-44c4-b7fb-3320684b5aa7-util\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:27:13.696728 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:13.696684 4493 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/11087140-99ef-44c4-b7fb-3320684b5aa7-bundle\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:27:13.696728 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:13.696700 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qf22\" (UniqueName: \"kubernetes.io/projected/11087140-99ef-44c4-b7fb-3320684b5aa7-kube-api-access-4qf22\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:27:14.158054 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:14.158027 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jd2lp" Oct 30 16:27:14.158221 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:14.158028 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7jd2lp" event={"ID":"a1933525-3f60-4e41-aad1-157f7b74c2c5","Type":"ContainerDied","Data":"1635b6846cd8c10e6cbf128a62931d5ce56286be27e366770543c91ce97ea6a0"} Oct 30 16:27:14.158221 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:14.158153 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1635b6846cd8c10e6cbf128a62931d5ce56286be27e366770543c91ce97ea6a0" Oct 30 16:27:14.159928 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:14.159909 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czd6zb" Oct 30 16:27:14.159928 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:14.159921 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czd6zb" event={"ID":"11087140-99ef-44c4-b7fb-3320684b5aa7","Type":"ContainerDied","Data":"3865f23b0cf35f5e13da71d34459bbc50593ec1d14326ecc67d65183ff81d3a2"} Oct 30 16:27:14.160096 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:14.159955 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3865f23b0cf35f5e13da71d34459bbc50593ec1d14326ecc67d65183ff81d3a2" Oct 30 16:27:14.161883 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:14.161863 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zcbk" event={"ID":"ac42d273-7025-4cdc-88b2-dfc39ed12a2a","Type":"ContainerDied","Data":"44264949079d01a83d0fda5e472a81612909ab708ab147fe175910fd05f2c512"} Oct 30 16:27:14.161883 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:14.161875 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9zcbk" Oct 30 16:27:14.162052 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:14.161894 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44264949079d01a83d0fda5e472a81612909ab708ab147fe175910fd05f2c512" Oct 30 16:27:14.163728 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:14.163708 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awt54c" event={"ID":"2eb25641-1607-4558-85ed-a51033782b02","Type":"ContainerDied","Data":"dbecaa2e0a217b7db408567a99a762a0e938200c39b285ea7d2f2a0f96631e42"} Oct 30 16:27:14.163840 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:14.163742 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dbecaa2e0a217b7db408567a99a762a0e938200c39b285ea7d2f2a0f96631e42" Oct 30 16:27:14.163840 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:14.163748 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847awt54c" Oct 30 16:27:15.169568 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:15.169538 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-hvbkm" event={"ID":"97327913-e240-41af-8e5a-dee9155bf11e","Type":"ContainerStarted","Data":"10c6b67c31782470759981538388072d0b2464179d5ec4c8c24e9149c8f28332"} Oct 30 16:27:16.177327 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:16.177292 4493 generic.go:356] "Generic (PLEG): container finished" podID="97327913-e240-41af-8e5a-dee9155bf11e" containerID="10c6b67c31782470759981538388072d0b2464179d5ec4c8c24e9149c8f28332" exitCode=0 Oct 30 16:27:16.177724 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:16.177344 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-hvbkm" event={"ID":"97327913-e240-41af-8e5a-dee9155bf11e","Type":"ContainerDied","Data":"10c6b67c31782470759981538388072d0b2464179d5ec4c8c24e9149c8f28332"} Oct 30 16:27:17.183046 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:17.183005 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-hvbkm" event={"ID":"97327913-e240-41af-8e5a-dee9155bf11e","Type":"ContainerStarted","Data":"71ef748e4754e98049646b8304e0ae4287c971baf90a24e0a17055687f20679d"} Oct 30 16:27:17.183498 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:17.183151 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-hvbkm" Oct 30 16:27:17.268721 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:17.268669 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-hvbkm" podStartSLOduration=2.282777408 podStartE2EDuration="6.268655307s" totalImagesPullingTime="3.512618211s" totalInitContainerRuntime="473.259688ms" isStatefulPod=false podCreationTimestamp="2026-10-30 16:27:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:27:17.256157997 +0000 UTC m=+264.483333096" watchObservedRunningTime="2026-10-30 16:27:17.268655307 +0000 UTC m=+264.495830455" Oct 30 16:27:27.189877 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:27.189842 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5pphd"] Oct 30 16:27:27.190301 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:27.190286 4493 cpu_manager.go:414] "containerMap: removing container" podUID="11087140-99ef-44c4-b7fb-3320684b5aa7" containerName="util" Oct 30 16:27:27.190345 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:27.190303 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11087140-99ef-44c4-b7fb-3320684b5aa7" containerName="util" Oct 30 16:27:27.190345 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:27.190322 4493 cpu_manager.go:414] "containerMap: removing container" podUID="2eb25641-1607-4558-85ed-a51033782b02" containerName="pull" Oct 30 16:27:27.190345 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:27.190330 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2eb25641-1607-4558-85ed-a51033782b02" containerName="pull" Oct 30 16:27:27.190345 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:27.190339 4493 cpu_manager.go:414] "containerMap: removing container" podUID="ac42d273-7025-4cdc-88b2-dfc39ed12a2a" containerName="util" Oct 30 16:27:27.190466 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:27.190347 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac42d273-7025-4cdc-88b2-dfc39ed12a2a" containerName="util" Oct 30 16:27:27.190466 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:27.190357 4493 cpu_manager.go:414] "containerMap: removing container" podUID="49dbf366-6e7b-4dcd-b357-b2460c81db05" containerName="util" Oct 30 16:27:27.190466 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:27.190364 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49dbf366-6e7b-4dcd-b357-b2460c81db05" containerName="util" Oct 30 16:27:27.190466 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:27.190373 4493 cpu_manager.go:414] "containerMap: removing container" podUID="ac42d273-7025-4cdc-88b2-dfc39ed12a2a" containerName="extract" Oct 30 16:27:27.190466 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:27.190379 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac42d273-7025-4cdc-88b2-dfc39ed12a2a" containerName="extract" Oct 30 16:27:27.190466 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:27.190396 4493 cpu_manager.go:414] "containerMap: removing container" podUID="2eb25641-1607-4558-85ed-a51033782b02" containerName="extract" Oct 30 16:27:27.190466 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:27.190403 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2eb25641-1607-4558-85ed-a51033782b02" containerName="extract" Oct 30 16:27:27.190466 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:27.190414 4493 cpu_manager.go:414] "containerMap: removing container" podUID="11087140-99ef-44c4-b7fb-3320684b5aa7" containerName="extract" Oct 30 16:27:27.190466 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:27.190421 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11087140-99ef-44c4-b7fb-3320684b5aa7" containerName="extract" Oct 30 16:27:27.190466 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:27.190438 4493 cpu_manager.go:414] "containerMap: removing container" podUID="49dbf366-6e7b-4dcd-b357-b2460c81db05" containerName="extract" Oct 30 16:27:27.190466 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:27.190446 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49dbf366-6e7b-4dcd-b357-b2460c81db05" containerName="extract" Oct 30 16:27:27.190466 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:27.190466 4493 cpu_manager.go:414] "containerMap: removing container" podUID="ac42d273-7025-4cdc-88b2-dfc39ed12a2a" containerName="pull" Oct 30 16:27:27.190836 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:27.190473 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac42d273-7025-4cdc-88b2-dfc39ed12a2a" containerName="pull" Oct 30 16:27:27.190836 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:27.190483 4493 cpu_manager.go:414] "containerMap: removing container" podUID="a1933525-3f60-4e41-aad1-157f7b74c2c5" containerName="util" Oct 30 16:27:27.190836 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:27.190490 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1933525-3f60-4e41-aad1-157f7b74c2c5" containerName="util" Oct 30 16:27:27.190836 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:27.190501 4493 cpu_manager.go:414] "containerMap: removing container" podUID="a1933525-3f60-4e41-aad1-157f7b74c2c5" containerName="pull" Oct 30 16:27:27.190836 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:27.190507 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1933525-3f60-4e41-aad1-157f7b74c2c5" containerName="pull" Oct 30 16:27:27.190836 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:27.190516 4493 cpu_manager.go:414] "containerMap: removing container" podUID="49dbf366-6e7b-4dcd-b357-b2460c81db05" containerName="pull" Oct 30 16:27:27.190836 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:27.190522 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49dbf366-6e7b-4dcd-b357-b2460c81db05" containerName="pull" Oct 30 16:27:27.190836 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:27.190531 4493 cpu_manager.go:414] "containerMap: removing container" podUID="2eb25641-1607-4558-85ed-a51033782b02" containerName="util" Oct 30 16:27:27.190836 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:27.190537 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2eb25641-1607-4558-85ed-a51033782b02" containerName="util" Oct 30 16:27:27.190836 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:27.190545 4493 cpu_manager.go:414] "containerMap: removing container" podUID="11087140-99ef-44c4-b7fb-3320684b5aa7" containerName="pull" Oct 30 16:27:27.190836 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:27.190551 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11087140-99ef-44c4-b7fb-3320684b5aa7" containerName="pull" Oct 30 16:27:27.190836 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:27.190559 4493 cpu_manager.go:414] "containerMap: removing container" podUID="a1933525-3f60-4e41-aad1-157f7b74c2c5" containerName="extract" Oct 30 16:27:27.190836 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:27.190565 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1933525-3f60-4e41-aad1-157f7b74c2c5" containerName="extract" Oct 30 16:27:27.190836 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:27.190627 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="2eb25641-1607-4558-85ed-a51033782b02" containerName="extract" Oct 30 16:27:27.190836 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:27.190638 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="11087140-99ef-44c4-b7fb-3320684b5aa7" containerName="extract" Oct 30 16:27:27.190836 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:27.190649 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="49dbf366-6e7b-4dcd-b357-b2460c81db05" containerName="extract" Oct 30 16:27:27.190836 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:27.190660 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1933525-3f60-4e41-aad1-157f7b74c2c5" containerName="extract" Oct 30 16:27:27.190836 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:27.190668 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac42d273-7025-4cdc-88b2-dfc39ed12a2a" containerName="extract" Oct 30 16:27:27.196875 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:27.196857 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5pphd" Oct 30 16:27:27.211314 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:27.211291 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5pphd"] Oct 30 16:27:27.234496 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:27.234454 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 16:27:27.234636 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:27.234498 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 16:27:27.234738 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:27.234719 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-qf7m7\"" Oct 30 16:27:27.309823 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:27.309771 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfvsz\" (UniqueName: \"kubernetes.io/projected/df551f9f-6ffb-4c16-bf9a-c0169466896d-kube-api-access-gfvsz\") pod \"kuadrant-operator-controller-manager-cf8469d7b-5pphd\" (UID: \"df551f9f-6ffb-4c16-bf9a-c0169466896d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5pphd" Oct 30 16:27:27.309996 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:27.309915 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/df551f9f-6ffb-4c16-bf9a-c0169466896d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-5pphd\" (UID: \"df551f9f-6ffb-4c16-bf9a-c0169466896d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5pphd" Oct 30 16:27:27.411135 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:27.411103 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/df551f9f-6ffb-4c16-bf9a-c0169466896d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-5pphd\" (UID: \"df551f9f-6ffb-4c16-bf9a-c0169466896d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5pphd" Oct 30 16:27:27.411287 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:27.411162 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfvsz\" (UniqueName: \"kubernetes.io/projected/df551f9f-6ffb-4c16-bf9a-c0169466896d-kube-api-access-gfvsz\") pod \"kuadrant-operator-controller-manager-cf8469d7b-5pphd\" (UID: \"df551f9f-6ffb-4c16-bf9a-c0169466896d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5pphd" Oct 30 16:27:27.411481 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:27.411462 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/df551f9f-6ffb-4c16-bf9a-c0169466896d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-5pphd\" (UID: \"df551f9f-6ffb-4c16-bf9a-c0169466896d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5pphd" Oct 30 16:27:27.447739 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:27.447679 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfvsz\" (UniqueName: \"kubernetes.io/projected/df551f9f-6ffb-4c16-bf9a-c0169466896d-kube-api-access-gfvsz\") pod \"kuadrant-operator-controller-manager-cf8469d7b-5pphd\" (UID: \"df551f9f-6ffb-4c16-bf9a-c0169466896d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5pphd" Oct 30 16:27:27.508650 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:27.508628 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5pphd" Oct 30 16:27:27.835546 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:27.835527 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5pphd"] Oct 30 16:27:27.837963 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:27:27.837920 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf551f9f_6ffb_4c16_bf9a_c0169466896d.slice/crio-d689ab476accc7354cf94984bdfb8c3869c7952f62b70eff0e47ce7226ebc108 WatchSource:0}: Error finding container d689ab476accc7354cf94984bdfb8c3869c7952f62b70eff0e47ce7226ebc108: Status 404 returned error can't find the container with id d689ab476accc7354cf94984bdfb8c3869c7952f62b70eff0e47ce7226ebc108 Oct 30 16:27:28.189643 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:28.189608 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-hvbkm" Oct 30 16:27:28.248299 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:28.248247 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5pphd" event={"ID":"df551f9f-6ffb-4c16-bf9a-c0169466896d","Type":"ContainerStarted","Data":"d689ab476accc7354cf94984bdfb8c3869c7952f62b70eff0e47ce7226ebc108"} Oct 30 16:27:30.348820 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:27:30.348740 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 16:27:30.358921 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:30.358891 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-24cc9"] Oct 30 16:27:30.369159 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:30.369135 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-24cc9" Oct 30 16:27:30.386865 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:30.386840 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-24cc9"] Oct 30 16:27:30.396430 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:30.396396 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Oct 30 16:27:30.396430 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:30.396403 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Oct 30 16:27:30.396430 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:30.396413 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-ltwf5\"" Oct 30 16:27:30.538573 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:30.538536 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a06d0128-4f48-44b3-86d2-5041a330d26c-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-24cc9\" (UID: \"a06d0128-4f48-44b3-86d2-5041a330d26c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-24cc9" Oct 30 16:27:30.538738 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:30.538633 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54dqp\" (UniqueName: \"kubernetes.io/projected/a06d0128-4f48-44b3-86d2-5041a330d26c-kube-api-access-54dqp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-24cc9\" (UID: \"a06d0128-4f48-44b3-86d2-5041a330d26c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-24cc9" Oct 30 16:27:30.639424 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:30.639351 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a06d0128-4f48-44b3-86d2-5041a330d26c-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-24cc9\" (UID: \"a06d0128-4f48-44b3-86d2-5041a330d26c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-24cc9" Oct 30 16:27:30.639424 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:30.639387 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54dqp\" (UniqueName: \"kubernetes.io/projected/a06d0128-4f48-44b3-86d2-5041a330d26c-kube-api-access-54dqp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-24cc9\" (UID: \"a06d0128-4f48-44b3-86d2-5041a330d26c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-24cc9" Oct 30 16:27:30.639776 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:30.639746 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a06d0128-4f48-44b3-86d2-5041a330d26c-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-24cc9\" (UID: \"a06d0128-4f48-44b3-86d2-5041a330d26c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-24cc9" Oct 30 16:27:30.662644 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:30.662611 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54dqp\" (UniqueName: \"kubernetes.io/projected/a06d0128-4f48-44b3-86d2-5041a330d26c-kube-api-access-54dqp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-24cc9\" (UID: \"a06d0128-4f48-44b3-86d2-5041a330d26c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-24cc9" Oct 30 16:27:30.684475 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:30.684446 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-24cc9" Oct 30 16:27:31.016910 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:31.016883 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-24cc9"] Oct 30 16:27:31.019352 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:27:31.019325 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda06d0128_4f48_44b3_86d2_5041a330d26c.slice/crio-244ef6b6ff9c78562242cf307236c09a75d78bd7e1fc56e208365ac67b1bb938 WatchSource:0}: Error finding container 244ef6b6ff9c78562242cf307236c09a75d78bd7e1fc56e208365ac67b1bb938: Status 404 returned error can't find the container with id 244ef6b6ff9c78562242cf307236c09a75d78bd7e1fc56e208365ac67b1bb938 Oct 30 16:27:31.230076 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:31.230035 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-h6wgm"] Oct 30 16:27:31.235430 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:31.235408 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-h6wgm" Oct 30 16:27:31.250121 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:31.250097 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-8rz5v\"" Oct 30 16:27:31.255445 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:31.255424 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-h6wgm"] Oct 30 16:27:31.266053 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:31.265978 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-24cc9" event={"ID":"a06d0128-4f48-44b3-86d2-5041a330d26c","Type":"ContainerStarted","Data":"244ef6b6ff9c78562242cf307236c09a75d78bd7e1fc56e208365ac67b1bb938"} Oct 30 16:27:31.347089 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:31.347053 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvr65\" (UniqueName: \"kubernetes.io/projected/3319a971-2f82-469a-8fc3-20687e710719-kube-api-access-vvr65\") pod \"limitador-operator-controller-manager-5778bb4449-h6wgm\" (UID: \"3319a971-2f82-469a-8fc3-20687e710719\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-h6wgm" Oct 30 16:27:31.448315 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:31.448282 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvr65\" (UniqueName: \"kubernetes.io/projected/3319a971-2f82-469a-8fc3-20687e710719-kube-api-access-vvr65\") pod \"limitador-operator-controller-manager-5778bb4449-h6wgm\" (UID: \"3319a971-2f82-469a-8fc3-20687e710719\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-h6wgm" Oct 30 16:27:31.474424 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:31.474392 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvr65\" (UniqueName: \"kubernetes.io/projected/3319a971-2f82-469a-8fc3-20687e710719-kube-api-access-vvr65\") pod \"limitador-operator-controller-manager-5778bb4449-h6wgm\" (UID: \"3319a971-2f82-469a-8fc3-20687e710719\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-h6wgm" Oct 30 16:27:31.552168 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:31.552091 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-h6wgm" Oct 30 16:27:31.896503 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:31.896476 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-h6wgm"] Oct 30 16:27:31.899030 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:27:31.898999 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3319a971_2f82_469a_8fc3_20687e710719.slice/crio-70bb56874466ef40dff9ee04cedbac0951e94a0d8b1147456da2127f7bdd5fe2 WatchSource:0}: Error finding container 70bb56874466ef40dff9ee04cedbac0951e94a0d8b1147456da2127f7bdd5fe2: Status 404 returned error can't find the container with id 70bb56874466ef40dff9ee04cedbac0951e94a0d8b1147456da2127f7bdd5fe2 Oct 30 16:27:32.273648 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:32.273612 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5pphd" event={"ID":"df551f9f-6ffb-4c16-bf9a-c0169466896d","Type":"ContainerStarted","Data":"6c9eb4984a12e2fe29ccfd4761211b4c2c4723d2f83d875469eb5fbfc225abea"} Oct 30 16:27:32.273836 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:32.273683 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5pphd" Oct 30 16:27:32.275134 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:32.275102 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-h6wgm" event={"ID":"3319a971-2f82-469a-8fc3-20687e710719","Type":"ContainerStarted","Data":"70bb56874466ef40dff9ee04cedbac0951e94a0d8b1147456da2127f7bdd5fe2"} Oct 30 16:27:32.319295 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:32.318996 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5pphd" podStartSLOduration=1.000607803 podStartE2EDuration="5.318981831s" totalImagesPullingTime="4.318374028s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:27:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:27:32.316629479 +0000 UTC m=+279.543804581" watchObservedRunningTime="2026-10-30 16:27:32.318981831 +0000 UTC m=+279.546156931" Oct 30 16:27:33.281796 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:33.281748 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-24cc9" event={"ID":"a06d0128-4f48-44b3-86d2-5041a330d26c","Type":"ContainerStarted","Data":"388fd91c37f94c1b604129db4eb38853770a20b1c757b9c2423b27ed074b3397"} Oct 30 16:27:33.350732 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:33.350675 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-24cc9" podStartSLOduration=1.6466142700000002 podStartE2EDuration="3.350661553s" totalImagesPullingTime="1.704047283s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:27:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:27:33.336960388 +0000 UTC m=+280.564135501" watchObservedRunningTime="2026-10-30 16:27:33.350661553 +0000 UTC m=+280.577836659" Oct 30 16:27:34.288162 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:34.288119 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-h6wgm" event={"ID":"3319a971-2f82-469a-8fc3-20687e710719","Type":"ContainerStarted","Data":"5efa55b141ec7f8db1da67e4712ca26a87c4fbe87d9df68e5a0d94b5ba6ccc52"} Oct 30 16:27:34.288546 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:34.288298 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-h6wgm" Oct 30 16:27:34.333352 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:34.333306 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-h6wgm" podStartSLOduration=1.621483543 podStartE2EDuration="3.333292422s" totalImagesPullingTime="1.711808879s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:27:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:27:34.333125305 +0000 UTC m=+281.560300406" watchObservedRunningTime="2026-10-30 16:27:34.333292422 +0000 UTC m=+281.560467909" Oct 30 16:27:36.725050 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:36.725023 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-gzvkk"] Oct 30 16:27:36.728237 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:36.728219 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-gzvkk" Oct 30 16:27:36.749923 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:36.749886 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-xjvl4\"" Oct 30 16:27:36.758520 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:36.758496 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-gzvkk"] Oct 30 16:27:36.899911 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:36.899881 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d6gm\" (UniqueName: \"kubernetes.io/projected/39b545e1-1a61-4ddf-a00a-f396058e6a88-kube-api-access-7d6gm\") pod \"authorino-operator-7b7bb4bc77-gzvkk\" (UID: \"39b545e1-1a61-4ddf-a00a-f396058e6a88\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-gzvkk" Oct 30 16:27:37.000855 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:37.000760 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d6gm\" (UniqueName: \"kubernetes.io/projected/39b545e1-1a61-4ddf-a00a-f396058e6a88-kube-api-access-7d6gm\") pod \"authorino-operator-7b7bb4bc77-gzvkk\" (UID: \"39b545e1-1a61-4ddf-a00a-f396058e6a88\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-gzvkk" Oct 30 16:27:37.031843 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:37.031813 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d6gm\" (UniqueName: \"kubernetes.io/projected/39b545e1-1a61-4ddf-a00a-f396058e6a88-kube-api-access-7d6gm\") pod \"authorino-operator-7b7bb4bc77-gzvkk\" (UID: \"39b545e1-1a61-4ddf-a00a-f396058e6a88\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-gzvkk" Oct 30 16:27:37.040535 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:37.040515 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-gzvkk" Oct 30 16:27:37.347592 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:37.347569 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-gzvkk"] Oct 30 16:27:37.349416 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:27:37.349387 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39b545e1_1a61_4ddf_a00a_f396058e6a88.slice/crio-de4d87932dd96defb4fd8ae9edcabff400e533c7d7a5f953aff78f178b5eff36 WatchSource:0}: Error finding container de4d87932dd96defb4fd8ae9edcabff400e533c7d7a5f953aff78f178b5eff36: Status 404 returned error can't find the container with id de4d87932dd96defb4fd8ae9edcabff400e533c7d7a5f953aff78f178b5eff36 Oct 30 16:27:38.308183 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:38.308146 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-gzvkk" event={"ID":"39b545e1-1a61-4ddf-a00a-f396058e6a88","Type":"ContainerStarted","Data":"de4d87932dd96defb4fd8ae9edcabff400e533c7d7a5f953aff78f178b5eff36"} Oct 30 16:27:39.315003 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:39.314969 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-gzvkk" event={"ID":"39b545e1-1a61-4ddf-a00a-f396058e6a88","Type":"ContainerStarted","Data":"e49a2610d47161822ad98fc2a39b88b9670a17d98cd78768f839fbf50ac3c65f"} Oct 30 16:27:39.315376 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:39.315096 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-gzvkk" Oct 30 16:27:39.357161 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:39.357113 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-gzvkk" podStartSLOduration=2.037597128 podStartE2EDuration="3.357099429s" totalImagesPullingTime="1.319502301s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:27:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:27:39.356079592 +0000 UTC m=+286.583254693" watchObservedRunningTime="2026-10-30 16:27:39.357099429 +0000 UTC m=+286.584274529" Oct 30 16:27:39.944880 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:39.944848 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-8xm6t"] Oct 30 16:27:39.948543 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:39.948522 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-8xm6t" Oct 30 16:27:39.964830 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:39.964804 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Oct 30 16:27:39.976245 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:39.976222 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-sqql5\"" Oct 30 16:27:39.978630 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:39.978609 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-8xm6t"] Oct 30 16:27:40.129632 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:40.129597 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj2dm\" (UniqueName: \"kubernetes.io/projected/cf7e09ae-b9ba-420d-b071-bc2f5ac8b009-kube-api-access-rj2dm\") pod \"dns-operator-controller-manager-8557cbdb84-8xm6t\" (UID: \"cf7e09ae-b9ba-420d-b071-bc2f5ac8b009\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-8xm6t" Oct 30 16:27:40.230828 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:40.230736 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj2dm\" (UniqueName: \"kubernetes.io/projected/cf7e09ae-b9ba-420d-b071-bc2f5ac8b009-kube-api-access-rj2dm\") pod \"dns-operator-controller-manager-8557cbdb84-8xm6t\" (UID: \"cf7e09ae-b9ba-420d-b071-bc2f5ac8b009\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-8xm6t" Oct 30 16:27:40.256269 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:40.256232 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj2dm\" (UniqueName: \"kubernetes.io/projected/cf7e09ae-b9ba-420d-b071-bc2f5ac8b009-kube-api-access-rj2dm\") pod \"dns-operator-controller-manager-8557cbdb84-8xm6t\" (UID: \"cf7e09ae-b9ba-420d-b071-bc2f5ac8b009\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-8xm6t" Oct 30 16:27:40.261437 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:40.261418 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-8xm6t" Oct 30 16:27:40.561926 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:40.561903 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-8xm6t"] Oct 30 16:27:40.564528 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:27:40.564501 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf7e09ae_b9ba_420d_b071_bc2f5ac8b009.slice/crio-b00941185c6f3ca392d50e8aef2284fc5ccbf7f47440a85a594dbbff814ec045 WatchSource:0}: Error finding container b00941185c6f3ca392d50e8aef2284fc5ccbf7f47440a85a594dbbff814ec045: Status 404 returned error can't find the container with id b00941185c6f3ca392d50e8aef2284fc5ccbf7f47440a85a594dbbff814ec045 Oct 30 16:27:41.330452 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:41.330412 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-8xm6t" event={"ID":"cf7e09ae-b9ba-420d-b071-bc2f5ac8b009","Type":"ContainerStarted","Data":"b00941185c6f3ca392d50e8aef2284fc5ccbf7f47440a85a594dbbff814ec045"} Oct 30 16:27:42.336656 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:42.336621 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-8xm6t" event={"ID":"cf7e09ae-b9ba-420d-b071-bc2f5ac8b009","Type":"ContainerStarted","Data":"ed135a121711d829ed066893e154d54ed2ea3efc3e14a628c46003408efa07aa"} Oct 30 16:27:42.337093 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:42.336743 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-8xm6t" Oct 30 16:27:42.378217 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:42.378164 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-8xm6t" podStartSLOduration=2.101879182 podStartE2EDuration="3.378148869s" totalImagesPullingTime="1.276269687s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:27:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:27:42.377612281 +0000 UTC m=+289.604787381" watchObservedRunningTime="2026-10-30 16:27:42.378148869 +0000 UTC m=+289.605323969" Oct 30 16:27:43.284805 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:43.284759 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5pphd" Oct 30 16:27:43.778141 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:43.778102 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-kksz8"] Oct 30 16:27:43.781556 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:43.781540 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-kksz8" Oct 30 16:27:43.809649 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:43.809626 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-kksz8"] Oct 30 16:27:43.865746 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:43.865713 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6594d922-089d-41e9-9444-5a888d5fa675-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-kksz8\" (UID: \"6594d922-089d-41e9-9444-5a888d5fa675\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-kksz8" Oct 30 16:27:43.865746 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:43.865750 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcjxz\" (UniqueName: \"kubernetes.io/projected/6594d922-089d-41e9-9444-5a888d5fa675-kube-api-access-pcjxz\") pod \"kuadrant-operator-controller-manager-678d87766f-kksz8\" (UID: \"6594d922-089d-41e9-9444-5a888d5fa675\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-kksz8" Oct 30 16:27:43.966986 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:43.966947 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcjxz\" (UniqueName: \"kubernetes.io/projected/6594d922-089d-41e9-9444-5a888d5fa675-kube-api-access-pcjxz\") pod \"kuadrant-operator-controller-manager-678d87766f-kksz8\" (UID: \"6594d922-089d-41e9-9444-5a888d5fa675\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-kksz8" Oct 30 16:27:43.967135 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:43.967040 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6594d922-089d-41e9-9444-5a888d5fa675-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-kksz8\" (UID: \"6594d922-089d-41e9-9444-5a888d5fa675\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-kksz8" Oct 30 16:27:43.967346 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:43.967331 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6594d922-089d-41e9-9444-5a888d5fa675-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-kksz8\" (UID: \"6594d922-089d-41e9-9444-5a888d5fa675\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-kksz8" Oct 30 16:27:43.990411 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:43.990387 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcjxz\" (UniqueName: \"kubernetes.io/projected/6594d922-089d-41e9-9444-5a888d5fa675-kube-api-access-pcjxz\") pod \"kuadrant-operator-controller-manager-678d87766f-kksz8\" (UID: \"6594d922-089d-41e9-9444-5a888d5fa675\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-kksz8" Oct 30 16:27:44.094105 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:44.094038 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-kksz8" Oct 30 16:27:44.417365 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:44.417342 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-kksz8"] Oct 30 16:27:44.419613 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:27:44.419581 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6594d922_089d_41e9_9444_5a888d5fa675.slice/crio-67c9d2a2ca661c97639f276f071476774d1de9602e0249e91a7db9f52c75dd59 WatchSource:0}: Error finding container 67c9d2a2ca661c97639f276f071476774d1de9602e0249e91a7db9f52c75dd59: Status 404 returned error can't find the container with id 67c9d2a2ca661c97639f276f071476774d1de9602e0249e91a7db9f52c75dd59 Oct 30 16:27:45.294553 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:45.294527 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-h6wgm" Oct 30 16:27:45.355236 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:45.355185 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-kksz8" event={"ID":"6594d922-089d-41e9-9444-5a888d5fa675","Type":"ContainerStarted","Data":"4750e14df33a8df4ccc10a8f082ad6fc1d596cceb8c12928ab3341b399a99203"} Oct 30 16:27:45.355449 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:45.355429 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-kksz8" Oct 30 16:27:45.355507 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:45.355480 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-kksz8" event={"ID":"6594d922-089d-41e9-9444-5a888d5fa675","Type":"ContainerStarted","Data":"67c9d2a2ca661c97639f276f071476774d1de9602e0249e91a7db9f52c75dd59"} Oct 30 16:27:45.399598 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:45.399453 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-kksz8" podStartSLOduration=2.399436776 podStartE2EDuration="2.399436776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:27:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:27:45.399067555 +0000 UTC m=+292.626242684" watchObservedRunningTime="2026-10-30 16:27:45.399436776 +0000 UTC m=+292.626611877" Oct 30 16:27:46.946932 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:46.946899 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789j7m84"] Oct 30 16:27:46.953874 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:46.953851 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789j7m84" Oct 30 16:27:46.969636 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:46.969615 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-n6ck9\"" Oct 30 16:27:46.983152 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:46.983127 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789j7m84"] Oct 30 16:27:46.993035 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:46.993005 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c9bc2df-9829-4346-a082-b8255c37bb3c-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789j7m84\" (UID: \"1c9bc2df-9829-4346-a082-b8255c37bb3c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789j7m84" Oct 30 16:27:46.993152 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:46.993049 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c9bc2df-9829-4346-a082-b8255c37bb3c-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789j7m84\" (UID: \"1c9bc2df-9829-4346-a082-b8255c37bb3c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789j7m84" Oct 30 16:27:46.993205 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:46.993149 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrdbl\" (UniqueName: \"kubernetes.io/projected/1c9bc2df-9829-4346-a082-b8255c37bb3c-kube-api-access-rrdbl\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789j7m84\" (UID: \"1c9bc2df-9829-4346-a082-b8255c37bb3c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789j7m84" Oct 30 16:27:46.993205 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:46.993187 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c9bc2df-9829-4346-a082-b8255c37bb3c-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789j7m84\" (UID: \"1c9bc2df-9829-4346-a082-b8255c37bb3c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789j7m84" Oct 30 16:27:46.993309 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:46.993215 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c9bc2df-9829-4346-a082-b8255c37bb3c-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789j7m84\" (UID: \"1c9bc2df-9829-4346-a082-b8255c37bb3c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789j7m84" Oct 30 16:27:46.993360 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:46.993317 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c9bc2df-9829-4346-a082-b8255c37bb3c-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789j7m84\" (UID: \"1c9bc2df-9829-4346-a082-b8255c37bb3c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789j7m84" Oct 30 16:27:46.993409 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:46.993360 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c9bc2df-9829-4346-a082-b8255c37bb3c-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789j7m84\" (UID: \"1c9bc2df-9829-4346-a082-b8255c37bb3c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789j7m84" Oct 30 16:27:46.993409 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:46.993391 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c9bc2df-9829-4346-a082-b8255c37bb3c-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789j7m84\" (UID: \"1c9bc2df-9829-4346-a082-b8255c37bb3c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789j7m84" Oct 30 16:27:46.993507 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:46.993463 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c9bc2df-9829-4346-a082-b8255c37bb3c-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789j7m84\" (UID: \"1c9bc2df-9829-4346-a082-b8255c37bb3c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789j7m84" Oct 30 16:27:47.094095 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:47.094059 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrdbl\" (UniqueName: \"kubernetes.io/projected/1c9bc2df-9829-4346-a082-b8255c37bb3c-kube-api-access-rrdbl\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789j7m84\" (UID: \"1c9bc2df-9829-4346-a082-b8255c37bb3c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789j7m84" Oct 30 16:27:47.094281 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:47.094106 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c9bc2df-9829-4346-a082-b8255c37bb3c-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789j7m84\" (UID: \"1c9bc2df-9829-4346-a082-b8255c37bb3c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789j7m84" Oct 30 16:27:47.094281 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:47.094143 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c9bc2df-9829-4346-a082-b8255c37bb3c-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789j7m84\" (UID: \"1c9bc2df-9829-4346-a082-b8255c37bb3c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789j7m84" Oct 30 16:27:47.094281 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:47.094167 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c9bc2df-9829-4346-a082-b8255c37bb3c-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789j7m84\" (UID: \"1c9bc2df-9829-4346-a082-b8255c37bb3c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789j7m84" Oct 30 16:27:47.094281 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:47.094195 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c9bc2df-9829-4346-a082-b8255c37bb3c-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789j7m84\" (UID: \"1c9bc2df-9829-4346-a082-b8255c37bb3c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789j7m84" Oct 30 16:27:47.094281 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:47.094221 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c9bc2df-9829-4346-a082-b8255c37bb3c-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789j7m84\" (UID: \"1c9bc2df-9829-4346-a082-b8255c37bb3c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789j7m84" Oct 30 16:27:47.094281 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:47.094262 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c9bc2df-9829-4346-a082-b8255c37bb3c-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789j7m84\" (UID: \"1c9bc2df-9829-4346-a082-b8255c37bb3c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789j7m84" Oct 30 16:27:47.094583 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:47.094300 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c9bc2df-9829-4346-a082-b8255c37bb3c-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789j7m84\" (UID: \"1c9bc2df-9829-4346-a082-b8255c37bb3c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789j7m84" Oct 30 16:27:47.094583 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:47.094350 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c9bc2df-9829-4346-a082-b8255c37bb3c-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789j7m84\" (UID: \"1c9bc2df-9829-4346-a082-b8255c37bb3c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789j7m84" Oct 30 16:27:47.094693 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:47.094616 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c9bc2df-9829-4346-a082-b8255c37bb3c-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789j7m84\" (UID: \"1c9bc2df-9829-4346-a082-b8255c37bb3c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789j7m84" Oct 30 16:27:47.094693 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:47.094653 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c9bc2df-9829-4346-a082-b8255c37bb3c-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789j7m84\" (UID: \"1c9bc2df-9829-4346-a082-b8255c37bb3c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789j7m84" Oct 30 16:27:47.094778 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:47.094761 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c9bc2df-9829-4346-a082-b8255c37bb3c-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789j7m84\" (UID: \"1c9bc2df-9829-4346-a082-b8255c37bb3c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789j7m84" Oct 30 16:27:47.094847 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:47.094828 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c9bc2df-9829-4346-a082-b8255c37bb3c-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789j7m84\" (UID: \"1c9bc2df-9829-4346-a082-b8255c37bb3c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789j7m84" Oct 30 16:27:47.095058 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:47.095040 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c9bc2df-9829-4346-a082-b8255c37bb3c-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789j7m84\" (UID: \"1c9bc2df-9829-4346-a082-b8255c37bb3c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789j7m84" Oct 30 16:27:47.097207 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:47.097181 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c9bc2df-9829-4346-a082-b8255c37bb3c-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789j7m84\" (UID: \"1c9bc2df-9829-4346-a082-b8255c37bb3c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789j7m84" Oct 30 16:27:47.097499 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:47.097477 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c9bc2df-9829-4346-a082-b8255c37bb3c-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789j7m84\" (UID: \"1c9bc2df-9829-4346-a082-b8255c37bb3c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789j7m84" Oct 30 16:27:47.120654 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:47.120624 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrdbl\" (UniqueName: \"kubernetes.io/projected/1c9bc2df-9829-4346-a082-b8255c37bb3c-kube-api-access-rrdbl\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789j7m84\" (UID: \"1c9bc2df-9829-4346-a082-b8255c37bb3c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789j7m84" Oct 30 16:27:47.124061 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:47.124041 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c9bc2df-9829-4346-a082-b8255c37bb3c-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7d45d5c789j7m84\" (UID: \"1c9bc2df-9829-4346-a082-b8255c37bb3c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789j7m84" Oct 30 16:27:47.266619 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:47.266528 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789j7m84" Oct 30 16:27:47.589158 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:47.589136 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789j7m84"] Oct 30 16:27:47.591120 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:27:47.591093 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c9bc2df_9829_4346_a082_b8255c37bb3c.slice/crio-706a75813d02377f601d7f8c31c942cd219c0b6f333d1eb6baf3c2a19ff764cd WatchSource:0}: Error finding container 706a75813d02377f601d7f8c31c942cd219c0b6f333d1eb6baf3c2a19ff764cd: Status 404 returned error can't find the container with id 706a75813d02377f601d7f8c31c942cd219c0b6f333d1eb6baf3c2a19ff764cd Oct 30 16:27:47.593228 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:47.593187 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:27:47.593328 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:47.593271 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:27:47.593328 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:47.593318 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:27:48.372518 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:48.372486 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789j7m84" event={"ID":"1c9bc2df-9829-4346-a082-b8255c37bb3c","Type":"ContainerStarted","Data":"b61f4c8f26f7a260ee95b9c783a8785c6dedb2636e2012ea2adcacfec11028fe"} Oct 30 16:27:48.372916 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:48.372538 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789j7m84" event={"ID":"1c9bc2df-9829-4346-a082-b8255c37bb3c","Type":"ContainerStarted","Data":"706a75813d02377f601d7f8c31c942cd219c0b6f333d1eb6baf3c2a19ff764cd"} Oct 30 16:27:48.428792 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:48.428746 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789j7m84" podStartSLOduration=2.428729732 podStartE2EDuration="2.428729732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:27:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:27:48.428108655 +0000 UTC m=+295.655283773" watchObservedRunningTime="2026-10-30 16:27:48.428729732 +0000 UTC m=+295.655904831" Oct 30 16:27:49.267367 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:49.267332 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789j7m84" Oct 30 16:27:49.272063 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:49.272033 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789j7m84" Oct 30 16:27:49.377016 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:49.376986 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789j7m84" Oct 30 16:27:49.377928 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:49.377911 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7d45d5c789j7m84" Oct 30 16:27:50.325465 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:50.325438 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-gzvkk" Oct 30 16:27:52.953042 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:52.953016 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-56v9s_d3a1c417-2698-4462-b7ff-1d2e66d7a10f/ovn-acl-logging/0.log" Oct 30 16:27:52.953924 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:52.953900 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-56v9s_d3a1c417-2698-4462-b7ff-1d2e66d7a10f/ovn-acl-logging/0.log" Oct 30 16:27:53.344702 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:53.344621 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-8xm6t" Oct 30 16:27:56.362322 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:56.362292 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-kksz8" Oct 30 16:27:56.451318 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:56.451287 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5pphd"] Oct 30 16:27:56.451569 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:56.451546 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5pphd" podUID="df551f9f-6ffb-4c16-bf9a-c0169466896d" containerName="manager" containerID="cri-o://6c9eb4984a12e2fe29ccfd4761211b4c2c4723d2f83d875469eb5fbfc225abea" gracePeriod=10 Oct 30 16:27:56.696342 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:56.696321 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5pphd" Oct 30 16:27:56.779830 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:56.779792 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df551f9f-6ffb-4c16-bf9a-c0169466896d-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/df551f9f-6ffb-4c16-bf9a-c0169466896d-extensions-socket-volume\") pod \"df551f9f-6ffb-4c16-bf9a-c0169466896d\" (UID: \"df551f9f-6ffb-4c16-bf9a-c0169466896d\") " Oct 30 16:27:56.780008 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:56.779858 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df551f9f-6ffb-4c16-bf9a-c0169466896d-kube-api-access-gfvsz\" (UniqueName: \"kubernetes.io/projected/df551f9f-6ffb-4c16-bf9a-c0169466896d-kube-api-access-gfvsz\") pod \"df551f9f-6ffb-4c16-bf9a-c0169466896d\" (UID: \"df551f9f-6ffb-4c16-bf9a-c0169466896d\") " Oct 30 16:27:56.780185 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:56.780159 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df551f9f-6ffb-4c16-bf9a-c0169466896d-extensions-socket-volume" pod "df551f9f-6ffb-4c16-bf9a-c0169466896d" (UID: "df551f9f-6ffb-4c16-bf9a-c0169466896d"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:27:56.782229 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:56.782205 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df551f9f-6ffb-4c16-bf9a-c0169466896d-kube-api-access-gfvsz" pod "df551f9f-6ffb-4c16-bf9a-c0169466896d" (UID: "df551f9f-6ffb-4c16-bf9a-c0169466896d"). InnerVolumeSpecName "kube-api-access-gfvsz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:27:56.881196 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:56.881112 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfvsz\" (UniqueName: \"kubernetes.io/projected/df551f9f-6ffb-4c16-bf9a-c0169466896d-kube-api-access-gfvsz\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:27:56.881196 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:56.881144 4493 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/df551f9f-6ffb-4c16-bf9a-c0169466896d-extensions-socket-volume\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:27:57.416945 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:57.416909 4493 generic.go:356] "Generic (PLEG): container finished" podID="df551f9f-6ffb-4c16-bf9a-c0169466896d" containerID="6c9eb4984a12e2fe29ccfd4761211b4c2c4723d2f83d875469eb5fbfc225abea" exitCode=0 Oct 30 16:27:57.417417 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:57.416976 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5pphd" Oct 30 16:27:57.417417 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:57.417001 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5pphd" event={"ID":"df551f9f-6ffb-4c16-bf9a-c0169466896d","Type":"ContainerDied","Data":"6c9eb4984a12e2fe29ccfd4761211b4c2c4723d2f83d875469eb5fbfc225abea"} Oct 30 16:27:57.417417 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:57.417094 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5pphd" event={"ID":"df551f9f-6ffb-4c16-bf9a-c0169466896d","Type":"ContainerDied","Data":"d689ab476accc7354cf94984bdfb8c3869c7952f62b70eff0e47ce7226ebc108"} Oct 30 16:27:57.417417 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:57.417126 4493 scope.go:122] "RemoveContainer" containerID="6c9eb4984a12e2fe29ccfd4761211b4c2c4723d2f83d875469eb5fbfc225abea" Oct 30 16:27:57.426497 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:57.426479 4493 scope.go:122] "RemoveContainer" containerID="6c9eb4984a12e2fe29ccfd4761211b4c2c4723d2f83d875469eb5fbfc225abea" Oct 30 16:27:57.426778 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:27:57.426750 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c9eb4984a12e2fe29ccfd4761211b4c2c4723d2f83d875469eb5fbfc225abea\": container with ID starting with 6c9eb4984a12e2fe29ccfd4761211b4c2c4723d2f83d875469eb5fbfc225abea not found: ID does not exist" containerID="6c9eb4984a12e2fe29ccfd4761211b4c2c4723d2f83d875469eb5fbfc225abea" Oct 30 16:27:57.426842 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:57.426804 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c9eb4984a12e2fe29ccfd4761211b4c2c4723d2f83d875469eb5fbfc225abea"} err="failed to get container status \"6c9eb4984a12e2fe29ccfd4761211b4c2c4723d2f83d875469eb5fbfc225abea\": rpc error: code = NotFound desc = could not find container \"6c9eb4984a12e2fe29ccfd4761211b4c2c4723d2f83d875469eb5fbfc225abea\": container with ID starting with 6c9eb4984a12e2fe29ccfd4761211b4c2c4723d2f83d875469eb5fbfc225abea not found: ID does not exist" Oct 30 16:27:57.478528 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:57.478504 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5pphd"] Oct 30 16:27:57.489942 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:57.489922 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5pphd"] Oct 30 16:27:58.932082 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:27:58.932048 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df551f9f-6ffb-4c16-bf9a-c0169466896d" path="/var/lib/kubelet/pods/df551f9f-6ffb-4c16-bf9a-c0169466896d/volumes" Oct 30 16:28:00.339826 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:28:00.339770 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 16:28:15.650061 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:15.649972 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 16:28:15.650651 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:15.650628 4493 cpu_manager.go:414] "containerMap: removing container" podUID="df551f9f-6ffb-4c16-bf9a-c0169466896d" containerName="manager" Oct 30 16:28:15.650651 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:15.650649 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df551f9f-6ffb-4c16-bf9a-c0169466896d" containerName="manager" Oct 30 16:28:15.650877 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:15.650861 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="df551f9f-6ffb-4c16-bf9a-c0169466896d" containerName="manager" Oct 30 16:28:15.653215 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:15.653194 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-k9d99" Oct 30 16:28:15.667450 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:15.667433 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 30 16:28:15.678842 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:15.678824 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-bl9vt\"" Oct 30 16:28:15.680674 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:15.680654 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 16:28:15.735806 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:15.735764 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 16:28:15.751137 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:15.751115 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/19632c83-f396-437f-ae2d-93229596ce61-config-file\") pod \"limitador-limitador-7c9b87d574-k9d99\" (UID: \"19632c83-f396-437f-ae2d-93229596ce61\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-k9d99" Oct 30 16:28:15.751230 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:15.751164 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n9tb\" (UniqueName: \"kubernetes.io/projected/19632c83-f396-437f-ae2d-93229596ce61-kube-api-access-8n9tb\") pod \"limitador-limitador-7c9b87d574-k9d99\" (UID: \"19632c83-f396-437f-ae2d-93229596ce61\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-k9d99" Oct 30 16:28:15.852548 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:15.852520 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/19632c83-f396-437f-ae2d-93229596ce61-config-file\") pod \"limitador-limitador-7c9b87d574-k9d99\" (UID: \"19632c83-f396-437f-ae2d-93229596ce61\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-k9d99" Oct 30 16:28:15.852705 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:15.852569 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n9tb\" (UniqueName: \"kubernetes.io/projected/19632c83-f396-437f-ae2d-93229596ce61-kube-api-access-8n9tb\") pod \"limitador-limitador-7c9b87d574-k9d99\" (UID: \"19632c83-f396-437f-ae2d-93229596ce61\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-k9d99" Oct 30 16:28:15.853154 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:15.853135 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/19632c83-f396-437f-ae2d-93229596ce61-config-file\") pod \"limitador-limitador-7c9b87d574-k9d99\" (UID: \"19632c83-f396-437f-ae2d-93229596ce61\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-k9d99" Oct 30 16:28:15.875445 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:15.875419 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n9tb\" (UniqueName: \"kubernetes.io/projected/19632c83-f396-437f-ae2d-93229596ce61-kube-api-access-8n9tb\") pod \"limitador-limitador-7c9b87d574-k9d99\" (UID: \"19632c83-f396-437f-ae2d-93229596ce61\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-k9d99" Oct 30 16:28:15.965483 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:15.965408 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-k9d99" Oct 30 16:28:16.276826 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:16.276775 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 16:28:16.279392 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:28:16.279363 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19632c83_f396_437f_ae2d_93229596ce61.slice/crio-cd77d37e450eaf6aa287b8d31b053781567fca8e3bbd6a5ab485caf0df9ebc52 WatchSource:0}: Error finding container cd77d37e450eaf6aa287b8d31b053781567fca8e3bbd6a5ab485caf0df9ebc52: Status 404 returned error can't find the container with id cd77d37e450eaf6aa287b8d31b053781567fca8e3bbd6a5ab485caf0df9ebc52 Oct 30 16:28:16.281479 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:16.281460 4493 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 16:28:16.511261 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:16.511230 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-k9d99" event={"ID":"19632c83-f396-437f-ae2d-93229596ce61","Type":"ContainerStarted","Data":"cd77d37e450eaf6aa287b8d31b053781567fca8e3bbd6a5ab485caf0df9ebc52"} Oct 30 16:28:16.549391 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:16.549304 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-4szpl"] Oct 30 16:28:16.552624 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:16.552605 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-4szpl" Oct 30 16:28:16.566494 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:16.566475 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-6q2gh\"" Oct 30 16:28:16.572942 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:16.572917 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-4szpl"] Oct 30 16:28:16.659703 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:16.659675 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv68r\" (UniqueName: \"kubernetes.io/projected/d949cf1b-f0b0-468a-b36c-75d26186648c-kube-api-access-gv68r\") pod \"authorino-57f7f8659f-4szpl\" (UID: \"d949cf1b-f0b0-468a-b36c-75d26186648c\") " pod="kuadrant-system/authorino-57f7f8659f-4szpl" Oct 30 16:28:16.760616 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:16.760579 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv68r\" (UniqueName: \"kubernetes.io/projected/d949cf1b-f0b0-468a-b36c-75d26186648c-kube-api-access-gv68r\") pod \"authorino-57f7f8659f-4szpl\" (UID: \"d949cf1b-f0b0-468a-b36c-75d26186648c\") " pod="kuadrant-system/authorino-57f7f8659f-4szpl" Oct 30 16:28:16.783132 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:16.783110 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv68r\" (UniqueName: \"kubernetes.io/projected/d949cf1b-f0b0-468a-b36c-75d26186648c-kube-api-access-gv68r\") pod \"authorino-57f7f8659f-4szpl\" (UID: \"d949cf1b-f0b0-468a-b36c-75d26186648c\") " pod="kuadrant-system/authorino-57f7f8659f-4szpl" Oct 30 16:28:16.875211 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:16.875146 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-4szpl" Oct 30 16:28:17.100191 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:17.100161 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-kl6v7"] Oct 30 16:28:17.103201 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:17.103184 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-kl6v7" Oct 30 16:28:17.124349 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:17.124322 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-kl6v7"] Oct 30 16:28:17.170801 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:17.170765 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-4szpl"] Oct 30 16:28:17.172768 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:28:17.172744 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd949cf1b_f0b0_468a_b36c_75d26186648c.slice/crio-fc59170c6e7b0acd0221de1a99c57575edbfe8e8ad5ff7d94517ddf6f675ab1d WatchSource:0}: Error finding container fc59170c6e7b0acd0221de1a99c57575edbfe8e8ad5ff7d94517ddf6f675ab1d: Status 404 returned error can't find the container with id fc59170c6e7b0acd0221de1a99c57575edbfe8e8ad5ff7d94517ddf6f675ab1d Oct 30 16:28:17.265026 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:17.265004 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqflq\" (UniqueName: \"kubernetes.io/projected/dbeb83cd-3cf8-4ebd-beec-7863f0e1f623-kube-api-access-gqflq\") pod \"authorino-56c6c9ddd5-kl6v7\" (UID: \"dbeb83cd-3cf8-4ebd-beec-7863f0e1f623\") " pod="kuadrant-system/authorino-56c6c9ddd5-kl6v7" Oct 30 16:28:17.366333 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:17.366306 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqflq\" (UniqueName: \"kubernetes.io/projected/dbeb83cd-3cf8-4ebd-beec-7863f0e1f623-kube-api-access-gqflq\") pod \"authorino-56c6c9ddd5-kl6v7\" (UID: \"dbeb83cd-3cf8-4ebd-beec-7863f0e1f623\") " pod="kuadrant-system/authorino-56c6c9ddd5-kl6v7" Oct 30 16:28:17.389331 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:17.389306 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqflq\" (UniqueName: \"kubernetes.io/projected/dbeb83cd-3cf8-4ebd-beec-7863f0e1f623-kube-api-access-gqflq\") pod \"authorino-56c6c9ddd5-kl6v7\" (UID: \"dbeb83cd-3cf8-4ebd-beec-7863f0e1f623\") " pod="kuadrant-system/authorino-56c6c9ddd5-kl6v7" Oct 30 16:28:17.413600 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:17.413576 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-kl6v7" Oct 30 16:28:17.517841 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:17.517811 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-4szpl" event={"ID":"d949cf1b-f0b0-468a-b36c-75d26186648c","Type":"ContainerStarted","Data":"fc59170c6e7b0acd0221de1a99c57575edbfe8e8ad5ff7d94517ddf6f675ab1d"} Oct 30 16:28:17.703949 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:17.703923 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-kl6v7"] Oct 30 16:28:17.706043 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:28:17.706018 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddbeb83cd_3cf8_4ebd_beec_7863f0e1f623.slice/crio-e832c026deeed1d5f5ddb58f968c081dd1d2cf42f8bfa8ed952cd5781a5e9a2b WatchSource:0}: Error finding container e832c026deeed1d5f5ddb58f968c081dd1d2cf42f8bfa8ed952cd5781a5e9a2b: Status 404 returned error can't find the container with id e832c026deeed1d5f5ddb58f968c081dd1d2cf42f8bfa8ed952cd5781a5e9a2b Oct 30 16:28:18.523794 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:18.523749 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-kl6v7" event={"ID":"dbeb83cd-3cf8-4ebd-beec-7863f0e1f623","Type":"ContainerStarted","Data":"e832c026deeed1d5f5ddb58f968c081dd1d2cf42f8bfa8ed952cd5781a5e9a2b"} Oct 30 16:28:19.530627 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:19.530548 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-k9d99" event={"ID":"19632c83-f396-437f-ae2d-93229596ce61","Type":"ContainerStarted","Data":"e7d64a703ab507e6d515f94e1df7c4e6e0c44e517ad6a818dcad4c5affa5c892"} Oct 30 16:28:19.531003 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:19.530722 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-k9d99" Oct 30 16:28:19.573618 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:19.573572 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-k9d99" podStartSLOduration=1.6127638709999998 podStartE2EDuration="4.573556538s" totalImagesPullingTime="2.960792667s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:28:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:28:19.570899952 +0000 UTC m=+326.798075053" watchObservedRunningTime="2026-10-30 16:28:19.573556538 +0000 UTC m=+326.800731646" Oct 30 16:28:21.542805 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:21.542753 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-kl6v7" event={"ID":"dbeb83cd-3cf8-4ebd-beec-7863f0e1f623","Type":"ContainerStarted","Data":"6bb7a540c56953ad5305d9558037562141991510f362a6a386ecdc78017e0f09"} Oct 30 16:28:21.583204 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:21.583154 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-kl6v7" podStartSLOduration=1.780237112 podStartE2EDuration="4.583140026s" totalImagesPullingTime="2.802902914s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:28:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:28:21.581589341 +0000 UTC m=+328.808764441" watchObservedRunningTime="2026-10-30 16:28:21.583140026 +0000 UTC m=+328.810315126" Oct 30 16:28:21.617727 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:21.617689 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-4szpl"] Oct 30 16:28:24.558850 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:24.558818 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-4szpl" event={"ID":"d949cf1b-f0b0-468a-b36c-75d26186648c","Type":"ContainerStarted","Data":"ae8eab8879f0546872ddc4f2224561529d2d2218e07751c03447bdf28ed0c095"} Oct 30 16:28:24.559162 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:24.558878 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-4szpl" podUID="d949cf1b-f0b0-468a-b36c-75d26186648c" containerName="authorino" containerID="cri-o://ae8eab8879f0546872ddc4f2224561529d2d2218e07751c03447bdf28ed0c095" gracePeriod=30 Oct 30 16:28:24.596186 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:24.596143 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-4szpl" podStartSLOduration=1.269974795 podStartE2EDuration="8.596130345s" totalImagesPullingTime="7.32615555s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:28:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:28:24.596048177 +0000 UTC m=+331.823223277" watchObservedRunningTime="2026-10-30 16:28:24.596130345 +0000 UTC m=+331.823305448" Oct 30 16:28:30.340523 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:28:30.340488 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 16:28:30.536329 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:30.536302 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-k9d99" Oct 30 16:28:46.827203 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:46.827167 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-6kvn2"] Oct 30 16:28:46.829575 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:46.829560 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-6kvn2" Oct 30 16:28:46.847839 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:46.847811 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 16:28:46.847946 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:46.847870 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 16:28:46.858697 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:46.858670 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Oct 30 16:28:46.858951 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:46.858933 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-df7pn\"" Oct 30 16:28:46.861326 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:46.861305 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-6kvn2"] Oct 30 16:28:46.871860 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:46.871835 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-p8vfx"] Oct 30 16:28:46.874385 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:46.874364 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-p8vfx" Oct 30 16:28:46.905657 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:46.905626 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-p8vfx"] Oct 30 16:28:46.933029 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:46.933000 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtgw4\" (UniqueName: \"kubernetes.io/projected/e823c774-d3cc-45a9-942a-31a1881ba08f-kube-api-access-vtgw4\") pod \"llmisvc-controller-manager-869d5dcc5d-p8vfx\" (UID: \"e823c774-d3cc-45a9-942a-31a1881ba08f\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-p8vfx" Oct 30 16:28:46.933195 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:46.933094 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e823c774-d3cc-45a9-942a-31a1881ba08f-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-p8vfx\" (UID: \"e823c774-d3cc-45a9-942a-31a1881ba08f\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-p8vfx" Oct 30 16:28:46.933195 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:46.933135 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/58874bb8-6c51-4f03-ab7d-ebf99148b4d8-cert\") pod \"kserve-controller-manager-776cc66d47-6kvn2\" (UID: \"58874bb8-6c51-4f03-ab7d-ebf99148b4d8\") " pod="opendatahub/kserve-controller-manager-776cc66d47-6kvn2" Oct 30 16:28:46.933195 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:46.933178 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngdjb\" (UniqueName: \"kubernetes.io/projected/58874bb8-6c51-4f03-ab7d-ebf99148b4d8-kube-api-access-ngdjb\") pod \"kserve-controller-manager-776cc66d47-6kvn2\" (UID: \"58874bb8-6c51-4f03-ab7d-ebf99148b4d8\") " pod="opendatahub/kserve-controller-manager-776cc66d47-6kvn2" Oct 30 16:28:46.948944 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:46.948923 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Oct 30 16:28:46.948944 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:46.948934 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-xmd6m\"" Oct 30 16:28:46.974171 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:46.974143 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-jcntx"] Oct 30 16:28:46.977087 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:46.977068 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-jcntx" Oct 30 16:28:47.004433 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:47.004409 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-jcntx"] Oct 30 16:28:47.034119 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:47.034094 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e823c774-d3cc-45a9-942a-31a1881ba08f-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-p8vfx\" (UID: \"e823c774-d3cc-45a9-942a-31a1881ba08f\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-p8vfx" Oct 30 16:28:47.034221 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:47.034131 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/58874bb8-6c51-4f03-ab7d-ebf99148b4d8-cert\") pod \"kserve-controller-manager-776cc66d47-6kvn2\" (UID: \"58874bb8-6c51-4f03-ab7d-ebf99148b4d8\") " pod="opendatahub/kserve-controller-manager-776cc66d47-6kvn2" Oct 30 16:28:47.034221 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:47.034172 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngdjb\" (UniqueName: \"kubernetes.io/projected/58874bb8-6c51-4f03-ab7d-ebf99148b4d8-kube-api-access-ngdjb\") pod \"kserve-controller-manager-776cc66d47-6kvn2\" (UID: \"58874bb8-6c51-4f03-ab7d-ebf99148b4d8\") " pod="opendatahub/kserve-controller-manager-776cc66d47-6kvn2" Oct 30 16:28:47.034328 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:47.034217 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtgw4\" (UniqueName: \"kubernetes.io/projected/e823c774-d3cc-45a9-942a-31a1881ba08f-kube-api-access-vtgw4\") pod \"llmisvc-controller-manager-869d5dcc5d-p8vfx\" (UID: \"e823c774-d3cc-45a9-942a-31a1881ba08f\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-p8vfx" Oct 30 16:28:47.034328 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:47.034259 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnrnq\" (UniqueName: \"kubernetes.io/projected/12f71e3d-d209-44f3-b5a8-9eab1f962214-kube-api-access-mnrnq\") pod \"odh-model-controller-769875db56-jcntx\" (UID: \"12f71e3d-d209-44f3-b5a8-9eab1f962214\") " pod="opendatahub/odh-model-controller-769875db56-jcntx" Oct 30 16:28:47.034328 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:47.034295 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12f71e3d-d209-44f3-b5a8-9eab1f962214-metrics-certs\") pod \"odh-model-controller-769875db56-jcntx\" (UID: \"12f71e3d-d209-44f3-b5a8-9eab1f962214\") " pod="opendatahub/odh-model-controller-769875db56-jcntx" Oct 30 16:28:47.034328 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:47.034322 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/12f71e3d-d209-44f3-b5a8-9eab1f962214-cert\") pod \"odh-model-controller-769875db56-jcntx\" (UID: \"12f71e3d-d209-44f3-b5a8-9eab1f962214\") " pod="opendatahub/odh-model-controller-769875db56-jcntx" Oct 30 16:28:47.036749 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:47.036723 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/58874bb8-6c51-4f03-ab7d-ebf99148b4d8-cert\") pod \"kserve-controller-manager-776cc66d47-6kvn2\" (UID: \"58874bb8-6c51-4f03-ab7d-ebf99148b4d8\") " pod="opendatahub/kserve-controller-manager-776cc66d47-6kvn2" Oct 30 16:28:47.037522 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:47.037417 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e823c774-d3cc-45a9-942a-31a1881ba08f-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-p8vfx\" (UID: \"e823c774-d3cc-45a9-942a-31a1881ba08f\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-p8vfx" Oct 30 16:28:47.049076 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:47.049059 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Oct 30 16:28:47.049300 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:47.049281 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Oct 30 16:28:47.049363 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:47.049349 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-slkn4\"" Oct 30 16:28:47.069257 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:47.069236 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtgw4\" (UniqueName: \"kubernetes.io/projected/e823c774-d3cc-45a9-942a-31a1881ba08f-kube-api-access-vtgw4\") pod \"llmisvc-controller-manager-869d5dcc5d-p8vfx\" (UID: \"e823c774-d3cc-45a9-942a-31a1881ba08f\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-p8vfx" Oct 30 16:28:47.069419 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:47.069401 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngdjb\" (UniqueName: \"kubernetes.io/projected/58874bb8-6c51-4f03-ab7d-ebf99148b4d8-kube-api-access-ngdjb\") pod \"kserve-controller-manager-776cc66d47-6kvn2\" (UID: \"58874bb8-6c51-4f03-ab7d-ebf99148b4d8\") " pod="opendatahub/kserve-controller-manager-776cc66d47-6kvn2" Oct 30 16:28:47.135375 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:47.135303 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnrnq\" (UniqueName: \"kubernetes.io/projected/12f71e3d-d209-44f3-b5a8-9eab1f962214-kube-api-access-mnrnq\") pod \"odh-model-controller-769875db56-jcntx\" (UID: \"12f71e3d-d209-44f3-b5a8-9eab1f962214\") " pod="opendatahub/odh-model-controller-769875db56-jcntx" Oct 30 16:28:47.135375 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:47.135341 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12f71e3d-d209-44f3-b5a8-9eab1f962214-metrics-certs\") pod \"odh-model-controller-769875db56-jcntx\" (UID: \"12f71e3d-d209-44f3-b5a8-9eab1f962214\") " pod="opendatahub/odh-model-controller-769875db56-jcntx" Oct 30 16:28:47.135375 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:47.135370 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/12f71e3d-d209-44f3-b5a8-9eab1f962214-cert\") pod \"odh-model-controller-769875db56-jcntx\" (UID: \"12f71e3d-d209-44f3-b5a8-9eab1f962214\") " pod="opendatahub/odh-model-controller-769875db56-jcntx" Oct 30 16:28:47.138107 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:47.138083 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12f71e3d-d209-44f3-b5a8-9eab1f962214-metrics-certs\") pod \"odh-model-controller-769875db56-jcntx\" (UID: \"12f71e3d-d209-44f3-b5a8-9eab1f962214\") " pod="opendatahub/odh-model-controller-769875db56-jcntx" Oct 30 16:28:47.138231 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:47.138188 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/12f71e3d-d209-44f3-b5a8-9eab1f962214-cert\") pod \"odh-model-controller-769875db56-jcntx\" (UID: \"12f71e3d-d209-44f3-b5a8-9eab1f962214\") " pod="opendatahub/odh-model-controller-769875db56-jcntx" Oct 30 16:28:47.141642 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:47.141624 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-6kvn2" Oct 30 16:28:47.158719 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:47.158701 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnrnq\" (UniqueName: \"kubernetes.io/projected/12f71e3d-d209-44f3-b5a8-9eab1f962214-kube-api-access-mnrnq\") pod \"odh-model-controller-769875db56-jcntx\" (UID: \"12f71e3d-d209-44f3-b5a8-9eab1f962214\") " pod="opendatahub/odh-model-controller-769875db56-jcntx" Oct 30 16:28:47.185652 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:47.185624 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-p8vfx" Oct 30 16:28:47.288655 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:47.288623 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-jcntx" Oct 30 16:28:47.444065 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:47.443974 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-6kvn2"] Oct 30 16:28:47.447263 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:28:47.447180 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58874bb8_6c51_4f03_ab7d_ebf99148b4d8.slice/crio-bc6362718a7918d25eb10ee4203638140cf12b7898067ffe38845b607f907d8c WatchSource:0}: Error finding container bc6362718a7918d25eb10ee4203638140cf12b7898067ffe38845b607f907d8c: Status 404 returned error can't find the container with id bc6362718a7918d25eb10ee4203638140cf12b7898067ffe38845b607f907d8c Oct 30 16:28:47.490955 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:47.490931 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-p8vfx"] Oct 30 16:28:47.492349 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:28:47.492325 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode823c774_d3cc_45a9_942a_31a1881ba08f.slice/crio-e7628c7c664920ddee83e9a63b8f62538fc4281da9042bedf58c59a333622bbf WatchSource:0}: Error finding container e7628c7c664920ddee83e9a63b8f62538fc4281da9042bedf58c59a333622bbf: Status 404 returned error can't find the container with id e7628c7c664920ddee83e9a63b8f62538fc4281da9042bedf58c59a333622bbf Oct 30 16:28:47.583078 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:47.583020 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-jcntx"] Oct 30 16:28:47.585122 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:28:47.585099 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12f71e3d_d209_44f3_b5a8_9eab1f962214.slice/crio-c831313e63747caff11e63b224b390692f54a0239525fb1fd8f0cbccfbbc9150 WatchSource:0}: Error finding container c831313e63747caff11e63b224b390692f54a0239525fb1fd8f0cbccfbbc9150: Status 404 returned error can't find the container with id c831313e63747caff11e63b224b390692f54a0239525fb1fd8f0cbccfbbc9150 Oct 30 16:28:47.689596 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:47.689561 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-6kvn2" event={"ID":"58874bb8-6c51-4f03-ab7d-ebf99148b4d8","Type":"ContainerStarted","Data":"bc6362718a7918d25eb10ee4203638140cf12b7898067ffe38845b607f907d8c"} Oct 30 16:28:47.690736 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:47.690710 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-jcntx" event={"ID":"12f71e3d-d209-44f3-b5a8-9eab1f962214","Type":"ContainerStarted","Data":"c831313e63747caff11e63b224b390692f54a0239525fb1fd8f0cbccfbbc9150"} Oct 30 16:28:47.691848 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:47.691827 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-p8vfx" event={"ID":"e823c774-d3cc-45a9-942a-31a1881ba08f","Type":"ContainerStarted","Data":"e7628c7c664920ddee83e9a63b8f62538fc4281da9042bedf58c59a333622bbf"} Oct 30 16:28:51.724153 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:51.724113 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-6kvn2" event={"ID":"58874bb8-6c51-4f03-ab7d-ebf99148b4d8","Type":"ContainerStarted","Data":"e703fbb836cfb38a25784ee912914ce4ba954c8b98bbc4863a17ebafb13ceafb"} Oct 30 16:28:51.724611 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:51.724189 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-6kvn2" Oct 30 16:28:51.726316 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:51.726287 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-jcntx" event={"ID":"12f71e3d-d209-44f3-b5a8-9eab1f962214","Type":"ContainerStarted","Data":"47c367ecfe5e72936946cddfe2743c1097dea861b13cd76c18ef39e944ab227c"} Oct 30 16:28:51.726477 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:51.726427 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-jcntx" Oct 30 16:28:51.728172 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:51.728149 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-p8vfx" event={"ID":"e823c774-d3cc-45a9-942a-31a1881ba08f","Type":"ContainerStarted","Data":"84bd37789b3f48060c065e51045ce8f8b368e9619b5abdbd7e189752c337970a"} Oct 30 16:28:51.728285 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:51.728192 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-p8vfx" Oct 30 16:28:51.766869 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:51.766816 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-6kvn2" podStartSLOduration=1.977412772 podStartE2EDuration="5.766804843s" totalImagesPullingTime="3.789392071s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:28:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:28:51.765334363 +0000 UTC m=+358.992509464" watchObservedRunningTime="2026-10-30 16:28:51.766804843 +0000 UTC m=+358.993979942" Oct 30 16:28:51.837618 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:51.837570 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-p8vfx" podStartSLOduration=2.227002198 podStartE2EDuration="5.83755734s" totalImagesPullingTime="3.610555142s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:28:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:28:51.835353949 +0000 UTC m=+359.062529065" watchObservedRunningTime="2026-10-30 16:28:51.83755734 +0000 UTC m=+359.064732440" Oct 30 16:28:51.887479 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:51.887430 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-jcntx" podStartSLOduration=2.305363324 podStartE2EDuration="5.887417472s" totalImagesPullingTime="3.582054148s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:28:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:28:51.886031452 +0000 UTC m=+359.113206553" watchObservedRunningTime="2026-10-30 16:28:51.887417472 +0000 UTC m=+359.114592572" Oct 30 16:28:54.614736 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:28:54.614701 4493 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd949cf1b_f0b0_468a_b36c_75d26186648c.slice/crio-conmon-ae8eab8879f0546872ddc4f2224561529d2d2218e07751c03447bdf28ed0c095.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd949cf1b_f0b0_468a_b36c_75d26186648c.slice/crio-ae8eab8879f0546872ddc4f2224561529d2d2218e07751c03447bdf28ed0c095.scope\": RecentStats: unable to find data in memory cache]" Oct 30 16:28:54.712662 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:54.712639 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-4szpl" Oct 30 16:28:54.755112 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:54.755080 4493 generic.go:356] "Generic (PLEG): container finished" podID="d949cf1b-f0b0-468a-b36c-75d26186648c" containerID="ae8eab8879f0546872ddc4f2224561529d2d2218e07751c03447bdf28ed0c095" exitCode=137 Oct 30 16:28:54.755271 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:54.755157 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-4szpl" Oct 30 16:28:54.755271 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:54.755171 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-4szpl" event={"ID":"d949cf1b-f0b0-468a-b36c-75d26186648c","Type":"ContainerDied","Data":"ae8eab8879f0546872ddc4f2224561529d2d2218e07751c03447bdf28ed0c095"} Oct 30 16:28:54.755271 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:54.755221 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-4szpl" event={"ID":"d949cf1b-f0b0-468a-b36c-75d26186648c","Type":"ContainerDied","Data":"fc59170c6e7b0acd0221de1a99c57575edbfe8e8ad5ff7d94517ddf6f675ab1d"} Oct 30 16:28:54.755271 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:54.755238 4493 scope.go:122] "RemoveContainer" containerID="ae8eab8879f0546872ddc4f2224561529d2d2218e07751c03447bdf28ed0c095" Oct 30 16:28:54.764620 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:54.764601 4493 scope.go:122] "RemoveContainer" containerID="ae8eab8879f0546872ddc4f2224561529d2d2218e07751c03447bdf28ed0c095" Oct 30 16:28:54.764906 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:28:54.764883 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae8eab8879f0546872ddc4f2224561529d2d2218e07751c03447bdf28ed0c095\": container with ID starting with ae8eab8879f0546872ddc4f2224561529d2d2218e07751c03447bdf28ed0c095 not found: ID does not exist" containerID="ae8eab8879f0546872ddc4f2224561529d2d2218e07751c03447bdf28ed0c095" Oct 30 16:28:54.764996 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:54.764914 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae8eab8879f0546872ddc4f2224561529d2d2218e07751c03447bdf28ed0c095"} err="failed to get container status \"ae8eab8879f0546872ddc4f2224561529d2d2218e07751c03447bdf28ed0c095\": rpc error: code = NotFound desc = could not find container \"ae8eab8879f0546872ddc4f2224561529d2d2218e07751c03447bdf28ed0c095\": container with ID starting with ae8eab8879f0546872ddc4f2224561529d2d2218e07751c03447bdf28ed0c095 not found: ID does not exist" Oct 30 16:28:54.812358 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:54.812295 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d949cf1b-f0b0-468a-b36c-75d26186648c-kube-api-access-gv68r\" (UniqueName: \"kubernetes.io/projected/d949cf1b-f0b0-468a-b36c-75d26186648c-kube-api-access-gv68r\") pod \"d949cf1b-f0b0-468a-b36c-75d26186648c\" (UID: \"d949cf1b-f0b0-468a-b36c-75d26186648c\") " Oct 30 16:28:54.814473 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:54.814448 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d949cf1b-f0b0-468a-b36c-75d26186648c-kube-api-access-gv68r" pod "d949cf1b-f0b0-468a-b36c-75d26186648c" (UID: "d949cf1b-f0b0-468a-b36c-75d26186648c"). InnerVolumeSpecName "kube-api-access-gv68r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:28:54.913836 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:54.913806 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gv68r\" (UniqueName: \"kubernetes.io/projected/d949cf1b-f0b0-468a-b36c-75d26186648c-kube-api-access-gv68r\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:28:55.095654 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:55.095577 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-4szpl"] Oct 30 16:28:55.101607 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:55.101577 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-4szpl"] Oct 30 16:28:56.931985 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:28:56.931949 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d949cf1b-f0b0-468a-b36c-75d26186648c" path="/var/lib/kubelet/pods/d949cf1b-f0b0-468a-b36c-75d26186648c/volumes" Oct 30 16:29:00.342378 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:29:00.342340 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 16:29:02.744437 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:02.744409 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-jcntx" Oct 30 16:29:22.744060 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:22.744029 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-p8vfx" Oct 30 16:29:22.746947 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:22.746925 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-6kvn2" Oct 30 16:29:30.345643 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:29:30.345599 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 16:29:30.406633 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:30.406594 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 16:29:38.350708 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:38.350669 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-rsssx"] Oct 30 16:29:38.351140 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:38.351125 4493 cpu_manager.go:414] "containerMap: removing container" podUID="d949cf1b-f0b0-468a-b36c-75d26186648c" containerName="authorino" Oct 30 16:29:38.351185 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:38.351142 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d949cf1b-f0b0-468a-b36c-75d26186648c" containerName="authorino" Oct 30 16:29:38.351220 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:38.351206 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="d949cf1b-f0b0-468a-b36c-75d26186648c" containerName="authorino" Oct 30 16:29:38.353301 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:38.353286 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-rsssx" Oct 30 16:29:38.375082 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:38.375056 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-rsssx"] Oct 30 16:29:38.394120 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:38.394092 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xvdm\" (UniqueName: \"kubernetes.io/projected/07923f4b-75d8-4101-93c7-bb427db0f144-kube-api-access-7xvdm\") pod \"authorino-6f6944bcdd-rsssx\" (UID: \"07923f4b-75d8-4101-93c7-bb427db0f144\") " pod="kuadrant-system/authorino-6f6944bcdd-rsssx" Oct 30 16:29:38.495451 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:38.495416 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xvdm\" (UniqueName: \"kubernetes.io/projected/07923f4b-75d8-4101-93c7-bb427db0f144-kube-api-access-7xvdm\") pod \"authorino-6f6944bcdd-rsssx\" (UID: \"07923f4b-75d8-4101-93c7-bb427db0f144\") " pod="kuadrant-system/authorino-6f6944bcdd-rsssx" Oct 30 16:29:38.514990 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:38.514954 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-rsssx"] Oct 30 16:29:38.515234 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:29:38.515210 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-7xvdm], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6f6944bcdd-rsssx" podUID="07923f4b-75d8-4101-93c7-bb427db0f144" Oct 30 16:29:38.533041 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:38.533009 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xvdm\" (UniqueName: \"kubernetes.io/projected/07923f4b-75d8-4101-93c7-bb427db0f144-kube-api-access-7xvdm\") pod \"authorino-6f6944bcdd-rsssx\" (UID: \"07923f4b-75d8-4101-93c7-bb427db0f144\") " pod="kuadrant-system/authorino-6f6944bcdd-rsssx" Oct 30 16:29:38.552994 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:38.552967 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-vpjdh"] Oct 30 16:29:38.555498 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:38.555474 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-vpjdh" Oct 30 16:29:38.575212 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:38.575187 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 30 16:29:38.577856 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:38.577823 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-vpjdh"] Oct 30 16:29:38.597264 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:38.597238 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b8fc6ca8-c441-496b-bcb0-f6b28a453951-tls-cert\") pod \"authorino-6889cbd8c4-vpjdh\" (UID: \"b8fc6ca8-c441-496b-bcb0-f6b28a453951\") " pod="kuadrant-system/authorino-6889cbd8c4-vpjdh" Oct 30 16:29:38.597412 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:38.597362 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmzt8\" (UniqueName: \"kubernetes.io/projected/b8fc6ca8-c441-496b-bcb0-f6b28a453951-kube-api-access-hmzt8\") pod \"authorino-6889cbd8c4-vpjdh\" (UID: \"b8fc6ca8-c441-496b-bcb0-f6b28a453951\") " pod="kuadrant-system/authorino-6889cbd8c4-vpjdh" Oct 30 16:29:38.698845 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:38.698778 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmzt8\" (UniqueName: \"kubernetes.io/projected/b8fc6ca8-c441-496b-bcb0-f6b28a453951-kube-api-access-hmzt8\") pod \"authorino-6889cbd8c4-vpjdh\" (UID: \"b8fc6ca8-c441-496b-bcb0-f6b28a453951\") " pod="kuadrant-system/authorino-6889cbd8c4-vpjdh" Oct 30 16:29:38.699005 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:38.698963 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b8fc6ca8-c441-496b-bcb0-f6b28a453951-tls-cert\") pod \"authorino-6889cbd8c4-vpjdh\" (UID: \"b8fc6ca8-c441-496b-bcb0-f6b28a453951\") " pod="kuadrant-system/authorino-6889cbd8c4-vpjdh" Oct 30 16:29:38.701600 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:38.701575 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b8fc6ca8-c441-496b-bcb0-f6b28a453951-tls-cert\") pod \"authorino-6889cbd8c4-vpjdh\" (UID: \"b8fc6ca8-c441-496b-bcb0-f6b28a453951\") " pod="kuadrant-system/authorino-6889cbd8c4-vpjdh" Oct 30 16:29:38.759183 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:38.759148 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmzt8\" (UniqueName: \"kubernetes.io/projected/b8fc6ca8-c441-496b-bcb0-f6b28a453951-kube-api-access-hmzt8\") pod \"authorino-6889cbd8c4-vpjdh\" (UID: \"b8fc6ca8-c441-496b-bcb0-f6b28a453951\") " pod="kuadrant-system/authorino-6889cbd8c4-vpjdh" Oct 30 16:29:38.782388 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:38.782354 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-vpjdh"] Oct 30 16:29:38.782572 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:38.782560 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-vpjdh" Oct 30 16:29:38.821217 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:38.821183 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5666cf5686-hpdq6"] Oct 30 16:29:38.824574 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:38.824549 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5666cf5686-hpdq6" Oct 30 16:29:38.845726 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:38.844318 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5666cf5686-hpdq6"] Oct 30 16:29:38.900996 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:38.900953 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg9dm\" (UniqueName: \"kubernetes.io/projected/1cfc4e62-2a4c-4d7a-ad44-1bed38b7f255-kube-api-access-dg9dm\") pod \"authorino-5666cf5686-hpdq6\" (UID: \"1cfc4e62-2a4c-4d7a-ad44-1bed38b7f255\") " pod="kuadrant-system/authorino-5666cf5686-hpdq6" Oct 30 16:29:38.901177 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:38.901004 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1cfc4e62-2a4c-4d7a-ad44-1bed38b7f255-tls-cert\") pod \"authorino-5666cf5686-hpdq6\" (UID: \"1cfc4e62-2a4c-4d7a-ad44-1bed38b7f255\") " pod="kuadrant-system/authorino-5666cf5686-hpdq6" Oct 30 16:29:38.983713 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:38.983632 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-rsssx" Oct 30 16:29:38.989656 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:38.989629 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-rsssx" Oct 30 16:29:39.001543 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:39.001517 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg9dm\" (UniqueName: \"kubernetes.io/projected/1cfc4e62-2a4c-4d7a-ad44-1bed38b7f255-kube-api-access-dg9dm\") pod \"authorino-5666cf5686-hpdq6\" (UID: \"1cfc4e62-2a4c-4d7a-ad44-1bed38b7f255\") " pod="kuadrant-system/authorino-5666cf5686-hpdq6" Oct 30 16:29:39.001660 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:39.001560 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1cfc4e62-2a4c-4d7a-ad44-1bed38b7f255-tls-cert\") pod \"authorino-5666cf5686-hpdq6\" (UID: \"1cfc4e62-2a4c-4d7a-ad44-1bed38b7f255\") " pod="kuadrant-system/authorino-5666cf5686-hpdq6" Oct 30 16:29:39.004400 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:39.004372 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1cfc4e62-2a4c-4d7a-ad44-1bed38b7f255-tls-cert\") pod \"authorino-5666cf5686-hpdq6\" (UID: \"1cfc4e62-2a4c-4d7a-ad44-1bed38b7f255\") " pod="kuadrant-system/authorino-5666cf5686-hpdq6" Oct 30 16:29:39.025020 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:39.024993 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg9dm\" (UniqueName: \"kubernetes.io/projected/1cfc4e62-2a4c-4d7a-ad44-1bed38b7f255-kube-api-access-dg9dm\") pod \"authorino-5666cf5686-hpdq6\" (UID: \"1cfc4e62-2a4c-4d7a-ad44-1bed38b7f255\") " pod="kuadrant-system/authorino-5666cf5686-hpdq6" Oct 30 16:29:39.083846 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:39.083820 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-vpjdh"] Oct 30 16:29:39.085558 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:29:39.085521 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb8fc6ca8_c441_496b_bcb0_f6b28a453951.slice/crio-2a648d878698fdc86ebe2ccaef642f03be9c9692ec05fed8627a2ba5bae29da7 WatchSource:0}: Error finding container 2a648d878698fdc86ebe2ccaef642f03be9c9692ec05fed8627a2ba5bae29da7: Status 404 returned error can't find the container with id 2a648d878698fdc86ebe2ccaef642f03be9c9692ec05fed8627a2ba5bae29da7 Oct 30 16:29:39.102260 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:39.102237 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07923f4b-75d8-4101-93c7-bb427db0f144-kube-api-access-7xvdm\" (UniqueName: \"kubernetes.io/projected/07923f4b-75d8-4101-93c7-bb427db0f144-kube-api-access-7xvdm\") pod \"07923f4b-75d8-4101-93c7-bb427db0f144\" (UID: \"07923f4b-75d8-4101-93c7-bb427db0f144\") " Oct 30 16:29:39.104526 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:39.104504 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07923f4b-75d8-4101-93c7-bb427db0f144-kube-api-access-7xvdm" pod "07923f4b-75d8-4101-93c7-bb427db0f144" (UID: "07923f4b-75d8-4101-93c7-bb427db0f144"). InnerVolumeSpecName "kube-api-access-7xvdm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:29:39.138062 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:39.138042 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5666cf5686-hpdq6" Oct 30 16:29:39.203567 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:39.203531 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xvdm\" (UniqueName: \"kubernetes.io/projected/07923f4b-75d8-4101-93c7-bb427db0f144-kube-api-access-7xvdm\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:29:39.435335 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:39.435062 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5666cf5686-hpdq6"] Oct 30 16:29:39.437042 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:29:39.437015 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1cfc4e62_2a4c_4d7a_ad44_1bed38b7f255.slice/crio-0288e4a461343ca8539fe1c1e7376d6bc99a50d3bd6f973cef682eabb9baf668 WatchSource:0}: Error finding container 0288e4a461343ca8539fe1c1e7376d6bc99a50d3bd6f973cef682eabb9baf668: Status 404 returned error can't find the container with id 0288e4a461343ca8539fe1c1e7376d6bc99a50d3bd6f973cef682eabb9baf668 Oct 30 16:29:39.994106 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:39.993991 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5666cf5686-hpdq6" event={"ID":"1cfc4e62-2a4c-4d7a-ad44-1bed38b7f255","Type":"ContainerStarted","Data":"d86798dba81d1d35dacf2fee0724e518cbbd950638ebba4f94ceec2be7ba010a"} Oct 30 16:29:39.994106 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:39.994062 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5666cf5686-hpdq6" event={"ID":"1cfc4e62-2a4c-4d7a-ad44-1bed38b7f255","Type":"ContainerStarted","Data":"0288e4a461343ca8539fe1c1e7376d6bc99a50d3bd6f973cef682eabb9baf668"} Oct 30 16:29:39.996836 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:39.996777 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-vpjdh" event={"ID":"b8fc6ca8-c441-496b-bcb0-f6b28a453951","Type":"ContainerStarted","Data":"7c79e63256089c42776f2d90e252c9172b59c7dfa07e880a0b10416940c90831"} Oct 30 16:29:39.996836 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:39.996849 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-vpjdh" event={"ID":"b8fc6ca8-c441-496b-bcb0-f6b28a453951","Type":"ContainerStarted","Data":"2a648d878698fdc86ebe2ccaef642f03be9c9692ec05fed8627a2ba5bae29da7"} Oct 30 16:29:39.997058 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:39.996956 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6889cbd8c4-vpjdh" podUID="b8fc6ca8-c441-496b-bcb0-f6b28a453951" containerName="authorino" containerID="cri-o://7c79e63256089c42776f2d90e252c9172b59c7dfa07e880a0b10416940c90831" gracePeriod=30 Oct 30 16:29:40.003212 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:40.003183 4493 generic.go:356] "Generic (PLEG): container finished" podID="5903f4b2-9604-4eab-adf5-e0f7452ffbba" containerID="0106d84270cd88a079caf167aba8d8c7db014d9244c9dc23c718710171feaf7f" exitCode=0 Oct 30 16:29:40.003340 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:40.003232 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6n8m6" event={"ID":"5903f4b2-9604-4eab-adf5-e0f7452ffbba","Type":"ContainerDied","Data":"0106d84270cd88a079caf167aba8d8c7db014d9244c9dc23c718710171feaf7f"} Oct 30 16:29:40.003340 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:40.003259 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-rsssx" Oct 30 16:29:40.004022 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:40.004002 4493 scope.go:122] "RemoveContainer" containerID="0106d84270cd88a079caf167aba8d8c7db014d9244c9dc23c718710171feaf7f" Oct 30 16:29:40.010885 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:40.009453 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:29:40.010885 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:40.009547 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:29:40.010885 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:40.010371 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:29:40.036090 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:40.036036 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5666cf5686-hpdq6" podStartSLOduration=1.732910941 podStartE2EDuration="2.03602208s" totalImagesPullingTime="303.111139ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:29:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:29:40.034489925 +0000 UTC m=+407.261665024" watchObservedRunningTime="2026-10-30 16:29:40.03602208 +0000 UTC m=+407.263197180" Oct 30 16:29:40.083015 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:40.082403 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-kl6v7"] Oct 30 16:29:40.083015 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:40.082628 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-kl6v7" podUID="dbeb83cd-3cf8-4ebd-beec-7863f0e1f623" containerName="authorino" containerID="cri-o://6bb7a540c56953ad5305d9558037562141991510f362a6a386ecdc78017e0f09" gracePeriod=30 Oct 30 16:29:40.084811 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:40.084452 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6889cbd8c4-vpjdh" podStartSLOduration=1.770195051 podStartE2EDuration="2.084435136s" totalImagesPullingTime="314.240085ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:29:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:29:40.07806425 +0000 UTC m=+407.305239351" watchObservedRunningTime="2026-10-30 16:29:40.084435136 +0000 UTC m=+407.311610233" Oct 30 16:29:40.206131 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:40.206058 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-rsssx"] Oct 30 16:29:40.209349 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:40.209301 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-rsssx"] Oct 30 16:29:40.374167 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:40.373874 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-vpjdh" Oct 30 16:29:40.409365 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:40.409342 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-kl6v7" Oct 30 16:29:40.417037 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:40.416998 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8fc6ca8-c441-496b-bcb0-f6b28a453951-kube-api-access-hmzt8\" (UniqueName: \"kubernetes.io/projected/b8fc6ca8-c441-496b-bcb0-f6b28a453951-kube-api-access-hmzt8\") pod \"b8fc6ca8-c441-496b-bcb0-f6b28a453951\" (UID: \"b8fc6ca8-c441-496b-bcb0-f6b28a453951\") " Oct 30 16:29:40.417180 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:40.417053 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8fc6ca8-c441-496b-bcb0-f6b28a453951-tls-cert\" (UniqueName: \"kubernetes.io/secret/b8fc6ca8-c441-496b-bcb0-f6b28a453951-tls-cert\") pod \"b8fc6ca8-c441-496b-bcb0-f6b28a453951\" (UID: \"b8fc6ca8-c441-496b-bcb0-f6b28a453951\") " Oct 30 16:29:40.420021 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:40.419938 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8fc6ca8-c441-496b-bcb0-f6b28a453951-kube-api-access-hmzt8" pod "b8fc6ca8-c441-496b-bcb0-f6b28a453951" (UID: "b8fc6ca8-c441-496b-bcb0-f6b28a453951"). InnerVolumeSpecName "kube-api-access-hmzt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:29:40.431611 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:40.431551 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8fc6ca8-c441-496b-bcb0-f6b28a453951-tls-cert" pod "b8fc6ca8-c441-496b-bcb0-f6b28a453951" (UID: "b8fc6ca8-c441-496b-bcb0-f6b28a453951"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:29:40.518169 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:40.518129 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbeb83cd-3cf8-4ebd-beec-7863f0e1f623-kube-api-access-gqflq\" (UniqueName: \"kubernetes.io/projected/dbeb83cd-3cf8-4ebd-beec-7863f0e1f623-kube-api-access-gqflq\") pod \"dbeb83cd-3cf8-4ebd-beec-7863f0e1f623\" (UID: \"dbeb83cd-3cf8-4ebd-beec-7863f0e1f623\") " Oct 30 16:29:40.518550 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:40.518474 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmzt8\" (UniqueName: \"kubernetes.io/projected/b8fc6ca8-c441-496b-bcb0-f6b28a453951-kube-api-access-hmzt8\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:29:40.518550 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:40.518496 4493 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b8fc6ca8-c441-496b-bcb0-f6b28a453951-tls-cert\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:29:40.520303 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:40.520277 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbeb83cd-3cf8-4ebd-beec-7863f0e1f623-kube-api-access-gqflq" pod "dbeb83cd-3cf8-4ebd-beec-7863f0e1f623" (UID: "dbeb83cd-3cf8-4ebd-beec-7863f0e1f623"). InnerVolumeSpecName "kube-api-access-gqflq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:29:40.626459 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:40.626423 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqflq\" (UniqueName: \"kubernetes.io/projected/dbeb83cd-3cf8-4ebd-beec-7863f0e1f623-kube-api-access-gqflq\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:29:40.935001 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:40.934957 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07923f4b-75d8-4101-93c7-bb427db0f144" path="/var/lib/kubelet/pods/07923f4b-75d8-4101-93c7-bb427db0f144/volumes" Oct 30 16:29:41.035259 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:41.035219 4493 generic.go:356] "Generic (PLEG): container finished" podID="b8fc6ca8-c441-496b-bcb0-f6b28a453951" containerID="7c79e63256089c42776f2d90e252c9172b59c7dfa07e880a0b10416940c90831" exitCode=0 Oct 30 16:29:41.035463 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:41.035266 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-vpjdh" Oct 30 16:29:41.035463 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:41.035292 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-vpjdh" event={"ID":"b8fc6ca8-c441-496b-bcb0-f6b28a453951","Type":"ContainerDied","Data":"7c79e63256089c42776f2d90e252c9172b59c7dfa07e880a0b10416940c90831"} Oct 30 16:29:41.035463 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:41.035339 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-vpjdh" event={"ID":"b8fc6ca8-c441-496b-bcb0-f6b28a453951","Type":"ContainerDied","Data":"2a648d878698fdc86ebe2ccaef642f03be9c9692ec05fed8627a2ba5bae29da7"} Oct 30 16:29:41.035463 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:41.035358 4493 scope.go:122] "RemoveContainer" containerID="7c79e63256089c42776f2d90e252c9172b59c7dfa07e880a0b10416940c90831" Oct 30 16:29:41.038078 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:41.038031 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6n8m6" event={"ID":"5903f4b2-9604-4eab-adf5-e0f7452ffbba","Type":"ContainerStarted","Data":"c9f797eaa2ae4e5a5a7be88173050e9a96417c80a640f384910aebe19b4ea7f7"} Oct 30 16:29:41.043927 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:41.043906 4493 generic.go:356] "Generic (PLEG): container finished" podID="dbeb83cd-3cf8-4ebd-beec-7863f0e1f623" containerID="6bb7a540c56953ad5305d9558037562141991510f362a6a386ecdc78017e0f09" exitCode=0 Oct 30 16:29:41.044033 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:41.043934 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-kl6v7" Oct 30 16:29:41.044033 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:41.043985 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-kl6v7" event={"ID":"dbeb83cd-3cf8-4ebd-beec-7863f0e1f623","Type":"ContainerDied","Data":"6bb7a540c56953ad5305d9558037562141991510f362a6a386ecdc78017e0f09"} Oct 30 16:29:41.044033 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:41.044004 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-kl6v7" event={"ID":"dbeb83cd-3cf8-4ebd-beec-7863f0e1f623","Type":"ContainerDied","Data":"e832c026deeed1d5f5ddb58f968c081dd1d2cf42f8bfa8ed952cd5781a5e9a2b"} Oct 30 16:29:41.046567 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:41.046551 4493 scope.go:122] "RemoveContainer" containerID="7c79e63256089c42776f2d90e252c9172b59c7dfa07e880a0b10416940c90831" Oct 30 16:29:41.046868 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:29:41.046849 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c79e63256089c42776f2d90e252c9172b59c7dfa07e880a0b10416940c90831\": container with ID starting with 7c79e63256089c42776f2d90e252c9172b59c7dfa07e880a0b10416940c90831 not found: ID does not exist" containerID="7c79e63256089c42776f2d90e252c9172b59c7dfa07e880a0b10416940c90831" Oct 30 16:29:41.046951 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:41.046879 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c79e63256089c42776f2d90e252c9172b59c7dfa07e880a0b10416940c90831"} err="failed to get container status \"7c79e63256089c42776f2d90e252c9172b59c7dfa07e880a0b10416940c90831\": rpc error: code = NotFound desc = could not find container \"7c79e63256089c42776f2d90e252c9172b59c7dfa07e880a0b10416940c90831\": container with ID starting with 7c79e63256089c42776f2d90e252c9172b59c7dfa07e880a0b10416940c90831 not found: ID does not exist" Oct 30 16:29:41.046951 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:41.046904 4493 scope.go:122] "RemoveContainer" containerID="6bb7a540c56953ad5305d9558037562141991510f362a6a386ecdc78017e0f09" Oct 30 16:29:41.056464 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:41.056433 4493 scope.go:122] "RemoveContainer" containerID="6bb7a540c56953ad5305d9558037562141991510f362a6a386ecdc78017e0f09" Oct 30 16:29:41.056707 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:29:41.056691 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bb7a540c56953ad5305d9558037562141991510f362a6a386ecdc78017e0f09\": container with ID starting with 6bb7a540c56953ad5305d9558037562141991510f362a6a386ecdc78017e0f09 not found: ID does not exist" containerID="6bb7a540c56953ad5305d9558037562141991510f362a6a386ecdc78017e0f09" Oct 30 16:29:41.056747 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:41.056715 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bb7a540c56953ad5305d9558037562141991510f362a6a386ecdc78017e0f09"} err="failed to get container status \"6bb7a540c56953ad5305d9558037562141991510f362a6a386ecdc78017e0f09\": rpc error: code = NotFound desc = could not find container \"6bb7a540c56953ad5305d9558037562141991510f362a6a386ecdc78017e0f09\": container with ID starting with 6bb7a540c56953ad5305d9558037562141991510f362a6a386ecdc78017e0f09 not found: ID does not exist" Oct 30 16:29:41.084664 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:41.084640 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-kl6v7"] Oct 30 16:29:41.088425 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:41.088404 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-kl6v7"] Oct 30 16:29:41.131706 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:41.131672 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-vpjdh"] Oct 30 16:29:41.133434 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:41.133407 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-vpjdh"] Oct 30 16:29:41.759124 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:41.759088 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6n8m6" Oct 30 16:29:41.759598 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:41.759309 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-6n8m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.22:15021/healthz/ready\": dial tcp 10.134.0.22:15021: connect: connection refused" start-of-body= Oct 30 16:29:41.759598 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:41.759387 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6n8m6" podUID="5903f4b2-9604-4eab-adf5-e0f7452ffbba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.22:15021/healthz/ready\": dial tcp 10.134.0.22:15021: connect: connection refused" Oct 30 16:29:42.759479 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:42.759447 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-6n8m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.22:15021/healthz/ready\": dial tcp 10.134.0.22:15021: connect: connection refused" start-of-body= Oct 30 16:29:42.760005 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:42.759515 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6n8m6" podUID="5903f4b2-9604-4eab-adf5-e0f7452ffbba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.22:15021/healthz/ready\": dial tcp 10.134.0.22:15021: connect: connection refused" Oct 30 16:29:42.933899 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:42.933869 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8fc6ca8-c441-496b-bcb0-f6b28a453951" path="/var/lib/kubelet/pods/b8fc6ca8-c441-496b-bcb0-f6b28a453951/volumes" Oct 30 16:29:42.934214 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:42.934201 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbeb83cd-3cf8-4ebd-beec-7863f0e1f623" path="/var/lib/kubelet/pods/dbeb83cd-3cf8-4ebd-beec-7863f0e1f623/volumes" Oct 30 16:29:43.759629 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:43.759593 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-6n8m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.22:15021/healthz/ready\": dial tcp 10.134.0.22:15021: connect: connection refused" start-of-body= Oct 30 16:29:43.760067 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:43.759658 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6n8m6" podUID="5903f4b2-9604-4eab-adf5-e0f7452ffbba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.22:15021/healthz/ready\": dial tcp 10.134.0.22:15021: connect: connection refused" Oct 30 16:29:44.758975 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:44.758941 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-6n8m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.22:15021/healthz/ready\": dial tcp 10.134.0.22:15021: connect: connection refused" start-of-body= Oct 30 16:29:44.759151 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:44.759013 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6n8m6" podUID="5903f4b2-9604-4eab-adf5-e0f7452ffbba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.22:15021/healthz/ready\": dial tcp 10.134.0.22:15021: connect: connection refused" Oct 30 16:29:45.758950 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:45.758911 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-6n8m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.22:15021/healthz/ready\": dial tcp 10.134.0.22:15021: connect: connection refused" start-of-body= Oct 30 16:29:45.759374 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:45.758967 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6n8m6" podUID="5903f4b2-9604-4eab-adf5-e0f7452ffbba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.22:15021/healthz/ready\": dial tcp 10.134.0.22:15021: connect: connection refused" Oct 30 16:29:46.762725 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:46.762691 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6n8m6" Oct 30 16:29:46.763137 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:46.763117 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6n8m6" Oct 30 16:29:46.763889 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:29:46.763869 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6n8m6" Oct 30 16:30:00.151814 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:00.151771 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889630-xd78j"] Oct 30 16:30:00.152210 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:00.152198 4493 cpu_manager.go:414] "containerMap: removing container" podUID="b8fc6ca8-c441-496b-bcb0-f6b28a453951" containerName="authorino" Oct 30 16:30:00.152255 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:00.152211 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8fc6ca8-c441-496b-bcb0-f6b28a453951" containerName="authorino" Oct 30 16:30:00.152255 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:00.152227 4493 cpu_manager.go:414] "containerMap: removing container" podUID="dbeb83cd-3cf8-4ebd-beec-7863f0e1f623" containerName="authorino" Oct 30 16:30:00.152255 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:00.152232 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbeb83cd-3cf8-4ebd-beec-7863f0e1f623" containerName="authorino" Oct 30 16:30:00.152967 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:00.152940 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8fc6ca8-c441-496b-bcb0-f6b28a453951" containerName="authorino" Oct 30 16:30:00.153099 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:00.152983 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbeb83cd-3cf8-4ebd-beec-7863f0e1f623" containerName="authorino" Oct 30 16:30:00.157649 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:00.157622 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889630-xd78j" Oct 30 16:30:00.177025 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:00.176996 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889630-xd78j"] Oct 30 16:30:00.185898 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:00.185872 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:30:00.185898 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:00.185884 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:30:00.186121 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:00.186018 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 16:30:00.296020 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:00.295976 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw2h2\" (UniqueName: \"kubernetes.io/projected/29b8ad5a-7c6a-4792-b1fc-fcb7c66145f9-kube-api-access-gw2h2\") pod \"maas-api-key-cleanup-29889630-xd78j\" (UID: \"29b8ad5a-7c6a-4792-b1fc-fcb7c66145f9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889630-xd78j" Oct 30 16:30:00.296206 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:00.296048 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29b8ad5a-7c6a-4792-b1fc-fcb7c66145f9-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889630-xd78j\" (UID: \"29b8ad5a-7c6a-4792-b1fc-fcb7c66145f9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889630-xd78j" Oct 30 16:30:00.340282 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:30:00.340246 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 16:30:00.397120 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:00.397078 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29b8ad5a-7c6a-4792-b1fc-fcb7c66145f9-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889630-xd78j\" (UID: \"29b8ad5a-7c6a-4792-b1fc-fcb7c66145f9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889630-xd78j" Oct 30 16:30:00.397288 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:00.397164 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw2h2\" (UniqueName: \"kubernetes.io/projected/29b8ad5a-7c6a-4792-b1fc-fcb7c66145f9-kube-api-access-gw2h2\") pod \"maas-api-key-cleanup-29889630-xd78j\" (UID: \"29b8ad5a-7c6a-4792-b1fc-fcb7c66145f9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889630-xd78j" Oct 30 16:30:00.397737 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:00.397716 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29b8ad5a-7c6a-4792-b1fc-fcb7c66145f9-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889630-xd78j\" (UID: \"29b8ad5a-7c6a-4792-b1fc-fcb7c66145f9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889630-xd78j" Oct 30 16:30:00.420890 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:00.420831 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw2h2\" (UniqueName: \"kubernetes.io/projected/29b8ad5a-7c6a-4792-b1fc-fcb7c66145f9-kube-api-access-gw2h2\") pod \"maas-api-key-cleanup-29889630-xd78j\" (UID: \"29b8ad5a-7c6a-4792-b1fc-fcb7c66145f9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889630-xd78j" Oct 30 16:30:00.469599 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:00.469571 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889630-xd78j" Oct 30 16:30:00.766364 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:00.766335 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889630-xd78j"] Oct 30 16:30:00.768801 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:30:00.768760 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29b8ad5a_7c6a_4792_b1fc_fcb7c66145f9.slice/crio-cf09c0ede3fbf21ac5d4adf6f57b23ff0f0b2cc2509c3a9f9dd2cad4973cc73b WatchSource:0}: Error finding container cf09c0ede3fbf21ac5d4adf6f57b23ff0f0b2cc2509c3a9f9dd2cad4973cc73b: Status 404 returned error can't find the container with id cf09c0ede3fbf21ac5d4adf6f57b23ff0f0b2cc2509c3a9f9dd2cad4973cc73b Oct 30 16:30:01.140675 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:01.140643 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-7blqd"] Oct 30 16:30:01.143558 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:01.143539 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-7blqd" Oct 30 16:30:01.154597 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:01.154570 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889630-xd78j" event={"ID":"29b8ad5a-7c6a-4792-b1fc-fcb7c66145f9","Type":"ContainerStarted","Data":"cf09c0ede3fbf21ac5d4adf6f57b23ff0f0b2cc2509c3a9f9dd2cad4973cc73b"} Oct 30 16:30:01.166920 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:01.166887 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-7blqd"] Oct 30 16:30:01.204237 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:01.204204 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2003d468-00b8-48c3-9b13-f152737d18ed-cert\") pod \"llmisvc-controller-manager-6466c554d8-7blqd\" (UID: \"2003d468-00b8-48c3-9b13-f152737d18ed\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-7blqd" Oct 30 16:30:01.204414 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:01.204286 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2brcd\" (UniqueName: \"kubernetes.io/projected/2003d468-00b8-48c3-9b13-f152737d18ed-kube-api-access-2brcd\") pod \"llmisvc-controller-manager-6466c554d8-7blqd\" (UID: \"2003d468-00b8-48c3-9b13-f152737d18ed\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-7blqd" Oct 30 16:30:01.305646 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:01.305608 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2brcd\" (UniqueName: \"kubernetes.io/projected/2003d468-00b8-48c3-9b13-f152737d18ed-kube-api-access-2brcd\") pod \"llmisvc-controller-manager-6466c554d8-7blqd\" (UID: \"2003d468-00b8-48c3-9b13-f152737d18ed\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-7blqd" Oct 30 16:30:01.305887 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:01.305671 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2003d468-00b8-48c3-9b13-f152737d18ed-cert\") pod \"llmisvc-controller-manager-6466c554d8-7blqd\" (UID: \"2003d468-00b8-48c3-9b13-f152737d18ed\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-7blqd" Oct 30 16:30:01.308206 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:01.308183 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2003d468-00b8-48c3-9b13-f152737d18ed-cert\") pod \"llmisvc-controller-manager-6466c554d8-7blqd\" (UID: \"2003d468-00b8-48c3-9b13-f152737d18ed\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-7blqd" Oct 30 16:30:01.329076 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:01.329052 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2brcd\" (UniqueName: \"kubernetes.io/projected/2003d468-00b8-48c3-9b13-f152737d18ed-kube-api-access-2brcd\") pod \"llmisvc-controller-manager-6466c554d8-7blqd\" (UID: \"2003d468-00b8-48c3-9b13-f152737d18ed\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-7blqd" Oct 30 16:30:01.455901 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:01.455813 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-7blqd" Oct 30 16:30:01.754938 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:01.754911 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-7blqd"] Oct 30 16:30:01.757459 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:30:01.757429 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2003d468_00b8_48c3_9b13_f152737d18ed.slice/crio-705d217c8a8a98d02226675bfabc42309f84ff9523fab9bb31eb2f5ed0529469 WatchSource:0}: Error finding container 705d217c8a8a98d02226675bfabc42309f84ff9523fab9bb31eb2f5ed0529469: Status 404 returned error can't find the container with id 705d217c8a8a98d02226675bfabc42309f84ff9523fab9bb31eb2f5ed0529469 Oct 30 16:30:02.163351 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:02.163312 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-7blqd" event={"ID":"2003d468-00b8-48c3-9b13-f152737d18ed","Type":"ContainerStarted","Data":"705d217c8a8a98d02226675bfabc42309f84ff9523fab9bb31eb2f5ed0529469"} Oct 30 16:30:03.169765 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:03.169722 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-7blqd" event={"ID":"2003d468-00b8-48c3-9b13-f152737d18ed","Type":"ContainerStarted","Data":"17c0800dcce509678d79bd29f3ce6a829c3cab2492cbf0a3f3d46df4e6a13215"} Oct 30 16:30:03.170190 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:03.169821 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-7blqd" Oct 30 16:30:03.210591 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:03.210539 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-7blqd" podStartSLOduration=1.743907751 podStartE2EDuration="2.210522852s" totalImagesPullingTime="466.615101ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:30:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:30:03.209588303 +0000 UTC m=+430.436763404" watchObservedRunningTime="2026-10-30 16:30:03.210522852 +0000 UTC m=+430.437697952" Oct 30 16:30:04.176353 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:04.176317 4493 generic.go:356] "Generic (PLEG): container finished" podID="29b8ad5a-7c6a-4792-b1fc-fcb7c66145f9" containerID="0ebc724deebb2d3a3dd97cb59895d6236b634ac5801be82bf2d8926551036281" exitCode=60 Oct 30 16:30:04.176804 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:04.176394 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889630-xd78j" event={"ID":"29b8ad5a-7c6a-4792-b1fc-fcb7c66145f9","Type":"ContainerDied","Data":"0ebc724deebb2d3a3dd97cb59895d6236b634ac5801be82bf2d8926551036281"} Oct 30 16:30:04.176804 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:04.176605 4493 scope.go:122] "RemoveContainer" containerID="0ebc724deebb2d3a3dd97cb59895d6236b634ac5801be82bf2d8926551036281" Oct 30 16:30:05.183602 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:05.183566 4493 generic.go:356] "Generic (PLEG): container finished" podID="29b8ad5a-7c6a-4792-b1fc-fcb7c66145f9" containerID="c334fdc4d2d5e4689df61c3d2a68a6c8d77b02ee1ed3cd16b04aaa32836a6ea4" exitCode=60 Oct 30 16:30:05.184027 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:05.183654 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889630-xd78j" event={"ID":"29b8ad5a-7c6a-4792-b1fc-fcb7c66145f9","Type":"ContainerDied","Data":"c334fdc4d2d5e4689df61c3d2a68a6c8d77b02ee1ed3cd16b04aaa32836a6ea4"} Oct 30 16:30:05.184027 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:05.183718 4493 scope.go:122] "RemoveContainer" containerID="0ebc724deebb2d3a3dd97cb59895d6236b634ac5801be82bf2d8926551036281" Oct 30 16:30:05.184027 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:05.183984 4493 scope.go:122] "RemoveContainer" containerID="c334fdc4d2d5e4689df61c3d2a68a6c8d77b02ee1ed3cd16b04aaa32836a6ea4" Oct 30 16:30:05.184232 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:30:05.184215 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889630-xd78j_odh-ai-gateway-infra(29b8ad5a-7c6a-4792-b1fc-fcb7c66145f9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889630-xd78j" podUID="29b8ad5a-7c6a-4792-b1fc-fcb7c66145f9" Oct 30 16:30:06.190376 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:06.190350 4493 scope.go:122] "RemoveContainer" containerID="c334fdc4d2d5e4689df61c3d2a68a6c8d77b02ee1ed3cd16b04aaa32836a6ea4" Oct 30 16:30:06.190780 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:30:06.190558 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889630-xd78j_odh-ai-gateway-infra(29b8ad5a-7c6a-4792-b1fc-fcb7c66145f9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889630-xd78j" podUID="29b8ad5a-7c6a-4792-b1fc-fcb7c66145f9" Oct 30 16:30:17.926392 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:17.926361 4493 scope.go:122] "RemoveContainer" containerID="c334fdc4d2d5e4689df61c3d2a68a6c8d77b02ee1ed3cd16b04aaa32836a6ea4" Oct 30 16:30:18.251961 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:18.251927 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889630-xd78j" event={"ID":"29b8ad5a-7c6a-4792-b1fc-fcb7c66145f9","Type":"ContainerStarted","Data":"d6d006ed34056abf978d97f0d5a3325fdc2a66c8264182ca2913b9ef3b9c7bbe"} Oct 30 16:30:18.293216 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:18.293162 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889630-xd78j" podStartSLOduration=15.963431577 podStartE2EDuration="18.293149575s" totalImagesPullingTime="2.329717998s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:30:18.290708946 +0000 UTC m=+445.517884042" watchObservedRunningTime="2026-10-30 16:30:18.293149575 +0000 UTC m=+445.520324698" Oct 30 16:30:19.258776 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:19.258744 4493 generic.go:356] "Generic (PLEG): container finished" podID="29b8ad5a-7c6a-4792-b1fc-fcb7c66145f9" containerID="d6d006ed34056abf978d97f0d5a3325fdc2a66c8264182ca2913b9ef3b9c7bbe" exitCode=60 Oct 30 16:30:19.259211 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:19.258822 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889630-xd78j" event={"ID":"29b8ad5a-7c6a-4792-b1fc-fcb7c66145f9","Type":"ContainerDied","Data":"d6d006ed34056abf978d97f0d5a3325fdc2a66c8264182ca2913b9ef3b9c7bbe"} Oct 30 16:30:19.259211 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:19.258880 4493 scope.go:122] "RemoveContainer" containerID="c334fdc4d2d5e4689df61c3d2a68a6c8d77b02ee1ed3cd16b04aaa32836a6ea4" Oct 30 16:30:19.259211 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:19.259148 4493 scope.go:122] "RemoveContainer" containerID="d6d006ed34056abf978d97f0d5a3325fdc2a66c8264182ca2913b9ef3b9c7bbe" Oct 30 16:30:19.259379 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:30:19.259362 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889630-xd78j_odh-ai-gateway-infra(29b8ad5a-7c6a-4792-b1fc-fcb7c66145f9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889630-xd78j" podUID="29b8ad5a-7c6a-4792-b1fc-fcb7c66145f9" Oct 30 16:30:19.307801 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:19.307753 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889630-xd78j"] Oct 30 16:30:20.402088 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:20.402064 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889630-xd78j" Oct 30 16:30:20.475247 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:20.475220 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29b8ad5a-7c6a-4792-b1fc-fcb7c66145f9-kube-api-access-gw2h2\" (UniqueName: \"kubernetes.io/projected/29b8ad5a-7c6a-4792-b1fc-fcb7c66145f9-kube-api-access-gw2h2\") pod \"29b8ad5a-7c6a-4792-b1fc-fcb7c66145f9\" (UID: \"29b8ad5a-7c6a-4792-b1fc-fcb7c66145f9\") " Oct 30 16:30:20.475420 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:20.475274 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29b8ad5a-7c6a-4792-b1fc-fcb7c66145f9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29b8ad5a-7c6a-4792-b1fc-fcb7c66145f9-maas-api-service-ca\") pod \"29b8ad5a-7c6a-4792-b1fc-fcb7c66145f9\" (UID: \"29b8ad5a-7c6a-4792-b1fc-fcb7c66145f9\") " Oct 30 16:30:20.475636 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:20.475612 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29b8ad5a-7c6a-4792-b1fc-fcb7c66145f9-maas-api-service-ca" pod "29b8ad5a-7c6a-4792-b1fc-fcb7c66145f9" (UID: "29b8ad5a-7c6a-4792-b1fc-fcb7c66145f9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:30:20.477486 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:20.477465 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29b8ad5a-7c6a-4792-b1fc-fcb7c66145f9-kube-api-access-gw2h2" pod "29b8ad5a-7c6a-4792-b1fc-fcb7c66145f9" (UID: "29b8ad5a-7c6a-4792-b1fc-fcb7c66145f9"). InnerVolumeSpecName "kube-api-access-gw2h2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:30:20.575894 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:20.575813 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29b8ad5a-7c6a-4792-b1fc-fcb7c66145f9-maas-api-service-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:30:20.575894 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:20.575842 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gw2h2\" (UniqueName: \"kubernetes.io/projected/29b8ad5a-7c6a-4792-b1fc-fcb7c66145f9-kube-api-access-gw2h2\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:30:21.273014 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:21.272938 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889630-xd78j" Oct 30 16:30:21.273014 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:21.272964 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889630-xd78j" event={"ID":"29b8ad5a-7c6a-4792-b1fc-fcb7c66145f9","Type":"ContainerDied","Data":"cf09c0ede3fbf21ac5d4adf6f57b23ff0f0b2cc2509c3a9f9dd2cad4973cc73b"} Oct 30 16:30:21.273237 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:21.273020 4493 scope.go:122] "RemoveContainer" containerID="d6d006ed34056abf978d97f0d5a3325fdc2a66c8264182ca2913b9ef3b9c7bbe" Oct 30 16:30:21.323184 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:21.323158 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889630-xd78j"] Oct 30 16:30:21.334228 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:21.334205 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889630-xd78j"] Oct 30 16:30:22.933740 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:22.933707 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29b8ad5a-7c6a-4792-b1fc-fcb7c66145f9" path="/var/lib/kubelet/pods/29b8ad5a-7c6a-4792-b1fc-fcb7c66145f9/volumes" Oct 30 16:30:30.340432 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:30:30.340386 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 16:30:34.179200 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:34.179171 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-7blqd" Oct 30 16:30:34.251171 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:34.251140 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-p8vfx"] Oct 30 16:30:34.251387 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:34.251359 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-p8vfx" podUID="e823c774-d3cc-45a9-942a-31a1881ba08f" containerName="manager" containerID="cri-o://84bd37789b3f48060c065e51045ce8f8b368e9619b5abdbd7e189752c337970a" gracePeriod=30 Oct 30 16:30:34.497437 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:34.497415 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-p8vfx" Oct 30 16:30:34.586049 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:34.586013 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e823c774-d3cc-45a9-942a-31a1881ba08f-kube-api-access-vtgw4\" (UniqueName: \"kubernetes.io/projected/e823c774-d3cc-45a9-942a-31a1881ba08f-kube-api-access-vtgw4\") pod \"e823c774-d3cc-45a9-942a-31a1881ba08f\" (UID: \"e823c774-d3cc-45a9-942a-31a1881ba08f\") " Oct 30 16:30:34.586232 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:34.586075 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e823c774-d3cc-45a9-942a-31a1881ba08f-cert\" (UniqueName: \"kubernetes.io/secret/e823c774-d3cc-45a9-942a-31a1881ba08f-cert\") pod \"e823c774-d3cc-45a9-942a-31a1881ba08f\" (UID: \"e823c774-d3cc-45a9-942a-31a1881ba08f\") " Oct 30 16:30:34.588292 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:34.588259 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e823c774-d3cc-45a9-942a-31a1881ba08f-kube-api-access-vtgw4" pod "e823c774-d3cc-45a9-942a-31a1881ba08f" (UID: "e823c774-d3cc-45a9-942a-31a1881ba08f"). InnerVolumeSpecName "kube-api-access-vtgw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:30:34.588414 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:34.588335 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e823c774-d3cc-45a9-942a-31a1881ba08f-cert" pod "e823c774-d3cc-45a9-942a-31a1881ba08f" (UID: "e823c774-d3cc-45a9-942a-31a1881ba08f"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:30:34.686705 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:34.686664 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtgw4\" (UniqueName: \"kubernetes.io/projected/e823c774-d3cc-45a9-942a-31a1881ba08f-kube-api-access-vtgw4\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:30:34.686705 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:34.686697 4493 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e823c774-d3cc-45a9-942a-31a1881ba08f-cert\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:30:34.973066 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:30:34.973036 4493 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-pode823c774_d3cc_45a9_942a_31a1881ba08f.slice/crio-e7628c7c664920ddee83e9a63b8f62538fc4281da9042bedf58c59a333622bbf\": RecentStats: unable to find data in memory cache]" Oct 30 16:30:35.345968 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:35.345882 4493 generic.go:356] "Generic (PLEG): container finished" podID="e823c774-d3cc-45a9-942a-31a1881ba08f" containerID="84bd37789b3f48060c065e51045ce8f8b368e9619b5abdbd7e189752c337970a" exitCode=0 Oct 30 16:30:35.345968 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:35.345919 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-p8vfx" event={"ID":"e823c774-d3cc-45a9-942a-31a1881ba08f","Type":"ContainerDied","Data":"84bd37789b3f48060c065e51045ce8f8b368e9619b5abdbd7e189752c337970a"} Oct 30 16:30:35.346475 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:35.345972 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-p8vfx" event={"ID":"e823c774-d3cc-45a9-942a-31a1881ba08f","Type":"ContainerDied","Data":"e7628c7c664920ddee83e9a63b8f62538fc4281da9042bedf58c59a333622bbf"} Oct 30 16:30:35.346475 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:35.345930 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-p8vfx" Oct 30 16:30:35.346475 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:35.345994 4493 scope.go:122] "RemoveContainer" containerID="84bd37789b3f48060c065e51045ce8f8b368e9619b5abdbd7e189752c337970a" Oct 30 16:30:35.355272 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:35.355254 4493 scope.go:122] "RemoveContainer" containerID="84bd37789b3f48060c065e51045ce8f8b368e9619b5abdbd7e189752c337970a" Oct 30 16:30:35.355544 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:30:35.355525 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84bd37789b3f48060c065e51045ce8f8b368e9619b5abdbd7e189752c337970a\": container with ID starting with 84bd37789b3f48060c065e51045ce8f8b368e9619b5abdbd7e189752c337970a not found: ID does not exist" containerID="84bd37789b3f48060c065e51045ce8f8b368e9619b5abdbd7e189752c337970a" Oct 30 16:30:35.355598 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:35.355554 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84bd37789b3f48060c065e51045ce8f8b368e9619b5abdbd7e189752c337970a"} err="failed to get container status \"84bd37789b3f48060c065e51045ce8f8b368e9619b5abdbd7e189752c337970a\": rpc error: code = NotFound desc = could not find container \"84bd37789b3f48060c065e51045ce8f8b368e9619b5abdbd7e189752c337970a\": container with ID starting with 84bd37789b3f48060c065e51045ce8f8b368e9619b5abdbd7e189752c337970a not found: ID does not exist" Oct 30 16:30:35.395886 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:35.395858 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-p8vfx"] Oct 30 16:30:35.407578 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:35.407550 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-p8vfx"] Oct 30 16:30:36.931904 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:30:36.931870 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e823c774-d3cc-45a9-942a-31a1881ba08f" path="/var/lib/kubelet/pods/e823c774-d3cc-45a9-942a-31a1881ba08f/volumes" Oct 30 16:31:00.339777 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:31:00.339729 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 16:31:10.355857 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:31:10.355327 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 16:31:14.644496 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:31:14.644449 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 16:31:22.389806 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:31:22.389066 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 16:31:29.075619 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:31:29.075575 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 16:31:30.356130 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:31:30.356086 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 16:31:44.742314 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:31:44.742283 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-78bcf8868f-w5c9k"] Oct 30 16:31:44.742842 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:31:44.742698 4493 cpu_manager.go:414] "containerMap: removing container" podUID="29b8ad5a-7c6a-4792-b1fc-fcb7c66145f9" containerName="cleanup" Oct 30 16:31:44.742842 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:31:44.742712 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29b8ad5a-7c6a-4792-b1fc-fcb7c66145f9" containerName="cleanup" Oct 30 16:31:44.742842 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:31:44.742723 4493 cpu_manager.go:414] "containerMap: removing container" podUID="29b8ad5a-7c6a-4792-b1fc-fcb7c66145f9" containerName="cleanup" Oct 30 16:31:44.742842 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:31:44.742728 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29b8ad5a-7c6a-4792-b1fc-fcb7c66145f9" containerName="cleanup" Oct 30 16:31:44.742842 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:31:44.742737 4493 cpu_manager.go:414] "containerMap: removing container" podUID="e823c774-d3cc-45a9-942a-31a1881ba08f" containerName="manager" Oct 30 16:31:44.742842 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:31:44.742741 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e823c774-d3cc-45a9-942a-31a1881ba08f" containerName="manager" Oct 30 16:31:44.742842 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:31:44.742753 4493 cpu_manager.go:414] "containerMap: removing container" podUID="29b8ad5a-7c6a-4792-b1fc-fcb7c66145f9" containerName="cleanup" Oct 30 16:31:44.742842 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:31:44.742757 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29b8ad5a-7c6a-4792-b1fc-fcb7c66145f9" containerName="cleanup" Oct 30 16:31:44.743143 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:31:44.742855 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="29b8ad5a-7c6a-4792-b1fc-fcb7c66145f9" containerName="cleanup" Oct 30 16:31:44.743143 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:31:44.742862 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="29b8ad5a-7c6a-4792-b1fc-fcb7c66145f9" containerName="cleanup" Oct 30 16:31:44.743143 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:31:44.742868 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="e823c774-d3cc-45a9-942a-31a1881ba08f" containerName="manager" Oct 30 16:31:44.745823 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:31:44.745805 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-78bcf8868f-w5c9k" Oct 30 16:31:44.767552 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:31:44.767527 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-78bcf8868f-w5c9k"] Oct 30 16:31:44.840794 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:31:44.840742 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8xg2\" (UniqueName: \"kubernetes.io/projected/ad767d7d-7fb3-480d-bf9b-6583a00936fb-kube-api-access-f8xg2\") pod \"authorino-78bcf8868f-w5c9k\" (UID: \"ad767d7d-7fb3-480d-bf9b-6583a00936fb\") " pod="kuadrant-system/authorino-78bcf8868f-w5c9k" Oct 30 16:31:44.840970 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:31:44.840814 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ad767d7d-7fb3-480d-bf9b-6583a00936fb-tls-cert\") pod \"authorino-78bcf8868f-w5c9k\" (UID: \"ad767d7d-7fb3-480d-bf9b-6583a00936fb\") " pod="kuadrant-system/authorino-78bcf8868f-w5c9k" Oct 30 16:31:44.941624 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:31:44.941595 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8xg2\" (UniqueName: \"kubernetes.io/projected/ad767d7d-7fb3-480d-bf9b-6583a00936fb-kube-api-access-f8xg2\") pod \"authorino-78bcf8868f-w5c9k\" (UID: \"ad767d7d-7fb3-480d-bf9b-6583a00936fb\") " pod="kuadrant-system/authorino-78bcf8868f-w5c9k" Oct 30 16:31:44.941624 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:31:44.941630 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ad767d7d-7fb3-480d-bf9b-6583a00936fb-tls-cert\") pod \"authorino-78bcf8868f-w5c9k\" (UID: \"ad767d7d-7fb3-480d-bf9b-6583a00936fb\") " pod="kuadrant-system/authorino-78bcf8868f-w5c9k" Oct 30 16:31:44.944184 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:31:44.944161 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ad767d7d-7fb3-480d-bf9b-6583a00936fb-tls-cert\") pod \"authorino-78bcf8868f-w5c9k\" (UID: \"ad767d7d-7fb3-480d-bf9b-6583a00936fb\") " pod="kuadrant-system/authorino-78bcf8868f-w5c9k" Oct 30 16:31:44.965393 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:31:44.965368 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8xg2\" (UniqueName: \"kubernetes.io/projected/ad767d7d-7fb3-480d-bf9b-6583a00936fb-kube-api-access-f8xg2\") pod \"authorino-78bcf8868f-w5c9k\" (UID: \"ad767d7d-7fb3-480d-bf9b-6583a00936fb\") " pod="kuadrant-system/authorino-78bcf8868f-w5c9k" Oct 30 16:31:45.057319 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:31:45.057195 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-78bcf8868f-w5c9k" Oct 30 16:31:45.357111 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:31:45.357040 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-78bcf8868f-w5c9k"] Oct 30 16:31:45.360393 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:31:45.360365 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podad767d7d_7fb3_480d_bf9b_6583a00936fb.slice/crio-a21ebfb3f44ab582a4be7e8b7527b0cca7fa5eaf52fcaa7a1eb0a70b735d32e9 WatchSource:0}: Error finding container a21ebfb3f44ab582a4be7e8b7527b0cca7fa5eaf52fcaa7a1eb0a70b735d32e9: Status 404 returned error can't find the container with id a21ebfb3f44ab582a4be7e8b7527b0cca7fa5eaf52fcaa7a1eb0a70b735d32e9 Oct 30 16:31:45.732156 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:31:45.732115 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-78bcf8868f-w5c9k" event={"ID":"ad767d7d-7fb3-480d-bf9b-6583a00936fb","Type":"ContainerStarted","Data":"a21ebfb3f44ab582a4be7e8b7527b0cca7fa5eaf52fcaa7a1eb0a70b735d32e9"} Oct 30 16:31:46.740886 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:31:46.740839 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-78bcf8868f-w5c9k" event={"ID":"ad767d7d-7fb3-480d-bf9b-6583a00936fb","Type":"ContainerStarted","Data":"1e3738cef43310a963c8a90eb047c7c7cd540f5b6ab77a335ef6c9e2d9acd368"} Oct 30 16:31:46.806394 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:31:46.806319 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-78bcf8868f-w5c9k" podStartSLOduration=2.107196084 podStartE2EDuration="2.806295154s" totalImagesPullingTime="699.09907ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:31:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:31:46.780683577 +0000 UTC m=+534.007858679" watchObservedRunningTime="2026-10-30 16:31:46.806295154 +0000 UTC m=+534.033470255" Oct 30 16:31:46.812941 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:31:46.812906 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5666cf5686-hpdq6"] Oct 30 16:31:46.813167 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:31:46.813118 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-5666cf5686-hpdq6" podUID="1cfc4e62-2a4c-4d7a-ad44-1bed38b7f255" containerName="authorino" containerID="cri-o://d86798dba81d1d35dacf2fee0724e518cbbd950638ebba4f94ceec2be7ba010a" gracePeriod=30 Oct 30 16:31:47.086407 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:31:47.086376 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5666cf5686-hpdq6" Oct 30 16:31:47.159777 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:31:47.159744 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1cfc4e62-2a4c-4d7a-ad44-1bed38b7f255-tls-cert\" (UniqueName: \"kubernetes.io/secret/1cfc4e62-2a4c-4d7a-ad44-1bed38b7f255-tls-cert\") pod \"1cfc4e62-2a4c-4d7a-ad44-1bed38b7f255\" (UID: \"1cfc4e62-2a4c-4d7a-ad44-1bed38b7f255\") " Oct 30 16:31:47.160008 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:31:47.159892 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cfc4e62-2a4c-4d7a-ad44-1bed38b7f255-kube-api-access-dg9dm\" (UniqueName: \"kubernetes.io/projected/1cfc4e62-2a4c-4d7a-ad44-1bed38b7f255-kube-api-access-dg9dm\") pod \"1cfc4e62-2a4c-4d7a-ad44-1bed38b7f255\" (UID: \"1cfc4e62-2a4c-4d7a-ad44-1bed38b7f255\") " Oct 30 16:31:47.162190 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:31:47.162154 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cfc4e62-2a4c-4d7a-ad44-1bed38b7f255-kube-api-access-dg9dm" pod "1cfc4e62-2a4c-4d7a-ad44-1bed38b7f255" (UID: "1cfc4e62-2a4c-4d7a-ad44-1bed38b7f255"). InnerVolumeSpecName "kube-api-access-dg9dm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:31:47.175736 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:31:47.175687 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cfc4e62-2a4c-4d7a-ad44-1bed38b7f255-tls-cert" pod "1cfc4e62-2a4c-4d7a-ad44-1bed38b7f255" (UID: "1cfc4e62-2a4c-4d7a-ad44-1bed38b7f255"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:31:47.260997 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:31:47.260957 4493 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1cfc4e62-2a4c-4d7a-ad44-1bed38b7f255-tls-cert\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:31:47.260997 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:31:47.260988 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dg9dm\" (UniqueName: \"kubernetes.io/projected/1cfc4e62-2a4c-4d7a-ad44-1bed38b7f255-kube-api-access-dg9dm\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:31:47.747911 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:31:47.747875 4493 generic.go:356] "Generic (PLEG): container finished" podID="1cfc4e62-2a4c-4d7a-ad44-1bed38b7f255" containerID="d86798dba81d1d35dacf2fee0724e518cbbd950638ebba4f94ceec2be7ba010a" exitCode=0 Oct 30 16:31:47.748413 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:31:47.747915 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5666cf5686-hpdq6" Oct 30 16:31:47.748413 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:31:47.747943 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5666cf5686-hpdq6" event={"ID":"1cfc4e62-2a4c-4d7a-ad44-1bed38b7f255","Type":"ContainerDied","Data":"d86798dba81d1d35dacf2fee0724e518cbbd950638ebba4f94ceec2be7ba010a"} Oct 30 16:31:47.748413 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:31:47.747992 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5666cf5686-hpdq6" event={"ID":"1cfc4e62-2a4c-4d7a-ad44-1bed38b7f255","Type":"ContainerDied","Data":"0288e4a461343ca8539fe1c1e7376d6bc99a50d3bd6f973cef682eabb9baf668"} Oct 30 16:31:47.748413 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:31:47.748013 4493 scope.go:122] "RemoveContainer" containerID="d86798dba81d1d35dacf2fee0724e518cbbd950638ebba4f94ceec2be7ba010a" Oct 30 16:31:47.759464 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:31:47.759430 4493 scope.go:122] "RemoveContainer" containerID="d86798dba81d1d35dacf2fee0724e518cbbd950638ebba4f94ceec2be7ba010a" Oct 30 16:31:47.759886 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:31:47.759858 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d86798dba81d1d35dacf2fee0724e518cbbd950638ebba4f94ceec2be7ba010a\": container with ID starting with d86798dba81d1d35dacf2fee0724e518cbbd950638ebba4f94ceec2be7ba010a not found: ID does not exist" containerID="d86798dba81d1d35dacf2fee0724e518cbbd950638ebba4f94ceec2be7ba010a" Oct 30 16:31:47.759970 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:31:47.759899 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d86798dba81d1d35dacf2fee0724e518cbbd950638ebba4f94ceec2be7ba010a"} err="failed to get container status \"d86798dba81d1d35dacf2fee0724e518cbbd950638ebba4f94ceec2be7ba010a\": rpc error: code = NotFound desc = could not find container \"d86798dba81d1d35dacf2fee0724e518cbbd950638ebba4f94ceec2be7ba010a\": container with ID starting with d86798dba81d1d35dacf2fee0724e518cbbd950638ebba4f94ceec2be7ba010a not found: ID does not exist" Oct 30 16:31:47.802766 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:31:47.802737 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5666cf5686-hpdq6"] Oct 30 16:31:47.813731 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:31:47.813694 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5666cf5686-hpdq6"] Oct 30 16:31:48.932658 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:31:48.932626 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cfc4e62-2a4c-4d7a-ad44-1bed38b7f255" path="/var/lib/kubelet/pods/1cfc4e62-2a4c-4d7a-ad44-1bed38b7f255/volumes" Oct 30 16:32:00.341082 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:32:00.341049 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 16:32:22.387206 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:22.387173 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 16:32:24.320558 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:24.320525 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp"] Oct 30 16:32:24.321170 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:24.321150 4493 cpu_manager.go:414] "containerMap: removing container" podUID="1cfc4e62-2a4c-4d7a-ad44-1bed38b7f255" containerName="authorino" Oct 30 16:32:24.321253 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:24.321172 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cfc4e62-2a4c-4d7a-ad44-1bed38b7f255" containerName="authorino" Oct 30 16:32:24.321307 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:24.321266 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cfc4e62-2a4c-4d7a-ad44-1bed38b7f255" containerName="authorino" Oct 30 16:32:24.321307 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:24.321282 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="29b8ad5a-7c6a-4792-b1fc-fcb7c66145f9" containerName="cleanup" Oct 30 16:32:24.324948 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:24.324926 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp" Oct 30 16:32:24.346283 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:24.346256 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-a3cc1e2b-openshift-default-dockercfg-4dlh2\"" Oct 30 16:32:24.361428 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:24.361394 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp"] Oct 30 16:32:24.405592 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:24.405557 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffdf199f-692a-4542-a7cb-0f836829419e-istio-envoy\") pod \"e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp\" (UID: \"ffdf199f-692a-4542-a7cb-0f836829419e\") " pod="openshift-ingress/e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp" Oct 30 16:32:24.405775 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:24.405604 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffdf199f-692a-4542-a7cb-0f836829419e-workload-certs\") pod \"e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp\" (UID: \"ffdf199f-692a-4542-a7cb-0f836829419e\") " pod="openshift-ingress/e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp" Oct 30 16:32:24.405775 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:24.405641 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffdf199f-692a-4542-a7cb-0f836829419e-istiod-ca-cert\") pod \"e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp\" (UID: \"ffdf199f-692a-4542-a7cb-0f836829419e\") " pod="openshift-ingress/e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp" Oct 30 16:32:24.405775 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:24.405679 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vdwc\" (UniqueName: \"kubernetes.io/projected/ffdf199f-692a-4542-a7cb-0f836829419e-kube-api-access-8vdwc\") pod \"e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp\" (UID: \"ffdf199f-692a-4542-a7cb-0f836829419e\") " pod="openshift-ingress/e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp" Oct 30 16:32:24.405775 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:24.405768 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffdf199f-692a-4542-a7cb-0f836829419e-istio-podinfo\") pod \"e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp\" (UID: \"ffdf199f-692a-4542-a7cb-0f836829419e\") " pod="openshift-ingress/e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp" Oct 30 16:32:24.406056 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:24.405826 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffdf199f-692a-4542-a7cb-0f836829419e-istio-token\") pod \"e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp\" (UID: \"ffdf199f-692a-4542-a7cb-0f836829419e\") " pod="openshift-ingress/e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp" Oct 30 16:32:24.406056 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:24.405863 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffdf199f-692a-4542-a7cb-0f836829419e-istio-data\") pod \"e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp\" (UID: \"ffdf199f-692a-4542-a7cb-0f836829419e\") " pod="openshift-ingress/e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp" Oct 30 16:32:24.406056 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:24.405897 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffdf199f-692a-4542-a7cb-0f836829419e-workload-socket\") pod \"e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp\" (UID: \"ffdf199f-692a-4542-a7cb-0f836829419e\") " pod="openshift-ingress/e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp" Oct 30 16:32:24.406056 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:24.405928 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffdf199f-692a-4542-a7cb-0f836829419e-credential-socket\") pod \"e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp\" (UID: \"ffdf199f-692a-4542-a7cb-0f836829419e\") " pod="openshift-ingress/e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp" Oct 30 16:32:24.507389 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:24.507357 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffdf199f-692a-4542-a7cb-0f836829419e-istio-podinfo\") pod \"e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp\" (UID: \"ffdf199f-692a-4542-a7cb-0f836829419e\") " pod="openshift-ingress/e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp" Oct 30 16:32:24.507389 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:24.507398 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffdf199f-692a-4542-a7cb-0f836829419e-istio-token\") pod \"e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp\" (UID: \"ffdf199f-692a-4542-a7cb-0f836829419e\") " pod="openshift-ingress/e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp" Oct 30 16:32:24.507639 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:24.507435 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffdf199f-692a-4542-a7cb-0f836829419e-istio-data\") pod \"e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp\" (UID: \"ffdf199f-692a-4542-a7cb-0f836829419e\") " pod="openshift-ingress/e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp" Oct 30 16:32:24.507639 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:24.507463 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffdf199f-692a-4542-a7cb-0f836829419e-workload-socket\") pod \"e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp\" (UID: \"ffdf199f-692a-4542-a7cb-0f836829419e\") " pod="openshift-ingress/e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp" Oct 30 16:32:24.507639 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:24.507490 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffdf199f-692a-4542-a7cb-0f836829419e-credential-socket\") pod \"e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp\" (UID: \"ffdf199f-692a-4542-a7cb-0f836829419e\") " pod="openshift-ingress/e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp" Oct 30 16:32:24.507639 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:24.507519 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffdf199f-692a-4542-a7cb-0f836829419e-istio-envoy\") pod \"e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp\" (UID: \"ffdf199f-692a-4542-a7cb-0f836829419e\") " pod="openshift-ingress/e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp" Oct 30 16:32:24.507639 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:24.507543 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffdf199f-692a-4542-a7cb-0f836829419e-workload-certs\") pod \"e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp\" (UID: \"ffdf199f-692a-4542-a7cb-0f836829419e\") " pod="openshift-ingress/e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp" Oct 30 16:32:24.507639 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:24.507577 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffdf199f-692a-4542-a7cb-0f836829419e-istiod-ca-cert\") pod \"e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp\" (UID: \"ffdf199f-692a-4542-a7cb-0f836829419e\") " pod="openshift-ingress/e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp" Oct 30 16:32:24.507639 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:24.507606 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vdwc\" (UniqueName: \"kubernetes.io/projected/ffdf199f-692a-4542-a7cb-0f836829419e-kube-api-access-8vdwc\") pod \"e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp\" (UID: \"ffdf199f-692a-4542-a7cb-0f836829419e\") " pod="openshift-ingress/e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp" Oct 30 16:32:24.508014 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:24.507947 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffdf199f-692a-4542-a7cb-0f836829419e-workload-socket\") pod \"e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp\" (UID: \"ffdf199f-692a-4542-a7cb-0f836829419e\") " pod="openshift-ingress/e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp" Oct 30 16:32:24.508266 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:24.508243 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffdf199f-692a-4542-a7cb-0f836829419e-credential-socket\") pod \"e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp\" (UID: \"ffdf199f-692a-4542-a7cb-0f836829419e\") " pod="openshift-ingress/e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp" Oct 30 16:32:24.508375 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:24.508249 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffdf199f-692a-4542-a7cb-0f836829419e-istio-data\") pod \"e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp\" (UID: \"ffdf199f-692a-4542-a7cb-0f836829419e\") " pod="openshift-ingress/e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp" Oct 30 16:32:24.508471 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:24.508441 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffdf199f-692a-4542-a7cb-0f836829419e-workload-certs\") pod \"e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp\" (UID: \"ffdf199f-692a-4542-a7cb-0f836829419e\") " pod="openshift-ingress/e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp" Oct 30 16:32:24.508909 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:24.508884 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffdf199f-692a-4542-a7cb-0f836829419e-istiod-ca-cert\") pod \"e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp\" (UID: \"ffdf199f-692a-4542-a7cb-0f836829419e\") " pod="openshift-ingress/e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp" Oct 30 16:32:24.510374 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:24.510351 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffdf199f-692a-4542-a7cb-0f836829419e-istio-podinfo\") pod \"e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp\" (UID: \"ffdf199f-692a-4542-a7cb-0f836829419e\") " pod="openshift-ingress/e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp" Oct 30 16:32:24.510692 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:24.510673 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffdf199f-692a-4542-a7cb-0f836829419e-istio-envoy\") pod \"e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp\" (UID: \"ffdf199f-692a-4542-a7cb-0f836829419e\") " pod="openshift-ingress/e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp" Oct 30 16:32:24.533901 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:24.533876 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffdf199f-692a-4542-a7cb-0f836829419e-istio-token\") pod \"e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp\" (UID: \"ffdf199f-692a-4542-a7cb-0f836829419e\") " pod="openshift-ingress/e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp" Oct 30 16:32:24.534430 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:24.534409 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vdwc\" (UniqueName: \"kubernetes.io/projected/ffdf199f-692a-4542-a7cb-0f836829419e-kube-api-access-8vdwc\") pod \"e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp\" (UID: \"ffdf199f-692a-4542-a7cb-0f836829419e\") " pod="openshift-ingress/e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp" Oct 30 16:32:24.643046 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:24.643012 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp" Oct 30 16:32:24.971951 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:24.971866 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp"] Oct 30 16:32:24.975682 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:32:24.975653 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffdf199f_692a_4542_a7cb_0f836829419e.slice/crio-39eeb86e0d6dc7b34ac11a04f8b7768ad5487b288276d90c54de17be83495437 WatchSource:0}: Error finding container 39eeb86e0d6dc7b34ac11a04f8b7768ad5487b288276d90c54de17be83495437: Status 404 returned error can't find the container with id 39eeb86e0d6dc7b34ac11a04f8b7768ad5487b288276d90c54de17be83495437 Oct 30 16:32:24.977911 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:24.977870 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:32:24.978006 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:24.977957 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:32:24.978006 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:24.978000 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:32:25.961676 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:25.961641 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp" event={"ID":"ffdf199f-692a-4542-a7cb-0f836829419e","Type":"ContainerStarted","Data":"897182655e46fe593dadb0d8537894485982d15adefb5d58d02bbc7b200c1f0b"} Oct 30 16:32:25.962136 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:25.961700 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp" event={"ID":"ffdf199f-692a-4542-a7cb-0f836829419e","Type":"ContainerStarted","Data":"39eeb86e0d6dc7b34ac11a04f8b7768ad5487b288276d90c54de17be83495437"} Oct 30 16:32:26.031550 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:26.031506 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp" podStartSLOduration=2.031492409 podStartE2EDuration="2.031492409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:32:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:32:26.018348795 +0000 UTC m=+573.245523876" watchObservedRunningTime="2026-10-30 16:32:26.031492409 +0000 UTC m=+573.258667584" Oct 30 16:32:26.644272 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:26.644231 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp" Oct 30 16:32:26.649483 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:26.649460 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp" Oct 30 16:32:26.976075 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:26.975989 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp" Oct 30 16:32:26.976996 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:26.976935 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp" Oct 30 16:32:30.129171 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:30.129138 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a3cc1e2b-77f9b8d87-d9qp8"] Oct 30 16:32:30.132675 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:30.132660 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a3cc1e2b-77f9b8d87-d9qp8" Oct 30 16:32:30.147012 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:30.146992 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-a3cc1e2b\"" Oct 30 16:32:30.159301 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:30.159264 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-a3cc1e2b-dockercfg-jq878\"" Oct 30 16:32:30.161382 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:30.161347 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a3cc1e2b-77f9b8d87-d9qp8"] Oct 30 16:32:30.216179 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:30.216127 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a3cc1e2b-5fbcc7fc48-64sc5"] Oct 30 16:32:30.220774 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:30.220749 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-a3cc1e2b-5fbcc7fc48-64sc5" Oct 30 16:32:30.252554 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:30.252522 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a3cc1e2b-5fbcc7fc48-64sc5"] Oct 30 16:32:30.290262 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:30.290229 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb3ac4c7-44b5-47c0-9e6c-2dd0c35a36c9-plugins-config-volume\") pod \"payload-processing-e2e-derive-a3cc1e2b-5fbcc7fc48-64sc5\" (UID: \"cb3ac4c7-44b5-47c0-9e6c-2dd0c35a36c9\") " pod="openshift-ingress/payload-processing-e2e-derive-a3cc1e2b-5fbcc7fc48-64sc5" Oct 30 16:32:30.290262 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:30.290263 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6vc7\" (UniqueName: \"kubernetes.io/projected/cb3ac4c7-44b5-47c0-9e6c-2dd0c35a36c9-kube-api-access-d6vc7\") pod \"payload-processing-e2e-derive-a3cc1e2b-5fbcc7fc48-64sc5\" (UID: \"cb3ac4c7-44b5-47c0-9e6c-2dd0c35a36c9\") " pod="openshift-ingress/payload-processing-e2e-derive-a3cc1e2b-5fbcc7fc48-64sc5" Oct 30 16:32:30.290510 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:30.290284 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pj8z\" (UniqueName: \"kubernetes.io/projected/97a5f2a3-73ee-4aa9-ae14-2a5b98f56143-kube-api-access-7pj8z\") pod \"payload-pre-processing-e2e-derive-a3cc1e2b-77f9b8d87-d9qp8\" (UID: \"97a5f2a3-73ee-4aa9-ae14-2a5b98f56143\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a3cc1e2b-77f9b8d87-d9qp8" Oct 30 16:32:30.290510 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:30.290423 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97a5f2a3-73ee-4aa9-ae14-2a5b98f56143-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-a3cc1e2b-77f9b8d87-d9qp8\" (UID: \"97a5f2a3-73ee-4aa9-ae14-2a5b98f56143\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a3cc1e2b-77f9b8d87-d9qp8" Oct 30 16:32:30.340422 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:32:30.340375 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 16:32:30.391440 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:30.391406 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb3ac4c7-44b5-47c0-9e6c-2dd0c35a36c9-plugins-config-volume\") pod \"payload-processing-e2e-derive-a3cc1e2b-5fbcc7fc48-64sc5\" (UID: \"cb3ac4c7-44b5-47c0-9e6c-2dd0c35a36c9\") " pod="openshift-ingress/payload-processing-e2e-derive-a3cc1e2b-5fbcc7fc48-64sc5" Oct 30 16:32:30.391440 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:30.391438 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6vc7\" (UniqueName: \"kubernetes.io/projected/cb3ac4c7-44b5-47c0-9e6c-2dd0c35a36c9-kube-api-access-d6vc7\") pod \"payload-processing-e2e-derive-a3cc1e2b-5fbcc7fc48-64sc5\" (UID: \"cb3ac4c7-44b5-47c0-9e6c-2dd0c35a36c9\") " pod="openshift-ingress/payload-processing-e2e-derive-a3cc1e2b-5fbcc7fc48-64sc5" Oct 30 16:32:30.391667 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:30.391457 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pj8z\" (UniqueName: \"kubernetes.io/projected/97a5f2a3-73ee-4aa9-ae14-2a5b98f56143-kube-api-access-7pj8z\") pod \"payload-pre-processing-e2e-derive-a3cc1e2b-77f9b8d87-d9qp8\" (UID: \"97a5f2a3-73ee-4aa9-ae14-2a5b98f56143\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a3cc1e2b-77f9b8d87-d9qp8" Oct 30 16:32:30.391667 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:30.391573 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97a5f2a3-73ee-4aa9-ae14-2a5b98f56143-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-a3cc1e2b-77f9b8d87-d9qp8\" (UID: \"97a5f2a3-73ee-4aa9-ae14-2a5b98f56143\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a3cc1e2b-77f9b8d87-d9qp8" Oct 30 16:32:30.392191 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:30.392170 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb3ac4c7-44b5-47c0-9e6c-2dd0c35a36c9-plugins-config-volume\") pod \"payload-processing-e2e-derive-a3cc1e2b-5fbcc7fc48-64sc5\" (UID: \"cb3ac4c7-44b5-47c0-9e6c-2dd0c35a36c9\") " pod="openshift-ingress/payload-processing-e2e-derive-a3cc1e2b-5fbcc7fc48-64sc5" Oct 30 16:32:30.392293 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:30.392275 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97a5f2a3-73ee-4aa9-ae14-2a5b98f56143-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-a3cc1e2b-77f9b8d87-d9qp8\" (UID: \"97a5f2a3-73ee-4aa9-ae14-2a5b98f56143\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a3cc1e2b-77f9b8d87-d9qp8" Oct 30 16:32:30.414711 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:30.414682 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pj8z\" (UniqueName: \"kubernetes.io/projected/97a5f2a3-73ee-4aa9-ae14-2a5b98f56143-kube-api-access-7pj8z\") pod \"payload-pre-processing-e2e-derive-a3cc1e2b-77f9b8d87-d9qp8\" (UID: \"97a5f2a3-73ee-4aa9-ae14-2a5b98f56143\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a3cc1e2b-77f9b8d87-d9qp8" Oct 30 16:32:30.414879 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:30.414691 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6vc7\" (UniqueName: \"kubernetes.io/projected/cb3ac4c7-44b5-47c0-9e6c-2dd0c35a36c9-kube-api-access-d6vc7\") pod \"payload-processing-e2e-derive-a3cc1e2b-5fbcc7fc48-64sc5\" (UID: \"cb3ac4c7-44b5-47c0-9e6c-2dd0c35a36c9\") " pod="openshift-ingress/payload-processing-e2e-derive-a3cc1e2b-5fbcc7fc48-64sc5" Oct 30 16:32:30.445578 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:30.445547 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a3cc1e2b-77f9b8d87-d9qp8" Oct 30 16:32:30.536266 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:30.536236 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-a3cc1e2b-5fbcc7fc48-64sc5" Oct 30 16:32:30.745224 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:30.745081 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a3cc1e2b-77f9b8d87-d9qp8"] Oct 30 16:32:30.747626 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:32:30.747600 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97a5f2a3_73ee_4aa9_ae14_2a5b98f56143.slice/crio-077aae0faa22e5e138f1f0a2d26bdf4ffc401550c72c082bda6bf1b221480c97 WatchSource:0}: Error finding container 077aae0faa22e5e138f1f0a2d26bdf4ffc401550c72c082bda6bf1b221480c97: Status 404 returned error can't find the container with id 077aae0faa22e5e138f1f0a2d26bdf4ffc401550c72c082bda6bf1b221480c97 Oct 30 16:32:30.840229 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:30.840205 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a3cc1e2b-5fbcc7fc48-64sc5"] Oct 30 16:32:30.842105 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:32:30.842076 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb3ac4c7_44b5_47c0_9e6c_2dd0c35a36c9.slice/crio-c5708cd98bd777858d5b54091a50d0a2451e5d2ba2d4e177213f745b3791a8e3 WatchSource:0}: Error finding container c5708cd98bd777858d5b54091a50d0a2451e5d2ba2d4e177213f745b3791a8e3: Status 404 returned error can't find the container with id c5708cd98bd777858d5b54091a50d0a2451e5d2ba2d4e177213f745b3791a8e3 Oct 30 16:32:30.996665 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:30.996576 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a3cc1e2b-77f9b8d87-d9qp8" event={"ID":"97a5f2a3-73ee-4aa9-ae14-2a5b98f56143","Type":"ContainerStarted","Data":"077aae0faa22e5e138f1f0a2d26bdf4ffc401550c72c082bda6bf1b221480c97"} Oct 30 16:32:30.997629 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:30.997607 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-a3cc1e2b-5fbcc7fc48-64sc5" event={"ID":"cb3ac4c7-44b5-47c0-9e6c-2dd0c35a36c9","Type":"ContainerStarted","Data":"c5708cd98bd777858d5b54091a50d0a2451e5d2ba2d4e177213f745b3791a8e3"} Oct 30 16:32:31.570825 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:31.564718 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 16:32:34.025367 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:34.025336 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-a3cc1e2b-5fbcc7fc48-64sc5" event={"ID":"cb3ac4c7-44b5-47c0-9e6c-2dd0c35a36c9","Type":"ContainerStarted","Data":"c1fa0af1d095b2a215272f2aca9dbc5c5eb45f2329c2d1c4f6ce6d5748ce223e"} Oct 30 16:32:34.025821 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:34.025415 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-a3cc1e2b-5fbcc7fc48-64sc5" Oct 30 16:32:34.027076 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:34.027051 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a3cc1e2b-77f9b8d87-d9qp8" event={"ID":"97a5f2a3-73ee-4aa9-ae14-2a5b98f56143","Type":"ContainerStarted","Data":"c55e51f13d640fef8bb1797dd25071c19273ce76d3115f748ccf89380098b235"} Oct 30 16:32:34.027184 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:34.027118 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-a3cc1e2b-77f9b8d87-d9qp8" Oct 30 16:32:34.065615 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:34.065563 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-a3cc1e2b-5fbcc7fc48-64sc5" podStartSLOduration=1.304494297 podStartE2EDuration="4.065552205s" totalImagesPullingTime="2.761057908s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:32:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:32:34.065472623 +0000 UTC m=+581.292647723" watchObservedRunningTime="2026-10-30 16:32:34.065552205 +0000 UTC m=+581.292727302" Oct 30 16:32:34.105690 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:34.105635 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-a3cc1e2b-77f9b8d87-d9qp8" podStartSLOduration=1.207884015 podStartE2EDuration="4.105618655s" totalImagesPullingTime="2.89773464s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:32:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:32:34.105054614 +0000 UTC m=+581.332229715" watchObservedRunningTime="2026-10-30 16:32:34.105618655 +0000 UTC m=+581.332793749" Oct 30 16:32:38.359408 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:38.359371 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-37964af0-j4tvw"] Oct 30 16:32:38.362906 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:38.362889 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-37964af0-j4tvw" Oct 30 16:32:38.377265 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:38.377248 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:32:38.377391 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:38.377377 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:32:38.389145 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:38.389120 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:32:38.389284 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:38.389234 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 16:32:38.392467 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:38.392440 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-37964af0-j4tvw"] Oct 30 16:32:38.489378 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:38.483914 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6jb5\" (UniqueName: \"kubernetes.io/projected/164bd713-ddec-41fd-bc27-81b43e3509c2-kube-api-access-q6jb5\") pod \"maas-api-revoke-keys-e2e-migrate-37964af0-j4tvw\" (UID: \"164bd713-ddec-41fd-bc27-81b43e3509c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-37964af0-j4tvw" Oct 30 16:32:38.489378 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:38.484012 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/164bd713-ddec-41fd-bc27-81b43e3509c2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-37964af0-j4tvw\" (UID: \"164bd713-ddec-41fd-bc27-81b43e3509c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-37964af0-j4tvw" Oct 30 16:32:38.489378 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:38.484051 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/164bd713-ddec-41fd-bc27-81b43e3509c2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-37964af0-j4tvw\" (UID: \"164bd713-ddec-41fd-bc27-81b43e3509c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-37964af0-j4tvw" Oct 30 16:32:38.585652 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:38.585611 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6jb5\" (UniqueName: \"kubernetes.io/projected/164bd713-ddec-41fd-bc27-81b43e3509c2-kube-api-access-q6jb5\") pod \"maas-api-revoke-keys-e2e-migrate-37964af0-j4tvw\" (UID: \"164bd713-ddec-41fd-bc27-81b43e3509c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-37964af0-j4tvw" Oct 30 16:32:38.585872 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:38.585697 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/164bd713-ddec-41fd-bc27-81b43e3509c2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-37964af0-j4tvw\" (UID: \"164bd713-ddec-41fd-bc27-81b43e3509c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-37964af0-j4tvw" Oct 30 16:32:38.585872 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:38.585725 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/164bd713-ddec-41fd-bc27-81b43e3509c2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-37964af0-j4tvw\" (UID: \"164bd713-ddec-41fd-bc27-81b43e3509c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-37964af0-j4tvw" Oct 30 16:32:38.586359 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:38.586339 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/164bd713-ddec-41fd-bc27-81b43e3509c2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-37964af0-j4tvw\" (UID: \"164bd713-ddec-41fd-bc27-81b43e3509c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-37964af0-j4tvw" Oct 30 16:32:38.588340 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:38.588314 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/164bd713-ddec-41fd-bc27-81b43e3509c2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-37964af0-j4tvw\" (UID: \"164bd713-ddec-41fd-bc27-81b43e3509c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-37964af0-j4tvw" Oct 30 16:32:38.609259 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:38.609233 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6jb5\" (UniqueName: \"kubernetes.io/projected/164bd713-ddec-41fd-bc27-81b43e3509c2-kube-api-access-q6jb5\") pod \"maas-api-revoke-keys-e2e-migrate-37964af0-j4tvw\" (UID: \"164bd713-ddec-41fd-bc27-81b43e3509c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-37964af0-j4tvw" Oct 30 16:32:38.676074 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:38.676050 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-37964af0-j4tvw" Oct 30 16:32:38.977004 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:38.976971 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-37964af0-j4tvw"] Oct 30 16:32:38.979593 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:32:38.979558 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod164bd713_ddec_41fd_bc27_81b43e3509c2.slice/crio-79da8c6aa46f28cc656eebb2687ed64b7545aae0a059f4feebcd8527135d5758 WatchSource:0}: Error finding container 79da8c6aa46f28cc656eebb2687ed64b7545aae0a059f4feebcd8527135d5758: Status 404 returned error can't find the container with id 79da8c6aa46f28cc656eebb2687ed64b7545aae0a059f4feebcd8527135d5758 Oct 30 16:32:39.066258 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:39.066220 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-37964af0-j4tvw" event={"ID":"164bd713-ddec-41fd-bc27-81b43e3509c2","Type":"ContainerStarted","Data":"79da8c6aa46f28cc656eebb2687ed64b7545aae0a059f4feebcd8527135d5758"} Oct 30 16:32:39.499904 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:39.499829 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f756b7bf-gldcf"] Oct 30 16:32:39.503362 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:39.503341 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f756b7bf-gldcf" Oct 30 16:32:39.526150 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:39.526119 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f756b7bf-gldcf"] Oct 30 16:32:39.694958 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:39.694923 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv6zn\" (UniqueName: \"kubernetes.io/projected/cfb3b25e-8504-4b5e-9145-43eab49d0d18-kube-api-access-rv6zn\") pod \"maas-api-revoke-keys-e2e-ait-f756b7bf-gldcf\" (UID: \"cfb3b25e-8504-4b5e-9145-43eab49d0d18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f756b7bf-gldcf" Oct 30 16:32:39.694958 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:39.694964 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfb3b25e-8504-4b5e-9145-43eab49d0d18-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f756b7bf-gldcf\" (UID: \"cfb3b25e-8504-4b5e-9145-43eab49d0d18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f756b7bf-gldcf" Oct 30 16:32:39.695132 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:39.694995 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfb3b25e-8504-4b5e-9145-43eab49d0d18-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f756b7bf-gldcf\" (UID: \"cfb3b25e-8504-4b5e-9145-43eab49d0d18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f756b7bf-gldcf" Oct 30 16:32:39.796138 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:39.796061 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfb3b25e-8504-4b5e-9145-43eab49d0d18-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f756b7bf-gldcf\" (UID: \"cfb3b25e-8504-4b5e-9145-43eab49d0d18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f756b7bf-gldcf" Oct 30 16:32:39.796284 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:39.796149 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv6zn\" (UniqueName: \"kubernetes.io/projected/cfb3b25e-8504-4b5e-9145-43eab49d0d18-kube-api-access-rv6zn\") pod \"maas-api-revoke-keys-e2e-ait-f756b7bf-gldcf\" (UID: \"cfb3b25e-8504-4b5e-9145-43eab49d0d18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f756b7bf-gldcf" Oct 30 16:32:39.796284 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:39.796174 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfb3b25e-8504-4b5e-9145-43eab49d0d18-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f756b7bf-gldcf\" (UID: \"cfb3b25e-8504-4b5e-9145-43eab49d0d18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f756b7bf-gldcf" Oct 30 16:32:39.796775 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:39.796756 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfb3b25e-8504-4b5e-9145-43eab49d0d18-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f756b7bf-gldcf\" (UID: \"cfb3b25e-8504-4b5e-9145-43eab49d0d18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f756b7bf-gldcf" Oct 30 16:32:39.798421 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:39.798399 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfb3b25e-8504-4b5e-9145-43eab49d0d18-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f756b7bf-gldcf\" (UID: \"cfb3b25e-8504-4b5e-9145-43eab49d0d18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f756b7bf-gldcf" Oct 30 16:32:39.819069 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:39.819044 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv6zn\" (UniqueName: \"kubernetes.io/projected/cfb3b25e-8504-4b5e-9145-43eab49d0d18-kube-api-access-rv6zn\") pod \"maas-api-revoke-keys-e2e-ait-f756b7bf-gldcf\" (UID: \"cfb3b25e-8504-4b5e-9145-43eab49d0d18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f756b7bf-gldcf" Oct 30 16:32:40.073485 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:40.073400 4493 generic.go:356] "Generic (PLEG): container finished" podID="164bd713-ddec-41fd-bc27-81b43e3509c2" containerID="b0ba9d1791f004d2c10ecc3081717a460f7b1ac2ead7104e0fa02513a96c008c" exitCode=0 Oct 30 16:32:40.073629 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:40.073490 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-37964af0-j4tvw" event={"ID":"164bd713-ddec-41fd-bc27-81b43e3509c2","Type":"ContainerDied","Data":"b0ba9d1791f004d2c10ecc3081717a460f7b1ac2ead7104e0fa02513a96c008c"} Oct 30 16:32:40.116092 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:40.116062 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f756b7bf-gldcf" Oct 30 16:32:40.415452 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:40.415418 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f756b7bf-gldcf"] Oct 30 16:32:40.417203 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:32:40.417164 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfb3b25e_8504_4b5e_9145_43eab49d0d18.slice/crio-d416550ab652eb85464c89b99f071ceac8fbb59cc9e1929024e27103ea2782c9 WatchSource:0}: Error finding container d416550ab652eb85464c89b99f071ceac8fbb59cc9e1929024e27103ea2782c9: Status 404 returned error can't find the container with id d416550ab652eb85464c89b99f071ceac8fbb59cc9e1929024e27103ea2782c9 Oct 30 16:32:41.080345 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:41.080263 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f756b7bf-gldcf" event={"ID":"cfb3b25e-8504-4b5e-9145-43eab49d0d18","Type":"ContainerStarted","Data":"8fd84d82bc6ef4ef3bbe5870c46c3a3fa547e6442e61e50e2d2d8dc9a5c76150"} Oct 30 16:32:41.080345 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:41.080318 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f756b7bf-gldcf" event={"ID":"cfb3b25e-8504-4b5e-9145-43eab49d0d18","Type":"ContainerStarted","Data":"d416550ab652eb85464c89b99f071ceac8fbb59cc9e1929024e27103ea2782c9"} Oct 30 16:32:41.127588 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:41.127516 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f756b7bf-gldcf" podStartSLOduration=2.127498779 podStartE2EDuration="2.127498779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:32:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:32:41.124402823 +0000 UTC m=+588.351577922" watchObservedRunningTime="2026-10-30 16:32:41.127498779 +0000 UTC m=+588.354673881" Oct 30 16:32:41.238445 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:41.238415 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-37964af0-j4tvw" Oct 30 16:32:41.411406 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:41.411377 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/164bd713-ddec-41fd-bc27-81b43e3509c2-kube-api-access-q6jb5\" (UniqueName: \"kubernetes.io/projected/164bd713-ddec-41fd-bc27-81b43e3509c2-kube-api-access-q6jb5\") pod \"164bd713-ddec-41fd-bc27-81b43e3509c2\" (UID: \"164bd713-ddec-41fd-bc27-81b43e3509c2\") " Oct 30 16:32:41.411566 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:41.411448 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/164bd713-ddec-41fd-bc27-81b43e3509c2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/164bd713-ddec-41fd-bc27-81b43e3509c2-maas-api-openshift-ca\") pod \"164bd713-ddec-41fd-bc27-81b43e3509c2\" (UID: \"164bd713-ddec-41fd-bc27-81b43e3509c2\") " Oct 30 16:32:41.411566 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:41.411488 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/164bd713-ddec-41fd-bc27-81b43e3509c2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/164bd713-ddec-41fd-bc27-81b43e3509c2-maas-api-xks-ca\") pod \"164bd713-ddec-41fd-bc27-81b43e3509c2\" (UID: \"164bd713-ddec-41fd-bc27-81b43e3509c2\") " Oct 30 16:32:41.411895 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:41.411864 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/164bd713-ddec-41fd-bc27-81b43e3509c2-maas-api-openshift-ca" pod "164bd713-ddec-41fd-bc27-81b43e3509c2" (UID: "164bd713-ddec-41fd-bc27-81b43e3509c2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:32:41.413552 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:41.413526 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/164bd713-ddec-41fd-bc27-81b43e3509c2-kube-api-access-q6jb5" pod "164bd713-ddec-41fd-bc27-81b43e3509c2" (UID: "164bd713-ddec-41fd-bc27-81b43e3509c2"). InnerVolumeSpecName "kube-api-access-q6jb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:32:41.413648 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:41.413599 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/164bd713-ddec-41fd-bc27-81b43e3509c2-maas-api-xks-ca" pod "164bd713-ddec-41fd-bc27-81b43e3509c2" (UID: "164bd713-ddec-41fd-bc27-81b43e3509c2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:32:41.485505 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:41.485462 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 16:32:41.514149 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:41.514110 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6jb5\" (UniqueName: \"kubernetes.io/projected/164bd713-ddec-41fd-bc27-81b43e3509c2-kube-api-access-q6jb5\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:32:41.514149 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:41.514143 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/164bd713-ddec-41fd-bc27-81b43e3509c2-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:32:41.514149 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:41.514156 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/164bd713-ddec-41fd-bc27-81b43e3509c2-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:32:42.088425 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:42.088394 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-37964af0-j4tvw" Oct 30 16:32:42.088892 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:42.088434 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-37964af0-j4tvw" event={"ID":"164bd713-ddec-41fd-bc27-81b43e3509c2","Type":"ContainerDied","Data":"79da8c6aa46f28cc656eebb2687ed64b7545aae0a059f4feebcd8527135d5758"} Oct 30 16:32:42.088892 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:42.088496 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79da8c6aa46f28cc656eebb2687ed64b7545aae0a059f4feebcd8527135d5758" Oct 30 16:32:42.090153 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:42.090125 4493 generic.go:356] "Generic (PLEG): container finished" podID="cfb3b25e-8504-4b5e-9145-43eab49d0d18" containerID="8fd84d82bc6ef4ef3bbe5870c46c3a3fa547e6442e61e50e2d2d8dc9a5c76150" exitCode=0 Oct 30 16:32:42.090266 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:42.090202 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f756b7bf-gldcf" event={"ID":"cfb3b25e-8504-4b5e-9145-43eab49d0d18","Type":"ContainerDied","Data":"8fd84d82bc6ef4ef3bbe5870c46c3a3fa547e6442e61e50e2d2d8dc9a5c76150"} Oct 30 16:32:43.283772 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:43.283748 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f756b7bf-gldcf" Oct 30 16:32:43.343105 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:43.343076 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfb3b25e-8504-4b5e-9145-43eab49d0d18-kube-api-access-rv6zn\" (UniqueName: \"kubernetes.io/projected/cfb3b25e-8504-4b5e-9145-43eab49d0d18-kube-api-access-rv6zn\") pod \"cfb3b25e-8504-4b5e-9145-43eab49d0d18\" (UID: \"cfb3b25e-8504-4b5e-9145-43eab49d0d18\") " Oct 30 16:32:43.345490 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:43.345426 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfb3b25e-8504-4b5e-9145-43eab49d0d18-kube-api-access-rv6zn" pod "cfb3b25e-8504-4b5e-9145-43eab49d0d18" (UID: "cfb3b25e-8504-4b5e-9145-43eab49d0d18"). InnerVolumeSpecName "kube-api-access-rv6zn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:32:43.444181 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:43.444150 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfb3b25e-8504-4b5e-9145-43eab49d0d18-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfb3b25e-8504-4b5e-9145-43eab49d0d18-maas-api-xks-ca\") pod \"cfb3b25e-8504-4b5e-9145-43eab49d0d18\" (UID: \"cfb3b25e-8504-4b5e-9145-43eab49d0d18\") " Oct 30 16:32:43.444181 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:43.444189 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfb3b25e-8504-4b5e-9145-43eab49d0d18-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfb3b25e-8504-4b5e-9145-43eab49d0d18-maas-api-openshift-ca\") pod \"cfb3b25e-8504-4b5e-9145-43eab49d0d18\" (UID: \"cfb3b25e-8504-4b5e-9145-43eab49d0d18\") " Oct 30 16:32:43.444428 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:43.444382 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rv6zn\" (UniqueName: \"kubernetes.io/projected/cfb3b25e-8504-4b5e-9145-43eab49d0d18-kube-api-access-rv6zn\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:32:43.444561 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:43.444536 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfb3b25e-8504-4b5e-9145-43eab49d0d18-maas-api-openshift-ca" pod "cfb3b25e-8504-4b5e-9145-43eab49d0d18" (UID: "cfb3b25e-8504-4b5e-9145-43eab49d0d18"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:32:43.446387 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:43.446349 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfb3b25e-8504-4b5e-9145-43eab49d0d18-maas-api-xks-ca" pod "cfb3b25e-8504-4b5e-9145-43eab49d0d18" (UID: "cfb3b25e-8504-4b5e-9145-43eab49d0d18"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:32:43.544715 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:43.544685 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfb3b25e-8504-4b5e-9145-43eab49d0d18-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:32:43.544715 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:43.544710 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfb3b25e-8504-4b5e-9145-43eab49d0d18-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:32:44.101304 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:44.101277 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f756b7bf-gldcf" Oct 30 16:32:44.101491 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:44.101278 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f756b7bf-gldcf" event={"ID":"cfb3b25e-8504-4b5e-9145-43eab49d0d18","Type":"ContainerDied","Data":"d416550ab652eb85464c89b99f071ceac8fbb59cc9e1929024e27103ea2782c9"} Oct 30 16:32:44.101491 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:44.101408 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d416550ab652eb85464c89b99f071ceac8fbb59cc9e1929024e27103ea2782c9" Oct 30 16:32:44.927704 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:44.927652 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cff59996-ktdqk"] Oct 30 16:32:44.930464 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:44.930430 4493 cpu_manager.go:414] "containerMap: removing container" podUID="cfb3b25e-8504-4b5e-9145-43eab49d0d18" containerName="revoke-keys" Oct 30 16:32:44.930464 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:44.930458 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfb3b25e-8504-4b5e-9145-43eab49d0d18" containerName="revoke-keys" Oct 30 16:32:44.930664 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:44.930495 4493 cpu_manager.go:414] "containerMap: removing container" podUID="164bd713-ddec-41fd-bc27-81b43e3509c2" containerName="revoke-keys" Oct 30 16:32:44.930664 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:44.930505 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="164bd713-ddec-41fd-bc27-81b43e3509c2" containerName="revoke-keys" Oct 30 16:32:44.930664 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:44.930602 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfb3b25e-8504-4b5e-9145-43eab49d0d18" containerName="revoke-keys" Oct 30 16:32:44.930664 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:44.930618 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="164bd713-ddec-41fd-bc27-81b43e3509c2" containerName="revoke-keys" Oct 30 16:32:44.934515 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:44.934492 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cff59996-ktdqk" Oct 30 16:32:44.949213 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:44.949188 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:32:44.952376 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:44.952345 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:32:44.955401 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:44.955381 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cff59996-ktdqk"] Oct 30 16:32:44.957023 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:44.956997 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91812800-a291-44cf-b47d-ee66eb4c2502-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-cff59996-ktdqk\" (UID: \"91812800-a291-44cf-b47d-ee66eb4c2502\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cff59996-ktdqk" Oct 30 16:32:44.957141 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:44.957039 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nn2x\" (UniqueName: \"kubernetes.io/projected/91812800-a291-44cf-b47d-ee66eb4c2502-kube-api-access-9nn2x\") pod \"maas-api-revoke-keys-e2e-ait-cff59996-ktdqk\" (UID: \"91812800-a291-44cf-b47d-ee66eb4c2502\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cff59996-ktdqk" Oct 30 16:32:44.957290 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:44.957265 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91812800-a291-44cf-b47d-ee66eb4c2502-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-cff59996-ktdqk\" (UID: \"91812800-a291-44cf-b47d-ee66eb4c2502\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cff59996-ktdqk" Oct 30 16:32:44.960987 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:44.960967 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 16:32:45.028832 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:45.027053 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:32:45.035454 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:45.035423 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-a3cc1e2b-5fbcc7fc48-64sc5" Oct 30 16:32:45.035583 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:45.035498 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-a3cc1e2b-77f9b8d87-d9qp8" Oct 30 16:32:45.058195 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:45.058166 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91812800-a291-44cf-b47d-ee66eb4c2502-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-cff59996-ktdqk\" (UID: \"91812800-a291-44cf-b47d-ee66eb4c2502\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cff59996-ktdqk" Oct 30 16:32:45.058343 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:45.058202 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nn2x\" (UniqueName: \"kubernetes.io/projected/91812800-a291-44cf-b47d-ee66eb4c2502-kube-api-access-9nn2x\") pod \"maas-api-revoke-keys-e2e-ait-cff59996-ktdqk\" (UID: \"91812800-a291-44cf-b47d-ee66eb4c2502\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cff59996-ktdqk" Oct 30 16:32:45.058343 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:45.058309 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91812800-a291-44cf-b47d-ee66eb4c2502-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-cff59996-ktdqk\" (UID: \"91812800-a291-44cf-b47d-ee66eb4c2502\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cff59996-ktdqk" Oct 30 16:32:45.058847 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:45.058827 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91812800-a291-44cf-b47d-ee66eb4c2502-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-cff59996-ktdqk\" (UID: \"91812800-a291-44cf-b47d-ee66eb4c2502\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cff59996-ktdqk" Oct 30 16:32:45.060686 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:45.060656 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91812800-a291-44cf-b47d-ee66eb4c2502-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-cff59996-ktdqk\" (UID: \"91812800-a291-44cf-b47d-ee66eb4c2502\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cff59996-ktdqk" Oct 30 16:32:45.082454 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:45.082433 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nn2x\" (UniqueName: \"kubernetes.io/projected/91812800-a291-44cf-b47d-ee66eb4c2502-kube-api-access-9nn2x\") pod \"maas-api-revoke-keys-e2e-ait-cff59996-ktdqk\" (UID: \"91812800-a291-44cf-b47d-ee66eb4c2502\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cff59996-ktdqk" Oct 30 16:32:45.247417 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:45.247327 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cff59996-ktdqk" Oct 30 16:32:45.547570 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:45.547490 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cff59996-ktdqk"] Oct 30 16:32:45.550340 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:32:45.550312 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91812800_a291_44cf_b47d_ee66eb4c2502.slice/crio-fd6ec44398da3880e16477b929823bf8760c826d95fb9dd6ff95059d6a257ac8 WatchSource:0}: Error finding container fd6ec44398da3880e16477b929823bf8760c826d95fb9dd6ff95059d6a257ac8: Status 404 returned error can't find the container with id fd6ec44398da3880e16477b929823bf8760c826d95fb9dd6ff95059d6a257ac8 Oct 30 16:32:46.141559 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:46.141528 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cff59996-ktdqk" event={"ID":"91812800-a291-44cf-b47d-ee66eb4c2502","Type":"ContainerStarted","Data":"579000dd3e8d320d666c49789abeb9627373b19b2d7eb4ec8c7b21bc381035bd"} Oct 30 16:32:46.142015 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:46.141580 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cff59996-ktdqk" event={"ID":"91812800-a291-44cf-b47d-ee66eb4c2502","Type":"ContainerStarted","Data":"fd6ec44398da3880e16477b929823bf8760c826d95fb9dd6ff95059d6a257ac8"} Oct 30 16:32:46.185602 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:46.185508 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cff59996-ktdqk" podStartSLOduration=2.185495862 podStartE2EDuration="2.185495862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:32:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:32:46.183262182 +0000 UTC m=+593.410437282" watchObservedRunningTime="2026-10-30 16:32:46.185495862 +0000 UTC m=+593.412670962" Oct 30 16:32:47.149653 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:47.149621 4493 generic.go:356] "Generic (PLEG): container finished" podID="91812800-a291-44cf-b47d-ee66eb4c2502" containerID="579000dd3e8d320d666c49789abeb9627373b19b2d7eb4ec8c7b21bc381035bd" exitCode=0 Oct 30 16:32:47.150050 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:47.149665 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cff59996-ktdqk" event={"ID":"91812800-a291-44cf-b47d-ee66eb4c2502","Type":"ContainerDied","Data":"579000dd3e8d320d666c49789abeb9627373b19b2d7eb4ec8c7b21bc381035bd"} Oct 30 16:32:48.323343 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:48.323313 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cff59996-ktdqk" Oct 30 16:32:48.388515 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:48.388481 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91812800-a291-44cf-b47d-ee66eb4c2502-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91812800-a291-44cf-b47d-ee66eb4c2502-maas-api-xks-ca\") pod \"91812800-a291-44cf-b47d-ee66eb4c2502\" (UID: \"91812800-a291-44cf-b47d-ee66eb4c2502\") " Oct 30 16:32:48.388691 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:48.388531 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91812800-a291-44cf-b47d-ee66eb4c2502-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91812800-a291-44cf-b47d-ee66eb4c2502-maas-api-openshift-ca\") pod \"91812800-a291-44cf-b47d-ee66eb4c2502\" (UID: \"91812800-a291-44cf-b47d-ee66eb4c2502\") " Oct 30 16:32:48.388691 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:48.388584 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91812800-a291-44cf-b47d-ee66eb4c2502-kube-api-access-9nn2x\" (UniqueName: \"kubernetes.io/projected/91812800-a291-44cf-b47d-ee66eb4c2502-kube-api-access-9nn2x\") pod \"91812800-a291-44cf-b47d-ee66eb4c2502\" (UID: \"91812800-a291-44cf-b47d-ee66eb4c2502\") " Oct 30 16:32:48.389009 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:48.388976 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91812800-a291-44cf-b47d-ee66eb4c2502-maas-api-openshift-ca" pod "91812800-a291-44cf-b47d-ee66eb4c2502" (UID: "91812800-a291-44cf-b47d-ee66eb4c2502"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:32:48.391024 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:48.390995 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91812800-a291-44cf-b47d-ee66eb4c2502-maas-api-xks-ca" pod "91812800-a291-44cf-b47d-ee66eb4c2502" (UID: "91812800-a291-44cf-b47d-ee66eb4c2502"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:32:48.391133 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:48.391062 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91812800-a291-44cf-b47d-ee66eb4c2502-kube-api-access-9nn2x" pod "91812800-a291-44cf-b47d-ee66eb4c2502" (UID: "91812800-a291-44cf-b47d-ee66eb4c2502"). InnerVolumeSpecName "kube-api-access-9nn2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:32:48.489796 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:48.489698 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91812800-a291-44cf-b47d-ee66eb4c2502-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:32:48.489796 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:48.489728 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91812800-a291-44cf-b47d-ee66eb4c2502-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:32:48.489796 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:48.489739 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nn2x\" (UniqueName: \"kubernetes.io/projected/91812800-a291-44cf-b47d-ee66eb4c2502-kube-api-access-9nn2x\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:32:49.162363 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:49.162338 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cff59996-ktdqk" Oct 30 16:32:49.162525 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:49.162330 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cff59996-ktdqk" event={"ID":"91812800-a291-44cf-b47d-ee66eb4c2502","Type":"ContainerDied","Data":"fd6ec44398da3880e16477b929823bf8760c826d95fb9dd6ff95059d6a257ac8"} Oct 30 16:32:49.162525 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:49.162477 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd6ec44398da3880e16477b929823bf8760c826d95fb9dd6ff95059d6a257ac8" Oct 30 16:32:51.173899 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:51.173871 4493 generic.go:356] "Generic (PLEG): container finished" podID="ffdf199f-692a-4542-a7cb-0f836829419e" containerID="897182655e46fe593dadb0d8537894485982d15adefb5d58d02bbc7b200c1f0b" exitCode=0 Oct 30 16:32:51.174528 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:51.173945 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp" event={"ID":"ffdf199f-692a-4542-a7cb-0f836829419e","Type":"ContainerDied","Data":"897182655e46fe593dadb0d8537894485982d15adefb5d58d02bbc7b200c1f0b"} Oct 30 16:32:51.174528 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:51.174301 4493 scope.go:122] "RemoveContainer" containerID="897182655e46fe593dadb0d8537894485982d15adefb5d58d02bbc7b200c1f0b" Oct 30 16:32:51.175490 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:51.175460 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:32:51.175590 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:51.175530 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:32:51.175590 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:51.175565 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:32:52.180843 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:52.180809 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp" event={"ID":"ffdf199f-692a-4542-a7cb-0f836829419e","Type":"ContainerStarted","Data":"c982ff5d85ebb59769f2c48fd066136171c6a653227fb753de331c28d401473f"} Oct 30 16:32:52.643935 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:52.643902 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp" Oct 30 16:32:52.645109 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:52.645079 4493 patch_prober.go:29] interesting pod/e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 30 16:32:52.645250 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:52.645128 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp" podUID="ffdf199f-692a-4542-a7cb-0f836829419e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 30 16:32:52.921861 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:52.914584 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a3cc1e2b-db877"] Oct 30 16:32:52.929340 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:52.929305 4493 cpu_manager.go:414] "containerMap: removing container" podUID="91812800-a291-44cf-b47d-ee66eb4c2502" containerName="revoke-keys" Oct 30 16:32:52.929340 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:52.929336 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91812800-a291-44cf-b47d-ee66eb4c2502" containerName="revoke-keys" Oct 30 16:32:52.929501 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:52.929449 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="91812800-a291-44cf-b47d-ee66eb4c2502" containerName="revoke-keys" Oct 30 16:32:52.933828 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:52.933805 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a3cc1e2b-db877" Oct 30 16:32:52.937088 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:52.937058 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32d78414-e90e-4273-ba07-2a9c34ed1871-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-a3cc1e2b-db877\" (UID: \"32d78414-e90e-4273-ba07-2a9c34ed1871\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a3cc1e2b-db877" Oct 30 16:32:52.937231 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:52.937205 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm288\" (UniqueName: \"kubernetes.io/projected/32d78414-e90e-4273-ba07-2a9c34ed1871-kube-api-access-qm288\") pod \"maas-api-revoke-keys-e2e-derive-a3cc1e2b-db877\" (UID: \"32d78414-e90e-4273-ba07-2a9c34ed1871\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a3cc1e2b-db877" Oct 30 16:32:52.937301 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:52.937280 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32d78414-e90e-4273-ba07-2a9c34ed1871-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-a3cc1e2b-db877\" (UID: \"32d78414-e90e-4273-ba07-2a9c34ed1871\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a3cc1e2b-db877" Oct 30 16:32:52.941702 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:52.941675 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a3cc1e2b-db877"] Oct 30 16:32:52.948396 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:52.948370 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:32:52.960519 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:52.959945 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:32:52.960519 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:52.959971 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:32:52.960519 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:52.959999 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 16:32:52.999718 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:52.999695 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-56v9s_d3a1c417-2698-4462-b7ff-1d2e66d7a10f/ovn-acl-logging/0.log" Oct 30 16:32:52.999916 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:52.999896 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-56v9s_d3a1c417-2698-4462-b7ff-1d2e66d7a10f/ovn-acl-logging/0.log" Oct 30 16:32:53.037931 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:53.037903 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32d78414-e90e-4273-ba07-2a9c34ed1871-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-a3cc1e2b-db877\" (UID: \"32d78414-e90e-4273-ba07-2a9c34ed1871\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a3cc1e2b-db877" Oct 30 16:32:53.038109 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:53.037958 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm288\" (UniqueName: \"kubernetes.io/projected/32d78414-e90e-4273-ba07-2a9c34ed1871-kube-api-access-qm288\") pod \"maas-api-revoke-keys-e2e-derive-a3cc1e2b-db877\" (UID: \"32d78414-e90e-4273-ba07-2a9c34ed1871\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a3cc1e2b-db877" Oct 30 16:32:53.038109 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:53.037990 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32d78414-e90e-4273-ba07-2a9c34ed1871-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-a3cc1e2b-db877\" (UID: \"32d78414-e90e-4273-ba07-2a9c34ed1871\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a3cc1e2b-db877" Oct 30 16:32:53.038616 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:53.038588 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32d78414-e90e-4273-ba07-2a9c34ed1871-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-a3cc1e2b-db877\" (UID: \"32d78414-e90e-4273-ba07-2a9c34ed1871\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a3cc1e2b-db877" Oct 30 16:32:53.040430 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:53.040406 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32d78414-e90e-4273-ba07-2a9c34ed1871-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-a3cc1e2b-db877\" (UID: \"32d78414-e90e-4273-ba07-2a9c34ed1871\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a3cc1e2b-db877" Oct 30 16:32:53.062937 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:53.062908 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm288\" (UniqueName: \"kubernetes.io/projected/32d78414-e90e-4273-ba07-2a9c34ed1871-kube-api-access-qm288\") pod \"maas-api-revoke-keys-e2e-derive-a3cc1e2b-db877\" (UID: \"32d78414-e90e-4273-ba07-2a9c34ed1871\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a3cc1e2b-db877" Oct 30 16:32:53.247639 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:53.247556 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a3cc1e2b-db877" Oct 30 16:32:53.546617 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:53.546594 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a3cc1e2b-db877"] Oct 30 16:32:53.548718 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:32:53.548691 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32d78414_e90e_4273_ba07_2a9c34ed1871.slice/crio-de90a5aa497c18f0224de00d808e2782015526552c3d48cbc921d0a84ea19c25 WatchSource:0}: Error finding container de90a5aa497c18f0224de00d808e2782015526552c3d48cbc921d0a84ea19c25: Status 404 returned error can't find the container with id de90a5aa497c18f0224de00d808e2782015526552c3d48cbc921d0a84ea19c25 Oct 30 16:32:53.643597 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:53.643563 4493 patch_prober.go:29] interesting pod/e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 30 16:32:53.643755 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:53.643612 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp" podUID="ffdf199f-692a-4542-a7cb-0f836829419e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 30 16:32:54.193051 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:54.192965 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a3cc1e2b-db877" event={"ID":"32d78414-e90e-4273-ba07-2a9c34ed1871","Type":"ContainerStarted","Data":"c33ba945c787a7d93de0178ec45729069d73adb4ff8f538a7bb378421453511a"} Oct 30 16:32:54.193051 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:54.193028 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a3cc1e2b-db877" event={"ID":"32d78414-e90e-4273-ba07-2a9c34ed1871","Type":"ContainerStarted","Data":"de90a5aa497c18f0224de00d808e2782015526552c3d48cbc921d0a84ea19c25"} Oct 30 16:32:54.250733 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:54.250687 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a3cc1e2b-db877" podStartSLOduration=2.25067477 podStartE2EDuration="2.25067477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:32:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:32:54.248909141 +0000 UTC m=+601.476084240" watchObservedRunningTime="2026-10-30 16:32:54.25067477 +0000 UTC m=+601.477849869" Oct 30 16:32:54.643678 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:54.643651 4493 patch_prober.go:29] interesting pod/e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 30 16:32:54.643859 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:54.643701 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp" podUID="ffdf199f-692a-4542-a7cb-0f836829419e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 30 16:32:55.201120 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:55.201091 4493 generic.go:356] "Generic (PLEG): container finished" podID="32d78414-e90e-4273-ba07-2a9c34ed1871" containerID="c33ba945c787a7d93de0178ec45729069d73adb4ff8f538a7bb378421453511a" exitCode=0 Oct 30 16:32:55.201328 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:55.201155 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a3cc1e2b-db877" event={"ID":"32d78414-e90e-4273-ba07-2a9c34ed1871","Type":"ContainerDied","Data":"c33ba945c787a7d93de0178ec45729069d73adb4ff8f538a7bb378421453511a"} Oct 30 16:32:55.643398 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:55.643361 4493 patch_prober.go:29] interesting pod/e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 30 16:32:55.643874 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:55.643417 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp" podUID="ffdf199f-692a-4542-a7cb-0f836829419e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 30 16:32:55.777407 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:55.777362 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-37964af0-j4tvw"] Oct 30 16:32:55.780886 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:55.780852 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-37964af0-j4tvw"] Oct 30 16:32:56.266694 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:56.266665 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f756b7bf-gldcf"] Oct 30 16:32:56.269253 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:56.269197 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f756b7bf-gldcf"] Oct 30 16:32:56.439908 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:56.439872 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a3cc1e2b-db877" Oct 30 16:32:56.468393 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:56.468366 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32d78414-e90e-4273-ba07-2a9c34ed1871-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32d78414-e90e-4273-ba07-2a9c34ed1871-maas-api-openshift-ca\") pod \"32d78414-e90e-4273-ba07-2a9c34ed1871\" (UID: \"32d78414-e90e-4273-ba07-2a9c34ed1871\") " Oct 30 16:32:56.468549 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:56.468423 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32d78414-e90e-4273-ba07-2a9c34ed1871-kube-api-access-qm288\" (UniqueName: \"kubernetes.io/projected/32d78414-e90e-4273-ba07-2a9c34ed1871-kube-api-access-qm288\") pod \"32d78414-e90e-4273-ba07-2a9c34ed1871\" (UID: \"32d78414-e90e-4273-ba07-2a9c34ed1871\") " Oct 30 16:32:56.468549 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:56.468480 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32d78414-e90e-4273-ba07-2a9c34ed1871-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32d78414-e90e-4273-ba07-2a9c34ed1871-maas-api-xks-ca\") pod \"32d78414-e90e-4273-ba07-2a9c34ed1871\" (UID: \"32d78414-e90e-4273-ba07-2a9c34ed1871\") " Oct 30 16:32:56.468737 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:56.468714 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32d78414-e90e-4273-ba07-2a9c34ed1871-maas-api-openshift-ca" pod "32d78414-e90e-4273-ba07-2a9c34ed1871" (UID: "32d78414-e90e-4273-ba07-2a9c34ed1871"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:32:56.468820 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:56.468805 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32d78414-e90e-4273-ba07-2a9c34ed1871-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:32:56.470540 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:56.470522 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32d78414-e90e-4273-ba07-2a9c34ed1871-maas-api-xks-ca" pod "32d78414-e90e-4273-ba07-2a9c34ed1871" (UID: "32d78414-e90e-4273-ba07-2a9c34ed1871"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:32:56.470604 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:56.470586 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32d78414-e90e-4273-ba07-2a9c34ed1871-kube-api-access-qm288" pod "32d78414-e90e-4273-ba07-2a9c34ed1871" (UID: "32d78414-e90e-4273-ba07-2a9c34ed1871"). InnerVolumeSpecName "kube-api-access-qm288". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:32:56.569972 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:56.569884 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qm288\" (UniqueName: \"kubernetes.io/projected/32d78414-e90e-4273-ba07-2a9c34ed1871-kube-api-access-qm288\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:32:56.569972 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:56.569914 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32d78414-e90e-4273-ba07-2a9c34ed1871-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:32:56.643497 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:56.643394 4493 patch_prober.go:29] interesting pod/e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 30 16:32:56.643497 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:56.643447 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp" podUID="ffdf199f-692a-4542-a7cb-0f836829419e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 30 16:32:56.939193 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:56.939150 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="164bd713-ddec-41fd-bc27-81b43e3509c2" path="/var/lib/kubelet/pods/164bd713-ddec-41fd-bc27-81b43e3509c2/volumes" Oct 30 16:32:56.939611 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:56.939597 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfb3b25e-8504-4b5e-9145-43eab49d0d18" path="/var/lib/kubelet/pods/cfb3b25e-8504-4b5e-9145-43eab49d0d18/volumes" Oct 30 16:32:56.976432 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:56.976409 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp" Oct 30 16:32:57.245374 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:57.242754 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a3cc1e2b-db877" event={"ID":"32d78414-e90e-4273-ba07-2a9c34ed1871","Type":"ContainerDied","Data":"de90a5aa497c18f0224de00d808e2782015526552c3d48cbc921d0a84ea19c25"} Oct 30 16:32:57.245374 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:57.242834 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de90a5aa497c18f0224de00d808e2782015526552c3d48cbc921d0a84ea19c25" Oct 30 16:32:57.245374 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:57.242906 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a3cc1e2b-db877" Oct 30 16:32:57.652396 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:57.650701 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp" Oct 30 16:32:57.652952 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:57.652521 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp" Oct 30 16:32:58.871652 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:58.871623 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 16:32:59.229227 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:59.229195 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cff59996-ktdqk"] Oct 30 16:32:59.240721 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:32:59.240696 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cff59996-ktdqk"] Oct 30 16:33:00.341379 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:33:00.341344 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 16:33:00.931729 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:00.931694 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91812800-a291-44cf-b47d-ee66eb4c2502" path="/var/lib/kubelet/pods/91812800-a291-44cf-b47d-ee66eb4c2502/volumes" Oct 30 16:33:03.871524 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:03.871477 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 16:33:04.281583 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:04.281538 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a3cc1e2b-5fbcc7fc48-64sc5"] Oct 30 16:33:04.305975 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:04.305939 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a3cc1e2b-77f9b8d87-d9qp8"] Oct 30 16:33:09.092255 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:09.092222 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-420ff3-7f7b8b4bdc-6d2x4"] Oct 30 16:33:09.092641 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:09.092629 4493 cpu_manager.go:414] "containerMap: removing container" podUID="32d78414-e90e-4273-ba07-2a9c34ed1871" containerName="revoke-keys" Oct 30 16:33:09.092684 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:09.092642 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32d78414-e90e-4273-ba07-2a9c34ed1871" containerName="revoke-keys" Oct 30 16:33:09.092734 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:09.092726 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="32d78414-e90e-4273-ba07-2a9c34ed1871" containerName="revoke-keys" Oct 30 16:33:09.096024 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:09.096006 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-420ff3-7f7b8b4bdc-6d2x4" Oct 30 16:33:09.125368 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:09.125336 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-420ff3-dockercfg-nsdsq\"" Oct 30 16:33:09.126201 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:09.126178 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-420ff3-7f7b8b4bdc-6d2x4"] Oct 30 16:33:09.154941 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:09.154844 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-420ff3\"" Oct 30 16:33:09.198095 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:09.198058 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw2vg\" (UniqueName: \"kubernetes.io/projected/a6fef0c5-c804-4953-a1be-9b49291e63d5-kube-api-access-fw2vg\") pod \"payload-pre-processing-e2e-worker-w4-420ff3-7f7b8b4bdc-6d2x4\" (UID: \"a6fef0c5-c804-4953-a1be-9b49291e63d5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-420ff3-7f7b8b4bdc-6d2x4" Oct 30 16:33:09.198266 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:09.198199 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6fef0c5-c804-4953-a1be-9b49291e63d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-420ff3-7f7b8b4bdc-6d2x4\" (UID: \"a6fef0c5-c804-4953-a1be-9b49291e63d5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-420ff3-7f7b8b4bdc-6d2x4" Oct 30 16:33:09.282282 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:09.282240 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-a3cc1e2b-5fbcc7fc48-64sc5" podUID="cb3ac4c7-44b5-47c0-9e6c-2dd0c35a36c9" containerName="payload-processing" containerID="cri-o://c1fa0af1d095b2a215272f2aca9dbc5c5eb45f2329c2d1c4f6ce6d5748ce223e" gracePeriod=25 Oct 30 16:33:09.299078 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:09.299053 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6fef0c5-c804-4953-a1be-9b49291e63d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-420ff3-7f7b8b4bdc-6d2x4\" (UID: \"a6fef0c5-c804-4953-a1be-9b49291e63d5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-420ff3-7f7b8b4bdc-6d2x4" Oct 30 16:33:09.299165 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:09.299107 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw2vg\" (UniqueName: \"kubernetes.io/projected/a6fef0c5-c804-4953-a1be-9b49291e63d5-kube-api-access-fw2vg\") pod \"payload-pre-processing-e2e-worker-w4-420ff3-7f7b8b4bdc-6d2x4\" (UID: \"a6fef0c5-c804-4953-a1be-9b49291e63d5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-420ff3-7f7b8b4bdc-6d2x4" Oct 30 16:33:09.299932 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:09.299913 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6fef0c5-c804-4953-a1be-9b49291e63d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-420ff3-7f7b8b4bdc-6d2x4\" (UID: \"a6fef0c5-c804-4953-a1be-9b49291e63d5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-420ff3-7f7b8b4bdc-6d2x4" Oct 30 16:33:09.306262 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:09.306234 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-a3cc1e2b-77f9b8d87-d9qp8" podUID="97a5f2a3-73ee-4aa9-ae14-2a5b98f56143" containerName="payload-pre-processing" containerID="cri-o://c55e51f13d640fef8bb1797dd25071c19273ce76d3115f748ccf89380098b235" gracePeriod=25 Oct 30 16:33:09.322030 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:09.322005 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw2vg\" (UniqueName: \"kubernetes.io/projected/a6fef0c5-c804-4953-a1be-9b49291e63d5-kube-api-access-fw2vg\") pod \"payload-pre-processing-e2e-worker-w4-420ff3-7f7b8b4bdc-6d2x4\" (UID: \"a6fef0c5-c804-4953-a1be-9b49291e63d5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-420ff3-7f7b8b4bdc-6d2x4" Oct 30 16:33:09.409478 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:09.409437 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-420ff3-7f7b8b4bdc-6d2x4" Oct 30 16:33:09.628225 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:09.628198 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-a3cc1e2b-5fbcc7fc48-64sc5" Oct 30 16:33:09.638767 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:09.638745 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a3cc1e2b-77f9b8d87-d9qp8" Oct 30 16:33:09.704253 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:09.704223 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97a5f2a3-73ee-4aa9-ae14-2a5b98f56143-kube-api-access-7pj8z\" (UniqueName: \"kubernetes.io/projected/97a5f2a3-73ee-4aa9-ae14-2a5b98f56143-kube-api-access-7pj8z\") pod \"97a5f2a3-73ee-4aa9-ae14-2a5b98f56143\" (UID: \"97a5f2a3-73ee-4aa9-ae14-2a5b98f56143\") " Oct 30 16:33:09.704440 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:09.704314 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb3ac4c7-44b5-47c0-9e6c-2dd0c35a36c9-kube-api-access-d6vc7\" (UniqueName: \"kubernetes.io/projected/cb3ac4c7-44b5-47c0-9e6c-2dd0c35a36c9-kube-api-access-d6vc7\") pod \"cb3ac4c7-44b5-47c0-9e6c-2dd0c35a36c9\" (UID: \"cb3ac4c7-44b5-47c0-9e6c-2dd0c35a36c9\") " Oct 30 16:33:09.704440 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:09.704370 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97a5f2a3-73ee-4aa9-ae14-2a5b98f56143-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97a5f2a3-73ee-4aa9-ae14-2a5b98f56143-plugins-config-volume\") pod \"97a5f2a3-73ee-4aa9-ae14-2a5b98f56143\" (UID: \"97a5f2a3-73ee-4aa9-ae14-2a5b98f56143\") " Oct 30 16:33:09.704552 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:09.704440 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb3ac4c7-44b5-47c0-9e6c-2dd0c35a36c9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb3ac4c7-44b5-47c0-9e6c-2dd0c35a36c9-plugins-config-volume\") pod \"cb3ac4c7-44b5-47c0-9e6c-2dd0c35a36c9\" (UID: \"cb3ac4c7-44b5-47c0-9e6c-2dd0c35a36c9\") " Oct 30 16:33:09.704918 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:09.704859 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97a5f2a3-73ee-4aa9-ae14-2a5b98f56143-plugins-config-volume" pod "97a5f2a3-73ee-4aa9-ae14-2a5b98f56143" (UID: "97a5f2a3-73ee-4aa9-ae14-2a5b98f56143"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:33:09.705232 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:09.705177 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb3ac4c7-44b5-47c0-9e6c-2dd0c35a36c9-plugins-config-volume" pod "cb3ac4c7-44b5-47c0-9e6c-2dd0c35a36c9" (UID: "cb3ac4c7-44b5-47c0-9e6c-2dd0c35a36c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:33:09.711548 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:09.707561 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb3ac4c7-44b5-47c0-9e6c-2dd0c35a36c9-kube-api-access-d6vc7" pod "cb3ac4c7-44b5-47c0-9e6c-2dd0c35a36c9" (UID: "cb3ac4c7-44b5-47c0-9e6c-2dd0c35a36c9"). InnerVolumeSpecName "kube-api-access-d6vc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:33:09.711548 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:09.707578 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97a5f2a3-73ee-4aa9-ae14-2a5b98f56143-kube-api-access-7pj8z" pod "97a5f2a3-73ee-4aa9-ae14-2a5b98f56143" (UID: "97a5f2a3-73ee-4aa9-ae14-2a5b98f56143"). InnerVolumeSpecName "kube-api-access-7pj8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:33:09.723928 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:09.723840 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-420ff3-7f7b8b4bdc-6d2x4"] Oct 30 16:33:09.727157 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:33:09.727110 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6fef0c5_c804_4953_a1be_9b49291e63d5.slice/crio-26f54ce07c633b7a62471943ae59b69a1d6c89633a68a527edf8cab623a0a342 WatchSource:0}: Error finding container 26f54ce07c633b7a62471943ae59b69a1d6c89633a68a527edf8cab623a0a342: Status 404 returned error can't find the container with id 26f54ce07c633b7a62471943ae59b69a1d6c89633a68a527edf8cab623a0a342 Oct 30 16:33:09.805699 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:09.805656 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pj8z\" (UniqueName: \"kubernetes.io/projected/97a5f2a3-73ee-4aa9-ae14-2a5b98f56143-kube-api-access-7pj8z\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:33:09.805699 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:09.805692 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6vc7\" (UniqueName: \"kubernetes.io/projected/cb3ac4c7-44b5-47c0-9e6c-2dd0c35a36c9-kube-api-access-d6vc7\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:33:09.805922 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:09.805706 4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97a5f2a3-73ee-4aa9-ae14-2a5b98f56143-plugins-config-volume\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:33:09.805922 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:09.805719 4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb3ac4c7-44b5-47c0-9e6c-2dd0c35a36c9-plugins-config-volume\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:33:10.223980 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:10.223953 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a3cc1e2b-db877"] Oct 30 16:33:10.227206 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:10.227184 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a3cc1e2b-db877"] Oct 30 16:33:10.316152 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:10.316120 4493 generic.go:356] "Generic (PLEG): container finished" podID="97a5f2a3-73ee-4aa9-ae14-2a5b98f56143" containerID="c55e51f13d640fef8bb1797dd25071c19273ce76d3115f748ccf89380098b235" exitCode=0 Oct 30 16:33:10.316314 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:10.316179 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a3cc1e2b-77f9b8d87-d9qp8" Oct 30 16:33:10.316314 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:10.316193 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a3cc1e2b-77f9b8d87-d9qp8" event={"ID":"97a5f2a3-73ee-4aa9-ae14-2a5b98f56143","Type":"ContainerDied","Data":"c55e51f13d640fef8bb1797dd25071c19273ce76d3115f748ccf89380098b235"} Oct 30 16:33:10.316314 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:10.316259 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a3cc1e2b-77f9b8d87-d9qp8" event={"ID":"97a5f2a3-73ee-4aa9-ae14-2a5b98f56143","Type":"ContainerDied","Data":"077aae0faa22e5e138f1f0a2d26bdf4ffc401550c72c082bda6bf1b221480c97"} Oct 30 16:33:10.316314 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:10.316293 4493 scope.go:122] "RemoveContainer" containerID="c55e51f13d640fef8bb1797dd25071c19273ce76d3115f748ccf89380098b235" Oct 30 16:33:10.318117 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:10.318089 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-420ff3-7f7b8b4bdc-6d2x4" event={"ID":"a6fef0c5-c804-4953-a1be-9b49291e63d5","Type":"ContainerStarted","Data":"f7ecc2170b4bb59f45b7186085f7a50f1981ea051891d3303e993c09cff69d3a"} Oct 30 16:33:10.318228 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:10.318138 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-420ff3-7f7b8b4bdc-6d2x4" event={"ID":"a6fef0c5-c804-4953-a1be-9b49291e63d5","Type":"ContainerStarted","Data":"26f54ce07c633b7a62471943ae59b69a1d6c89633a68a527edf8cab623a0a342"} Oct 30 16:33:10.318287 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:10.318270 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-420ff3-7f7b8b4bdc-6d2x4" Oct 30 16:33:10.319859 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:10.319835 4493 generic.go:356] "Generic (PLEG): container finished" podID="cb3ac4c7-44b5-47c0-9e6c-2dd0c35a36c9" containerID="c1fa0af1d095b2a215272f2aca9dbc5c5eb45f2329c2d1c4f6ce6d5748ce223e" exitCode=0 Oct 30 16:33:10.319979 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:10.319884 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-a3cc1e2b-5fbcc7fc48-64sc5" Oct 30 16:33:10.319979 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:10.319911 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-a3cc1e2b-5fbcc7fc48-64sc5" event={"ID":"cb3ac4c7-44b5-47c0-9e6c-2dd0c35a36c9","Type":"ContainerDied","Data":"c1fa0af1d095b2a215272f2aca9dbc5c5eb45f2329c2d1c4f6ce6d5748ce223e"} Oct 30 16:33:10.319979 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:10.319948 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-a3cc1e2b-5fbcc7fc48-64sc5" event={"ID":"cb3ac4c7-44b5-47c0-9e6c-2dd0c35a36c9","Type":"ContainerDied","Data":"c5708cd98bd777858d5b54091a50d0a2451e5d2ba2d4e177213f745b3791a8e3"} Oct 30 16:33:10.328751 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:10.328732 4493 scope.go:122] "RemoveContainer" containerID="c55e51f13d640fef8bb1797dd25071c19273ce76d3115f748ccf89380098b235" Oct 30 16:33:10.329127 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:33:10.329101 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c55e51f13d640fef8bb1797dd25071c19273ce76d3115f748ccf89380098b235\": container with ID starting with c55e51f13d640fef8bb1797dd25071c19273ce76d3115f748ccf89380098b235 not found: ID does not exist" containerID="c55e51f13d640fef8bb1797dd25071c19273ce76d3115f748ccf89380098b235" Oct 30 16:33:10.329216 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:10.329136 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c55e51f13d640fef8bb1797dd25071c19273ce76d3115f748ccf89380098b235"} err="failed to get container status \"c55e51f13d640fef8bb1797dd25071c19273ce76d3115f748ccf89380098b235\": rpc error: code = NotFound desc = could not find container \"c55e51f13d640fef8bb1797dd25071c19273ce76d3115f748ccf89380098b235\": container with ID starting with c55e51f13d640fef8bb1797dd25071c19273ce76d3115f748ccf89380098b235 not found: ID does not exist" Oct 30 16:33:10.329216 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:10.329157 4493 scope.go:122] "RemoveContainer" containerID="c1fa0af1d095b2a215272f2aca9dbc5c5eb45f2329c2d1c4f6ce6d5748ce223e" Oct 30 16:33:10.355494 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:10.355444 4493 scope.go:122] "RemoveContainer" containerID="c1fa0af1d095b2a215272f2aca9dbc5c5eb45f2329c2d1c4f6ce6d5748ce223e" Oct 30 16:33:10.356145 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:33:10.356023 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1fa0af1d095b2a215272f2aca9dbc5c5eb45f2329c2d1c4f6ce6d5748ce223e\": container with ID starting with c1fa0af1d095b2a215272f2aca9dbc5c5eb45f2329c2d1c4f6ce6d5748ce223e not found: ID does not exist" containerID="c1fa0af1d095b2a215272f2aca9dbc5c5eb45f2329c2d1c4f6ce6d5748ce223e" Oct 30 16:33:10.356145 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:10.356063 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1fa0af1d095b2a215272f2aca9dbc5c5eb45f2329c2d1c4f6ce6d5748ce223e"} err="failed to get container status \"c1fa0af1d095b2a215272f2aca9dbc5c5eb45f2329c2d1c4f6ce6d5748ce223e\": rpc error: code = NotFound desc = could not find container \"c1fa0af1d095b2a215272f2aca9dbc5c5eb45f2329c2d1c4f6ce6d5748ce223e\": container with ID starting with c1fa0af1d095b2a215272f2aca9dbc5c5eb45f2329c2d1c4f6ce6d5748ce223e not found: ID does not exist" Oct 30 16:33:10.361487 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:10.361464 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp"] Oct 30 16:33:10.361758 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:10.361735 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp" podUID="ffdf199f-692a-4542-a7cb-0f836829419e" containerName="istio-proxy" containerID="cri-o://c982ff5d85ebb59769f2c48fd066136171c6a653227fb753de331c28d401473f" gracePeriod=30 Oct 30 16:33:10.376033 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:10.375980 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-420ff3-7f7b8b4bdc-6d2x4" podStartSLOduration=1.375963446 podStartE2EDuration="1.375963446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:33:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:33:10.37363966 +0000 UTC m=+617.600814760" watchObservedRunningTime="2026-10-30 16:33:10.375963446 +0000 UTC m=+617.603138548" Oct 30 16:33:10.425026 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:10.424997 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a3cc1e2b-77f9b8d87-d9qp8"] Oct 30 16:33:10.429304 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:10.429284 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a3cc1e2b-77f9b8d87-d9qp8"] Oct 30 16:33:10.510058 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:10.509955 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a3cc1e2b-5fbcc7fc48-64sc5"] Oct 30 16:33:10.513581 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:10.513556 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a3cc1e2b-5fbcc7fc48-64sc5"] Oct 30 16:33:10.931888 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:10.931857 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32d78414-e90e-4273-ba07-2a9c34ed1871" path="/var/lib/kubelet/pods/32d78414-e90e-4273-ba07-2a9c34ed1871/volumes" Oct 30 16:33:10.932203 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:10.932191 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97a5f2a3-73ee-4aa9-ae14-2a5b98f56143" path="/var/lib/kubelet/pods/97a5f2a3-73ee-4aa9-ae14-2a5b98f56143/volumes" Oct 30 16:33:10.932491 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:10.932481 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb3ac4c7-44b5-47c0-9e6c-2dd0c35a36c9" path="/var/lib/kubelet/pods/cb3ac4c7-44b5-47c0-9e6c-2dd0c35a36c9/volumes" Oct 30 16:33:11.604367 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:11.604336 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2"] Oct 30 16:33:11.604921 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:11.604901 4493 cpu_manager.go:414] "containerMap: removing container" podUID="97a5f2a3-73ee-4aa9-ae14-2a5b98f56143" containerName="payload-pre-processing" Oct 30 16:33:11.605008 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:11.604924 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97a5f2a3-73ee-4aa9-ae14-2a5b98f56143" containerName="payload-pre-processing" Oct 30 16:33:11.605008 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:11.604976 4493 cpu_manager.go:414] "containerMap: removing container" podUID="cb3ac4c7-44b5-47c0-9e6c-2dd0c35a36c9" containerName="payload-processing" Oct 30 16:33:11.605008 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:11.604984 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb3ac4c7-44b5-47c0-9e6c-2dd0c35a36c9" containerName="payload-processing" Oct 30 16:33:11.605159 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:11.605112 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb3ac4c7-44b5-47c0-9e6c-2dd0c35a36c9" containerName="payload-processing" Oct 30 16:33:11.605159 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:11.605128 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="97a5f2a3-73ee-4aa9-ae14-2a5b98f56143" containerName="payload-pre-processing" Oct 30 16:33:11.610279 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:11.610257 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2" Oct 30 16:33:11.627689 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:11.627660 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-a77b48-openshift-default-dockercfg-725dg\"" Oct 30 16:33:11.640733 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:11.640699 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2"] Oct 30 16:33:11.723904 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:11.723866 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q99cb\" (UniqueName: \"kubernetes.io/projected/c7ae7018-798b-4a3a-afef-b2ed1a13b05f-kube-api-access-q99cb\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2\" (UID: \"c7ae7018-798b-4a3a-afef-b2ed1a13b05f\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2" Oct 30 16:33:11.724113 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:11.723929 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7ae7018-798b-4a3a-afef-b2ed1a13b05f-credential-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2\" (UID: \"c7ae7018-798b-4a3a-afef-b2ed1a13b05f\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2" Oct 30 16:33:11.724113 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:11.723977 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7ae7018-798b-4a3a-afef-b2ed1a13b05f-workload-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2\" (UID: \"c7ae7018-798b-4a3a-afef-b2ed1a13b05f\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2" Oct 30 16:33:11.724113 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:11.724003 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7ae7018-798b-4a3a-afef-b2ed1a13b05f-workload-certs\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2\" (UID: \"c7ae7018-798b-4a3a-afef-b2ed1a13b05f\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2" Oct 30 16:33:11.724113 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:11.724037 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7ae7018-798b-4a3a-afef-b2ed1a13b05f-istio-data\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2\" (UID: \"c7ae7018-798b-4a3a-afef-b2ed1a13b05f\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2" Oct 30 16:33:11.724113 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:11.724078 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7ae7018-798b-4a3a-afef-b2ed1a13b05f-istio-envoy\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2\" (UID: \"c7ae7018-798b-4a3a-afef-b2ed1a13b05f\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2" Oct 30 16:33:11.724444 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:11.724170 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7ae7018-798b-4a3a-afef-b2ed1a13b05f-istio-podinfo\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2\" (UID: \"c7ae7018-798b-4a3a-afef-b2ed1a13b05f\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2" Oct 30 16:33:11.724444 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:11.724207 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7ae7018-798b-4a3a-afef-b2ed1a13b05f-istio-token\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2\" (UID: \"c7ae7018-798b-4a3a-afef-b2ed1a13b05f\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2" Oct 30 16:33:11.724444 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:11.724246 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7ae7018-798b-4a3a-afef-b2ed1a13b05f-istiod-ca-cert\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2\" (UID: \"c7ae7018-798b-4a3a-afef-b2ed1a13b05f\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2" Oct 30 16:33:11.825052 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:11.825013 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q99cb\" (UniqueName: \"kubernetes.io/projected/c7ae7018-798b-4a3a-afef-b2ed1a13b05f-kube-api-access-q99cb\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2\" (UID: \"c7ae7018-798b-4a3a-afef-b2ed1a13b05f\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2" Oct 30 16:33:11.825242 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:11.825065 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7ae7018-798b-4a3a-afef-b2ed1a13b05f-credential-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2\" (UID: \"c7ae7018-798b-4a3a-afef-b2ed1a13b05f\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2" Oct 30 16:33:11.825242 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:11.825106 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7ae7018-798b-4a3a-afef-b2ed1a13b05f-workload-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2\" (UID: \"c7ae7018-798b-4a3a-afef-b2ed1a13b05f\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2" Oct 30 16:33:11.825242 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:11.825129 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7ae7018-798b-4a3a-afef-b2ed1a13b05f-workload-certs\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2\" (UID: \"c7ae7018-798b-4a3a-afef-b2ed1a13b05f\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2" Oct 30 16:33:11.825242 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:11.825151 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7ae7018-798b-4a3a-afef-b2ed1a13b05f-istio-data\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2\" (UID: \"c7ae7018-798b-4a3a-afef-b2ed1a13b05f\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2" Oct 30 16:33:11.825242 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:11.825187 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7ae7018-798b-4a3a-afef-b2ed1a13b05f-istio-envoy\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2\" (UID: \"c7ae7018-798b-4a3a-afef-b2ed1a13b05f\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2" Oct 30 16:33:11.825511 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:11.825269 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7ae7018-798b-4a3a-afef-b2ed1a13b05f-istio-podinfo\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2\" (UID: \"c7ae7018-798b-4a3a-afef-b2ed1a13b05f\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2" Oct 30 16:33:11.825511 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:11.825303 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7ae7018-798b-4a3a-afef-b2ed1a13b05f-istio-token\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2\" (UID: \"c7ae7018-798b-4a3a-afef-b2ed1a13b05f\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2" Oct 30 16:33:11.825511 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:11.825343 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7ae7018-798b-4a3a-afef-b2ed1a13b05f-istiod-ca-cert\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2\" (UID: \"c7ae7018-798b-4a3a-afef-b2ed1a13b05f\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2" Oct 30 16:33:11.825668 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:11.825530 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7ae7018-798b-4a3a-afef-b2ed1a13b05f-credential-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2\" (UID: \"c7ae7018-798b-4a3a-afef-b2ed1a13b05f\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2" Oct 30 16:33:11.825668 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:11.825566 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7ae7018-798b-4a3a-afef-b2ed1a13b05f-istio-data\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2\" (UID: \"c7ae7018-798b-4a3a-afef-b2ed1a13b05f\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2" Oct 30 16:33:11.825668 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:11.825599 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7ae7018-798b-4a3a-afef-b2ed1a13b05f-workload-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2\" (UID: \"c7ae7018-798b-4a3a-afef-b2ed1a13b05f\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2" Oct 30 16:33:11.825873 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:11.825852 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7ae7018-798b-4a3a-afef-b2ed1a13b05f-workload-certs\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2\" (UID: \"c7ae7018-798b-4a3a-afef-b2ed1a13b05f\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2" Oct 30 16:33:11.825995 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:11.825969 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7ae7018-798b-4a3a-afef-b2ed1a13b05f-istiod-ca-cert\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2\" (UID: \"c7ae7018-798b-4a3a-afef-b2ed1a13b05f\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2" Oct 30 16:33:11.827825 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:11.827800 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7ae7018-798b-4a3a-afef-b2ed1a13b05f-istio-envoy\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2\" (UID: \"c7ae7018-798b-4a3a-afef-b2ed1a13b05f\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2" Oct 30 16:33:11.827972 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:11.827955 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7ae7018-798b-4a3a-afef-b2ed1a13b05f-istio-podinfo\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2\" (UID: \"c7ae7018-798b-4a3a-afef-b2ed1a13b05f\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2" Oct 30 16:33:11.849232 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:11.849209 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7ae7018-798b-4a3a-afef-b2ed1a13b05f-istio-token\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2\" (UID: \"c7ae7018-798b-4a3a-afef-b2ed1a13b05f\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2" Oct 30 16:33:11.849752 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:11.849727 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q99cb\" (UniqueName: \"kubernetes.io/projected/c7ae7018-798b-4a3a-afef-b2ed1a13b05f-kube-api-access-q99cb\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2\" (UID: \"c7ae7018-798b-4a3a-afef-b2ed1a13b05f\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2" Oct 30 16:33:11.926740 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:11.926706 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2" Oct 30 16:33:12.254683 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:12.254603 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2"] Oct 30 16:33:12.257701 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:33:12.257673 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7ae7018_798b_4a3a_afef_b2ed1a13b05f.slice/crio-de9909497d0546b2a0f5f881fde3cdc0d52397ce9b48048cac6f9586467bd377 WatchSource:0}: Error finding container de9909497d0546b2a0f5f881fde3cdc0d52397ce9b48048cac6f9586467bd377: Status 404 returned error can't find the container with id de9909497d0546b2a0f5f881fde3cdc0d52397ce9b48048cac6f9586467bd377 Oct 30 16:33:12.259998 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:12.259968 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:33:12.260052 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:12.260030 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:33:12.260085 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:12.260059 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:33:12.337469 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:12.337436 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2" event={"ID":"c7ae7018-798b-4a3a-afef-b2ed1a13b05f","Type":"ContainerStarted","Data":"2b851adaac88aed5e816a099ff1adcf586423cae5811e65fa657dd00ed2b18ae"} Oct 30 16:33:12.337588 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:12.337497 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2" event={"ID":"c7ae7018-798b-4a3a-afef-b2ed1a13b05f","Type":"ContainerStarted","Data":"de9909497d0546b2a0f5f881fde3cdc0d52397ce9b48048cac6f9586467bd377"} Oct 30 16:33:12.384442 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:12.384386 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2" podStartSLOduration=1.384366572 podStartE2EDuration="1.384366572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:33:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:33:12.382336681 +0000 UTC m=+619.609511781" watchObservedRunningTime="2026-10-30 16:33:12.384366572 +0000 UTC m=+619.611541677" Oct 30 16:33:12.653900 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:12.653858 4493 patch_prober.go:29] interesting pod/e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 16:33:12.654281 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:12.653927 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp" podUID="ffdf199f-692a-4542-a7cb-0f836829419e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 16:33:12.932432 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:12.932366 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2" Oct 30 16:33:13.160178 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:13.160143 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 16:33:13.934304 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:13.934266 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2" Oct 30 16:33:14.351456 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:14.351372 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2" Oct 30 16:33:14.352375 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:14.352350 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2" Oct 30 16:33:15.615093 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:15.615059 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp" Oct 30 16:33:15.662646 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:15.662615 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffdf199f-692a-4542-a7cb-0f836829419e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffdf199f-692a-4542-a7cb-0f836829419e-istio-envoy\") pod \"ffdf199f-692a-4542-a7cb-0f836829419e\" (UID: \"ffdf199f-692a-4542-a7cb-0f836829419e\") " Oct 30 16:33:15.662816 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:15.662658 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffdf199f-692a-4542-a7cb-0f836829419e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffdf199f-692a-4542-a7cb-0f836829419e-istiod-ca-cert\") pod \"ffdf199f-692a-4542-a7cb-0f836829419e\" (UID: \"ffdf199f-692a-4542-a7cb-0f836829419e\") " Oct 30 16:33:15.662816 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:15.662682 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffdf199f-692a-4542-a7cb-0f836829419e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffdf199f-692a-4542-a7cb-0f836829419e-workload-socket\") pod \"ffdf199f-692a-4542-a7cb-0f836829419e\" (UID: \"ffdf199f-692a-4542-a7cb-0f836829419e\") " Oct 30 16:33:15.662816 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:15.662701 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffdf199f-692a-4542-a7cb-0f836829419e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffdf199f-692a-4542-a7cb-0f836829419e-istio-data\") pod \"ffdf199f-692a-4542-a7cb-0f836829419e\" (UID: \"ffdf199f-692a-4542-a7cb-0f836829419e\") " Oct 30 16:33:15.662965 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:15.662841 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffdf199f-692a-4542-a7cb-0f836829419e-kube-api-access-8vdwc\" (UniqueName: \"kubernetes.io/projected/ffdf199f-692a-4542-a7cb-0f836829419e-kube-api-access-8vdwc\") pod \"ffdf199f-692a-4542-a7cb-0f836829419e\" (UID: \"ffdf199f-692a-4542-a7cb-0f836829419e\") " Oct 30 16:33:15.663020 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:15.662962 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffdf199f-692a-4542-a7cb-0f836829419e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffdf199f-692a-4542-a7cb-0f836829419e-credential-socket\") pod \"ffdf199f-692a-4542-a7cb-0f836829419e\" (UID: \"ffdf199f-692a-4542-a7cb-0f836829419e\") " Oct 30 16:33:15.663020 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:15.662992 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffdf199f-692a-4542-a7cb-0f836829419e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffdf199f-692a-4542-a7cb-0f836829419e-workload-certs\") pod \"ffdf199f-692a-4542-a7cb-0f836829419e\" (UID: \"ffdf199f-692a-4542-a7cb-0f836829419e\") " Oct 30 16:33:15.663122 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:15.662969 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffdf199f-692a-4542-a7cb-0f836829419e-workload-socket" pod "ffdf199f-692a-4542-a7cb-0f836829419e" (UID: "ffdf199f-692a-4542-a7cb-0f836829419e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:33:15.663122 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:15.663028 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffdf199f-692a-4542-a7cb-0f836829419e-istio-data" pod "ffdf199f-692a-4542-a7cb-0f836829419e" (UID: "ffdf199f-692a-4542-a7cb-0f836829419e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:33:15.663122 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:15.663047 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffdf199f-692a-4542-a7cb-0f836829419e-istio-token\" (UniqueName: \"kubernetes.io/projected/ffdf199f-692a-4542-a7cb-0f836829419e-istio-token\") pod \"ffdf199f-692a-4542-a7cb-0f836829419e\" (UID: \"ffdf199f-692a-4542-a7cb-0f836829419e\") " Oct 30 16:33:15.663284 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:15.663152 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ffdf199f-692a-4542-a7cb-0f836829419e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffdf199f-692a-4542-a7cb-0f836829419e-istio-podinfo\") pod \"ffdf199f-692a-4542-a7cb-0f836829419e\" (UID: \"ffdf199f-692a-4542-a7cb-0f836829419e\") " Oct 30 16:33:15.663284 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:15.663202 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffdf199f-692a-4542-a7cb-0f836829419e-istiod-ca-cert" pod "ffdf199f-692a-4542-a7cb-0f836829419e" (UID: "ffdf199f-692a-4542-a7cb-0f836829419e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:33:15.663284 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:15.663217 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffdf199f-692a-4542-a7cb-0f836829419e-credential-socket" pod "ffdf199f-692a-4542-a7cb-0f836829419e" (UID: "ffdf199f-692a-4542-a7cb-0f836829419e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:33:15.663284 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:15.663231 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffdf199f-692a-4542-a7cb-0f836829419e-workload-certs" pod "ffdf199f-692a-4542-a7cb-0f836829419e" (UID: "ffdf199f-692a-4542-a7cb-0f836829419e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:33:15.663827 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:15.663599 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffdf199f-692a-4542-a7cb-0f836829419e-istiod-ca-cert\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:33:15.663827 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:15.663624 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffdf199f-692a-4542-a7cb-0f836829419e-workload-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:33:15.663827 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:15.663639 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffdf199f-692a-4542-a7cb-0f836829419e-istio-data\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:33:15.663827 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:15.663653 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffdf199f-692a-4542-a7cb-0f836829419e-credential-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:33:15.663827 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:15.663665 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffdf199f-692a-4542-a7cb-0f836829419e-workload-certs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:33:15.665039 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:15.665020 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffdf199f-692a-4542-a7cb-0f836829419e-istio-envoy" pod "ffdf199f-692a-4542-a7cb-0f836829419e" (UID: "ffdf199f-692a-4542-a7cb-0f836829419e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:33:15.665295 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:15.665188 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ffdf199f-692a-4542-a7cb-0f836829419e-istio-podinfo" pod "ffdf199f-692a-4542-a7cb-0f836829419e" (UID: "ffdf199f-692a-4542-a7cb-0f836829419e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:33:15.665295 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:15.665238 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffdf199f-692a-4542-a7cb-0f836829419e-kube-api-access-8vdwc" pod "ffdf199f-692a-4542-a7cb-0f836829419e" (UID: "ffdf199f-692a-4542-a7cb-0f836829419e"). InnerVolumeSpecName "kube-api-access-8vdwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:33:15.665295 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:15.665265 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffdf199f-692a-4542-a7cb-0f836829419e-istio-token" pod "ffdf199f-692a-4542-a7cb-0f836829419e" (UID: "ffdf199f-692a-4542-a7cb-0f836829419e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:33:15.764638 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:15.764598 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vdwc\" (UniqueName: \"kubernetes.io/projected/ffdf199f-692a-4542-a7cb-0f836829419e-kube-api-access-8vdwc\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:33:15.764638 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:15.764631 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffdf199f-692a-4542-a7cb-0f836829419e-istio-token\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:33:15.764638 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:15.764645 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffdf199f-692a-4542-a7cb-0f836829419e-istio-podinfo\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:33:15.764922 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:15.764658 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffdf199f-692a-4542-a7cb-0f836829419e-istio-envoy\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:33:16.363597 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:16.363558 4493 generic.go:356] "Generic (PLEG): container finished" podID="ffdf199f-692a-4542-a7cb-0f836829419e" containerID="c982ff5d85ebb59769f2c48fd066136171c6a653227fb753de331c28d401473f" exitCode=0 Oct 30 16:33:16.363597 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:16.363600 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp" Oct 30 16:33:16.363877 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:16.363641 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp" event={"ID":"ffdf199f-692a-4542-a7cb-0f836829419e","Type":"ContainerDied","Data":"c982ff5d85ebb59769f2c48fd066136171c6a653227fb753de331c28d401473f"} Oct 30 16:33:16.363877 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:16.363689 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp" event={"ID":"ffdf199f-692a-4542-a7cb-0f836829419e","Type":"ContainerDied","Data":"39eeb86e0d6dc7b34ac11a04f8b7768ad5487b288276d90c54de17be83495437"} Oct 30 16:33:16.363877 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:16.363711 4493 scope.go:122] "RemoveContainer" containerID="c982ff5d85ebb59769f2c48fd066136171c6a653227fb753de331c28d401473f" Oct 30 16:33:16.373974 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:16.373954 4493 scope.go:122] "RemoveContainer" containerID="897182655e46fe593dadb0d8537894485982d15adefb5d58d02bbc7b200c1f0b" Oct 30 16:33:16.382487 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:16.382470 4493 scope.go:122] "RemoveContainer" containerID="c982ff5d85ebb59769f2c48fd066136171c6a653227fb753de331c28d401473f" Oct 30 16:33:16.382756 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:33:16.382737 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c982ff5d85ebb59769f2c48fd066136171c6a653227fb753de331c28d401473f\": container with ID starting with c982ff5d85ebb59769f2c48fd066136171c6a653227fb753de331c28d401473f not found: ID does not exist" containerID="c982ff5d85ebb59769f2c48fd066136171c6a653227fb753de331c28d401473f" Oct 30 16:33:16.382846 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:16.382765 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c982ff5d85ebb59769f2c48fd066136171c6a653227fb753de331c28d401473f"} err="failed to get container status \"c982ff5d85ebb59769f2c48fd066136171c6a653227fb753de331c28d401473f\": rpc error: code = NotFound desc = could not find container \"c982ff5d85ebb59769f2c48fd066136171c6a653227fb753de331c28d401473f\": container with ID starting with c982ff5d85ebb59769f2c48fd066136171c6a653227fb753de331c28d401473f not found: ID does not exist" Oct 30 16:33:16.382846 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:16.382798 4493 scope.go:122] "RemoveContainer" containerID="897182655e46fe593dadb0d8537894485982d15adefb5d58d02bbc7b200c1f0b" Oct 30 16:33:16.383028 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:33:16.383013 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"897182655e46fe593dadb0d8537894485982d15adefb5d58d02bbc7b200c1f0b\": container with ID starting with 897182655e46fe593dadb0d8537894485982d15adefb5d58d02bbc7b200c1f0b not found: ID does not exist" containerID="897182655e46fe593dadb0d8537894485982d15adefb5d58d02bbc7b200c1f0b" Oct 30 16:33:16.383075 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:16.383032 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"897182655e46fe593dadb0d8537894485982d15adefb5d58d02bbc7b200c1f0b"} err="failed to get container status \"897182655e46fe593dadb0d8537894485982d15adefb5d58d02bbc7b200c1f0b\": rpc error: code = NotFound desc = could not find container \"897182655e46fe593dadb0d8537894485982d15adefb5d58d02bbc7b200c1f0b\": container with ID starting with 897182655e46fe593dadb0d8537894485982d15adefb5d58d02bbc7b200c1f0b not found: ID does not exist" Oct 30 16:33:16.429399 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:16.429369 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp"] Oct 30 16:33:16.442463 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:16.442432 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-a3cc1e2b-openshift-default-6c4d59ffbf-j2hhp"] Oct 30 16:33:16.898640 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:16.898610 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-a77b48-699d989875-5w52s"] Oct 30 16:33:16.899069 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:16.899032 4493 cpu_manager.go:414] "containerMap: removing container" podUID="ffdf199f-692a-4542-a7cb-0f836829419e" containerName="istio-proxy" Oct 30 16:33:16.899069 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:16.899044 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffdf199f-692a-4542-a7cb-0f836829419e" containerName="istio-proxy" Oct 30 16:33:16.899150 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:16.899114 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffdf199f-692a-4542-a7cb-0f836829419e" containerName="istio-proxy" Oct 30 16:33:16.903212 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:16.903189 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a77b48-699d989875-5w52s" Oct 30 16:33:16.918724 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:16.918702 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-a77b48\"" Oct 30 16:33:16.918928 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:16.918905 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-a77b48-dockercfg-s4tl4\"" Oct 30 16:33:16.924467 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:16.924447 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-a77b48-699d989875-5w52s"] Oct 30 16:33:16.934058 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:16.933914 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffdf199f-692a-4542-a7cb-0f836829419e" path="/var/lib/kubelet/pods/ffdf199f-692a-4542-a7cb-0f836829419e/volumes" Oct 30 16:33:16.974619 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:16.974559 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a5145a9-7bd1-4f48-b116-6f08637fe08a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-a77b48-699d989875-5w52s\" (UID: \"7a5145a9-7bd1-4f48-b116-6f08637fe08a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a77b48-699d989875-5w52s" Oct 30 16:33:16.974776 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:16.974640 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkf27\" (UniqueName: \"kubernetes.io/projected/7a5145a9-7bd1-4f48-b116-6f08637fe08a-kube-api-access-fkf27\") pod \"payload-pre-processing-e2e-worker-w1-a77b48-699d989875-5w52s\" (UID: \"7a5145a9-7bd1-4f48-b116-6f08637fe08a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a77b48-699d989875-5w52s" Oct 30 16:33:17.007097 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:17.007066 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-a77b48-596b55946-4kzrx"] Oct 30 16:33:17.007493 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:17.007480 4493 cpu_manager.go:414] "containerMap: removing container" podUID="ffdf199f-692a-4542-a7cb-0f836829419e" containerName="istio-proxy" Oct 30 16:33:17.007493 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:17.007493 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffdf199f-692a-4542-a7cb-0f836829419e" containerName="istio-proxy" Oct 30 16:33:17.007614 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:17.007571 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffdf199f-692a-4542-a7cb-0f836829419e" containerName="istio-proxy" Oct 30 16:33:17.010889 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:17.010867 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-a77b48-596b55946-4kzrx" Oct 30 16:33:17.039703 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:17.039674 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-a77b48-596b55946-4kzrx"] Oct 30 16:33:17.075361 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:17.075322 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65nz6\" (UniqueName: \"kubernetes.io/projected/a6aa4382-931c-456e-a739-4580c07adf09-kube-api-access-65nz6\") pod \"payload-processing-e2e-worker-w1-a77b48-596b55946-4kzrx\" (UID: \"a6aa4382-931c-456e-a739-4580c07adf09\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-a77b48-596b55946-4kzrx" Oct 30 16:33:17.075536 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:17.075400 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a5145a9-7bd1-4f48-b116-6f08637fe08a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-a77b48-699d989875-5w52s\" (UID: \"7a5145a9-7bd1-4f48-b116-6f08637fe08a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a77b48-699d989875-5w52s" Oct 30 16:33:17.075536 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:17.075463 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkf27\" (UniqueName: \"kubernetes.io/projected/7a5145a9-7bd1-4f48-b116-6f08637fe08a-kube-api-access-fkf27\") pod \"payload-pre-processing-e2e-worker-w1-a77b48-699d989875-5w52s\" (UID: \"7a5145a9-7bd1-4f48-b116-6f08637fe08a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a77b48-699d989875-5w52s" Oct 30 16:33:17.075536 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:17.075512 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6aa4382-931c-456e-a739-4580c07adf09-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-a77b48-596b55946-4kzrx\" (UID: \"a6aa4382-931c-456e-a739-4580c07adf09\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-a77b48-596b55946-4kzrx" Oct 30 16:33:17.076536 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:17.076492 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a5145a9-7bd1-4f48-b116-6f08637fe08a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-a77b48-699d989875-5w52s\" (UID: \"7a5145a9-7bd1-4f48-b116-6f08637fe08a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a77b48-699d989875-5w52s" Oct 30 16:33:17.098895 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:17.098866 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkf27\" (UniqueName: \"kubernetes.io/projected/7a5145a9-7bd1-4f48-b116-6f08637fe08a-kube-api-access-fkf27\") pod \"payload-pre-processing-e2e-worker-w1-a77b48-699d989875-5w52s\" (UID: \"7a5145a9-7bd1-4f48-b116-6f08637fe08a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a77b48-699d989875-5w52s" Oct 30 16:33:17.176392 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:17.176323 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65nz6\" (UniqueName: \"kubernetes.io/projected/a6aa4382-931c-456e-a739-4580c07adf09-kube-api-access-65nz6\") pod \"payload-processing-e2e-worker-w1-a77b48-596b55946-4kzrx\" (UID: \"a6aa4382-931c-456e-a739-4580c07adf09\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-a77b48-596b55946-4kzrx" Oct 30 16:33:17.176392 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:17.176371 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6aa4382-931c-456e-a739-4580c07adf09-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-a77b48-596b55946-4kzrx\" (UID: \"a6aa4382-931c-456e-a739-4580c07adf09\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-a77b48-596b55946-4kzrx" Oct 30 16:33:17.177210 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:17.177183 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6aa4382-931c-456e-a739-4580c07adf09-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-a77b48-596b55946-4kzrx\" (UID: \"a6aa4382-931c-456e-a739-4580c07adf09\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-a77b48-596b55946-4kzrx" Oct 30 16:33:17.199627 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:17.199600 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65nz6\" (UniqueName: \"kubernetes.io/projected/a6aa4382-931c-456e-a739-4580c07adf09-kube-api-access-65nz6\") pod \"payload-processing-e2e-worker-w1-a77b48-596b55946-4kzrx\" (UID: \"a6aa4382-931c-456e-a739-4580c07adf09\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-a77b48-596b55946-4kzrx" Oct 30 16:33:17.217507 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:17.217485 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a77b48-699d989875-5w52s" Oct 30 16:33:17.325352 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:17.325319 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-a77b48-596b55946-4kzrx" Oct 30 16:33:17.516152 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:17.516128 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-a77b48-699d989875-5w52s"] Oct 30 16:33:17.518560 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:33:17.518533 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a5145a9_7bd1_4f48_b116_6f08637fe08a.slice/crio-8d2c92c33e615616991d0517ee5374deb2d91a22095c75442ff8dd43660a08a1 WatchSource:0}: Error finding container 8d2c92c33e615616991d0517ee5374deb2d91a22095c75442ff8dd43660a08a1: Status 404 returned error can't find the container with id 8d2c92c33e615616991d0517ee5374deb2d91a22095c75442ff8dd43660a08a1 Oct 30 16:33:17.625161 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:17.625137 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-a77b48-596b55946-4kzrx"] Oct 30 16:33:17.627726 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:33:17.627690 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6aa4382_931c_456e_a739_4580c07adf09.slice/crio-15a00ba2bb7902055a3df4c548ce483fc51294aff36be4ca9e713f101ee3ca40 WatchSource:0}: Error finding container 15a00ba2bb7902055a3df4c548ce483fc51294aff36be4ca9e713f101ee3ca40: Status 404 returned error can't find the container with id 15a00ba2bb7902055a3df4c548ce483fc51294aff36be4ca9e713f101ee3ca40 Oct 30 16:33:18.381231 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:18.381197 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-a77b48-596b55946-4kzrx" event={"ID":"a6aa4382-931c-456e-a739-4580c07adf09","Type":"ContainerStarted","Data":"6c70d0060d3d042006442faf627fd352bb3c0e9a566be62e36b273dcfd789bf0"} Oct 30 16:33:18.381715 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:18.381259 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-a77b48-596b55946-4kzrx" event={"ID":"a6aa4382-931c-456e-a739-4580c07adf09","Type":"ContainerStarted","Data":"15a00ba2bb7902055a3df4c548ce483fc51294aff36be4ca9e713f101ee3ca40"} Oct 30 16:33:18.381715 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:18.381290 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-a77b48-596b55946-4kzrx" Oct 30 16:33:18.382702 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:18.382680 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a77b48-699d989875-5w52s" event={"ID":"7a5145a9-7bd1-4f48-b116-6f08637fe08a","Type":"ContainerStarted","Data":"2ec3ec76b0d1bb6e9115b1b812b8b0be6ed98fa99f13f0154a345462696fe920"} Oct 30 16:33:18.382832 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:18.382721 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a77b48-699d989875-5w52s" event={"ID":"7a5145a9-7bd1-4f48-b116-6f08637fe08a","Type":"ContainerStarted","Data":"8d2c92c33e615616991d0517ee5374deb2d91a22095c75442ff8dd43660a08a1"} Oct 30 16:33:18.382874 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:18.382826 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a77b48-699d989875-5w52s" Oct 30 16:33:18.421593 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:18.421540 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-a77b48-596b55946-4kzrx" podStartSLOduration=2.421522518 podStartE2EDuration="2.421522518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:33:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:33:18.421268583 +0000 UTC m=+625.648443662" watchObservedRunningTime="2026-10-30 16:33:18.421522518 +0000 UTC m=+625.648697619" Oct 30 16:33:18.462999 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:18.462951 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a77b48-699d989875-5w52s" podStartSLOduration=2.462939571 podStartE2EDuration="2.462939571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:33:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:33:18.461013333 +0000 UTC m=+625.688188614" watchObservedRunningTime="2026-10-30 16:33:18.462939571 +0000 UTC m=+625.690114670" Oct 30 16:33:18.875844 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:18.875801 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 16:33:21.332328 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:21.332299 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-420ff3-7f7b8b4bdc-6d2x4" Oct 30 16:33:23.584811 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:23.584764 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl"] Oct 30 16:33:23.589569 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:23.589547 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" Oct 30 16:33:23.616022 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:23.615997 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-2d365c\"/\"default-dockercfg-skqw6\"" Oct 30 16:33:23.616022 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:23.616006 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-2d365c\"/\"kube-root-ca.crt\"" Oct 30 16:33:23.616178 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:23.616034 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-2d365c\"/\"openshift-service-ca.crt\"" Oct 30 16:33:23.616235 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:23.616216 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-2d365c\"/\"simulator-w3-2d365c-kserve-self-signed-certs\"" Oct 30 16:33:23.618471 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:23.618446 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl"] Oct 30 16:33:23.760032 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:23.759996 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsrnh\" (UniqueName: \"kubernetes.io/projected/e43738f3-d55f-4c0a-a364-bf2ac76605c6-kube-api-access-fsrnh\") pod \"simulator-w3-2d365c-kserve-9f985c58c-zzjfl\" (UID: \"e43738f3-d55f-4c0a-a364-bf2ac76605c6\") " pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" Oct 30 16:33:23.760206 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:23.760044 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e43738f3-d55f-4c0a-a364-bf2ac76605c6-dshm\") pod \"simulator-w3-2d365c-kserve-9f985c58c-zzjfl\" (UID: \"e43738f3-d55f-4c0a-a364-bf2ac76605c6\") " pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" Oct 30 16:33:23.760206 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:23.760063 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e43738f3-d55f-4c0a-a364-bf2ac76605c6-home\") pod \"simulator-w3-2d365c-kserve-9f985c58c-zzjfl\" (UID: \"e43738f3-d55f-4c0a-a364-bf2ac76605c6\") " pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" Oct 30 16:33:23.760206 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:23.760078 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e43738f3-d55f-4c0a-a364-bf2ac76605c6-tmp-dir\") pod \"simulator-w3-2d365c-kserve-9f985c58c-zzjfl\" (UID: \"e43738f3-d55f-4c0a-a364-bf2ac76605c6\") " pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" Oct 30 16:33:23.760206 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:23.760181 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e43738f3-d55f-4c0a-a364-bf2ac76605c6-model-cache\") pod \"simulator-w3-2d365c-kserve-9f985c58c-zzjfl\" (UID: \"e43738f3-d55f-4c0a-a364-bf2ac76605c6\") " pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" Oct 30 16:33:23.760403 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:23.760251 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e43738f3-d55f-4c0a-a364-bf2ac76605c6-tls-certs\") pod \"simulator-w3-2d365c-kserve-9f985c58c-zzjfl\" (UID: \"e43738f3-d55f-4c0a-a364-bf2ac76605c6\") " pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" Oct 30 16:33:23.861130 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:23.861049 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsrnh\" (UniqueName: \"kubernetes.io/projected/e43738f3-d55f-4c0a-a364-bf2ac76605c6-kube-api-access-fsrnh\") pod \"simulator-w3-2d365c-kserve-9f985c58c-zzjfl\" (UID: \"e43738f3-d55f-4c0a-a364-bf2ac76605c6\") " pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" Oct 30 16:33:23.861130 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:23.861093 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e43738f3-d55f-4c0a-a364-bf2ac76605c6-dshm\") pod \"simulator-w3-2d365c-kserve-9f985c58c-zzjfl\" (UID: \"e43738f3-d55f-4c0a-a364-bf2ac76605c6\") " pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" Oct 30 16:33:23.861130 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:23.861115 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e43738f3-d55f-4c0a-a364-bf2ac76605c6-home\") pod \"simulator-w3-2d365c-kserve-9f985c58c-zzjfl\" (UID: \"e43738f3-d55f-4c0a-a364-bf2ac76605c6\") " pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" Oct 30 16:33:23.861423 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:23.861225 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e43738f3-d55f-4c0a-a364-bf2ac76605c6-tmp-dir\") pod \"simulator-w3-2d365c-kserve-9f985c58c-zzjfl\" (UID: \"e43738f3-d55f-4c0a-a364-bf2ac76605c6\") " pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" Oct 30 16:33:23.861423 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:23.861263 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e43738f3-d55f-4c0a-a364-bf2ac76605c6-model-cache\") pod \"simulator-w3-2d365c-kserve-9f985c58c-zzjfl\" (UID: \"e43738f3-d55f-4c0a-a364-bf2ac76605c6\") " pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" Oct 30 16:33:23.861423 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:23.861326 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e43738f3-d55f-4c0a-a364-bf2ac76605c6-tls-certs\") pod \"simulator-w3-2d365c-kserve-9f985c58c-zzjfl\" (UID: \"e43738f3-d55f-4c0a-a364-bf2ac76605c6\") " pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" Oct 30 16:33:23.861563 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:23.861439 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e43738f3-d55f-4c0a-a364-bf2ac76605c6-home\") pod \"simulator-w3-2d365c-kserve-9f985c58c-zzjfl\" (UID: \"e43738f3-d55f-4c0a-a364-bf2ac76605c6\") " pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" Oct 30 16:33:23.861563 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:23.861532 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e43738f3-d55f-4c0a-a364-bf2ac76605c6-tmp-dir\") pod \"simulator-w3-2d365c-kserve-9f985c58c-zzjfl\" (UID: \"e43738f3-d55f-4c0a-a364-bf2ac76605c6\") " pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" Oct 30 16:33:23.861632 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:23.861563 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e43738f3-d55f-4c0a-a364-bf2ac76605c6-model-cache\") pod \"simulator-w3-2d365c-kserve-9f985c58c-zzjfl\" (UID: \"e43738f3-d55f-4c0a-a364-bf2ac76605c6\") " pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" Oct 30 16:33:23.863515 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:23.863489 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e43738f3-d55f-4c0a-a364-bf2ac76605c6-dshm\") pod \"simulator-w3-2d365c-kserve-9f985c58c-zzjfl\" (UID: \"e43738f3-d55f-4c0a-a364-bf2ac76605c6\") " pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" Oct 30 16:33:23.863844 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:23.863827 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e43738f3-d55f-4c0a-a364-bf2ac76605c6-tls-certs\") pod \"simulator-w3-2d365c-kserve-9f985c58c-zzjfl\" (UID: \"e43738f3-d55f-4c0a-a364-bf2ac76605c6\") " pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" Oct 30 16:33:23.883683 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:23.883656 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsrnh\" (UniqueName: \"kubernetes.io/projected/e43738f3-d55f-4c0a-a364-bf2ac76605c6-kube-api-access-fsrnh\") pod \"simulator-w3-2d365c-kserve-9f985c58c-zzjfl\" (UID: \"e43738f3-d55f-4c0a-a364-bf2ac76605c6\") " pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" Oct 30 16:33:23.903839 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:23.903819 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" Oct 30 16:33:24.208173 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:24.208134 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl"] Oct 30 16:33:24.211697 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:33:24.211663 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode43738f3_d55f_4c0a_a364_bf2ac76605c6.slice/crio-705ff074bda8b6906f62ccb4880e81dfec1d1279a4b87f8d28799444044eda52 WatchSource:0}: Error finding container 705ff074bda8b6906f62ccb4880e81dfec1d1279a4b87f8d28799444044eda52: Status 404 returned error can't find the container with id 705ff074bda8b6906f62ccb4880e81dfec1d1279a4b87f8d28799444044eda52 Oct 30 16:33:24.214100 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:24.214083 4493 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 16:33:24.458635 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:24.458547 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" event={"ID":"e43738f3-d55f-4c0a-a364-bf2ac76605c6","Type":"ContainerStarted","Data":"705ff074bda8b6906f62ccb4880e81dfec1d1279a4b87f8d28799444044eda52"} Oct 30 16:33:26.346735 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:26.346698 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 16:33:26.472719 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:26.472666 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" event={"ID":"e43738f3-d55f-4c0a-a364-bf2ac76605c6","Type":"ContainerStarted","Data":"c6a0c1fe11d893b0b5c92c409685d11fefe2250cba9742fb8acc3ffc9d7ff5ce"} Oct 30 16:33:26.519359 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:26.519294 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" podStartSLOduration=2.308921889 podStartE2EDuration="3.519273059s" totalImagesPullingTime="1.21035117s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:33:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:33:26.517849015 +0000 UTC m=+633.745024116" watchObservedRunningTime="2026-10-30 16:33:26.519273059 +0000 UTC m=+633.746448159" Oct 30 16:33:26.903974 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:26.903939 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" Oct 30 16:33:27.879296 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:27.879085 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh"] Oct 30 16:33:27.883924 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:27.883899 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" Oct 30 16:33:27.910776 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:27.910751 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-18519c\"/\"simulator-w2-18519c-kserve-self-signed-certs\"" Oct 30 16:33:27.910927 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:27.910830 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-18519c\"/\"default-dockercfg-rzlfm\"" Oct 30 16:33:27.913013 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:27.912987 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh"] Oct 30 16:33:27.968296 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:27.968262 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-18519c\"/\"openshift-service-ca.crt\"" Oct 30 16:33:27.969621 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:27.969601 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-18519c\"/\"kube-root-ca.crt\"" Oct 30 16:33:28.002074 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:28.002041 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80b9bd9a-30d6-45ff-b318-26fcdc608e5b-model-cache\") pod \"simulator-w2-18519c-kserve-7994695cdd-rlwgh\" (UID: \"80b9bd9a-30d6-45ff-b318-26fcdc608e5b\") " pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" Oct 30 16:33:28.002198 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:28.002161 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80b9bd9a-30d6-45ff-b318-26fcdc608e5b-home\") pod \"simulator-w2-18519c-kserve-7994695cdd-rlwgh\" (UID: \"80b9bd9a-30d6-45ff-b318-26fcdc608e5b\") " pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" Oct 30 16:33:28.002258 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:28.002204 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cc4wt\" (UniqueName: \"kubernetes.io/projected/80b9bd9a-30d6-45ff-b318-26fcdc608e5b-kube-api-access-cc4wt\") pod \"simulator-w2-18519c-kserve-7994695cdd-rlwgh\" (UID: \"80b9bd9a-30d6-45ff-b318-26fcdc608e5b\") " pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" Oct 30 16:33:28.002335 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:28.002315 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80b9bd9a-30d6-45ff-b318-26fcdc608e5b-dshm\") pod \"simulator-w2-18519c-kserve-7994695cdd-rlwgh\" (UID: \"80b9bd9a-30d6-45ff-b318-26fcdc608e5b\") " pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" Oct 30 16:33:28.002389 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:28.002361 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80b9bd9a-30d6-45ff-b318-26fcdc608e5b-tmp-dir\") pod \"simulator-w2-18519c-kserve-7994695cdd-rlwgh\" (UID: \"80b9bd9a-30d6-45ff-b318-26fcdc608e5b\") " pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" Oct 30 16:33:28.002453 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:28.002438 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80b9bd9a-30d6-45ff-b318-26fcdc608e5b-tls-certs\") pod \"simulator-w2-18519c-kserve-7994695cdd-rlwgh\" (UID: \"80b9bd9a-30d6-45ff-b318-26fcdc608e5b\") " pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" Oct 30 16:33:28.103997 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:28.103965 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cc4wt\" (UniqueName: \"kubernetes.io/projected/80b9bd9a-30d6-45ff-b318-26fcdc608e5b-kube-api-access-cc4wt\") pod \"simulator-w2-18519c-kserve-7994695cdd-rlwgh\" (UID: \"80b9bd9a-30d6-45ff-b318-26fcdc608e5b\") " pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" Oct 30 16:33:28.104255 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:28.104047 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80b9bd9a-30d6-45ff-b318-26fcdc608e5b-dshm\") pod \"simulator-w2-18519c-kserve-7994695cdd-rlwgh\" (UID: \"80b9bd9a-30d6-45ff-b318-26fcdc608e5b\") " pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" Oct 30 16:33:28.104255 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:28.104086 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80b9bd9a-30d6-45ff-b318-26fcdc608e5b-tmp-dir\") pod \"simulator-w2-18519c-kserve-7994695cdd-rlwgh\" (UID: \"80b9bd9a-30d6-45ff-b318-26fcdc608e5b\") " pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" Oct 30 16:33:28.104255 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:28.104140 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80b9bd9a-30d6-45ff-b318-26fcdc608e5b-tls-certs\") pod \"simulator-w2-18519c-kserve-7994695cdd-rlwgh\" (UID: \"80b9bd9a-30d6-45ff-b318-26fcdc608e5b\") " pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" Oct 30 16:33:28.104255 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:28.104176 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80b9bd9a-30d6-45ff-b318-26fcdc608e5b-model-cache\") pod \"simulator-w2-18519c-kserve-7994695cdd-rlwgh\" (UID: \"80b9bd9a-30d6-45ff-b318-26fcdc608e5b\") " pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" Oct 30 16:33:28.104533 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:28.104278 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80b9bd9a-30d6-45ff-b318-26fcdc608e5b-home\") pod \"simulator-w2-18519c-kserve-7994695cdd-rlwgh\" (UID: \"80b9bd9a-30d6-45ff-b318-26fcdc608e5b\") " pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" Oct 30 16:33:28.104665 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:28.104637 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80b9bd9a-30d6-45ff-b318-26fcdc608e5b-home\") pod \"simulator-w2-18519c-kserve-7994695cdd-rlwgh\" (UID: \"80b9bd9a-30d6-45ff-b318-26fcdc608e5b\") " pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" Oct 30 16:33:28.104957 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:28.104933 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80b9bd9a-30d6-45ff-b318-26fcdc608e5b-model-cache\") pod \"simulator-w2-18519c-kserve-7994695cdd-rlwgh\" (UID: \"80b9bd9a-30d6-45ff-b318-26fcdc608e5b\") " pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" Oct 30 16:33:28.105186 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:28.105167 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80b9bd9a-30d6-45ff-b318-26fcdc608e5b-tmp-dir\") pod \"simulator-w2-18519c-kserve-7994695cdd-rlwgh\" (UID: \"80b9bd9a-30d6-45ff-b318-26fcdc608e5b\") " pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" Oct 30 16:33:28.106831 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:28.106806 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80b9bd9a-30d6-45ff-b318-26fcdc608e5b-dshm\") pod \"simulator-w2-18519c-kserve-7994695cdd-rlwgh\" (UID: \"80b9bd9a-30d6-45ff-b318-26fcdc608e5b\") " pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" Oct 30 16:33:28.107010 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:28.106993 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80b9bd9a-30d6-45ff-b318-26fcdc608e5b-tls-certs\") pod \"simulator-w2-18519c-kserve-7994695cdd-rlwgh\" (UID: \"80b9bd9a-30d6-45ff-b318-26fcdc608e5b\") " pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" Oct 30 16:33:28.132601 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:28.132544 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc4wt\" (UniqueName: \"kubernetes.io/projected/80b9bd9a-30d6-45ff-b318-26fcdc608e5b-kube-api-access-cc4wt\") pod \"simulator-w2-18519c-kserve-7994695cdd-rlwgh\" (UID: \"80b9bd9a-30d6-45ff-b318-26fcdc608e5b\") " pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" Oct 30 16:33:28.197183 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:28.197156 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" Oct 30 16:33:28.528703 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:28.528680 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh"] Oct 30 16:33:28.531381 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:33:28.531342 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod80b9bd9a_30d6_45ff_b318_26fcdc608e5b.slice/crio-136fc8afab04aebcdb2163ec3bc7873ff6e2904d643b100b0b4ead51c4b4decd WatchSource:0}: Error finding container 136fc8afab04aebcdb2163ec3bc7873ff6e2904d643b100b0b4ead51c4b4decd: Status 404 returned error can't find the container with id 136fc8afab04aebcdb2163ec3bc7873ff6e2904d643b100b0b4ead51c4b4decd Oct 30 16:33:29.422608 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:29.422579 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-a77b48-596b55946-4kzrx" Oct 30 16:33:29.423005 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:29.422646 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a77b48-699d989875-5w52s" Oct 30 16:33:29.494431 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:29.494396 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" event={"ID":"80b9bd9a-30d6-45ff-b318-26fcdc608e5b","Type":"ContainerStarted","Data":"e68e01ba784a12f2a8823555b0a1d6d474c9c4857fe5bfbed1d5c7d4b68e06fd"} Oct 30 16:33:29.494608 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:29.494440 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" event={"ID":"80b9bd9a-30d6-45ff-b318-26fcdc608e5b","Type":"ContainerStarted","Data":"136fc8afab04aebcdb2163ec3bc7873ff6e2904d643b100b0b4ead51c4b4decd"} Oct 30 16:33:29.550143 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:29.550084 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" podStartSLOduration=2.550066461 podStartE2EDuration="2.550066461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:33:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:33:29.546835224 +0000 UTC m=+636.774010350" watchObservedRunningTime="2026-10-30 16:33:29.550066461 +0000 UTC m=+636.777241561" Oct 30 16:33:30.209327 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:30.207888 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" Oct 30 16:33:30.347492 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:33:30.347458 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 16:33:33.908140 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:33.906864 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" Oct 30 16:33:33.950775 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:33.950745 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" Oct 30 16:33:34.573261 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:34.573230 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" Oct 30 16:33:36.789391 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:36.789358 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 16:33:38.197645 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:38.197614 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" Oct 30 16:33:38.210727 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:38.210704 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" Oct 30 16:33:38.601622 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:38.601537 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" Oct 30 16:33:40.394427 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:40.394390 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq"] Oct 30 16:33:40.396909 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:40.396886 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" Oct 30 16:33:40.413072 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:40.413042 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-f3503b\"/\"kube-root-ca.crt\"" Oct 30 16:33:40.424725 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:40.424701 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-f3503b\"/\"openshift-service-ca.crt\"" Oct 30 16:33:40.424884 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:40.424732 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f3503b\"/\"simulator-w0-f3503b-kserve-self-signed-certs\"" Oct 30 16:33:40.424939 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:40.424890 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f3503b\"/\"default-dockercfg-n7c78\"" Oct 30 16:33:40.428604 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:40.428576 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq"] Oct 30 16:33:40.439003 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:40.438972 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08baad08-cd72-4305-b92d-c11a84ef276f-model-cache\") pod \"simulator-w0-f3503b-kserve-76869b4554-tj8hq\" (UID: \"08baad08-cd72-4305-b92d-c11a84ef276f\") " pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" Oct 30 16:33:40.439123 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:40.439044 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08baad08-cd72-4305-b92d-c11a84ef276f-home\") pod \"simulator-w0-f3503b-kserve-76869b4554-tj8hq\" (UID: \"08baad08-cd72-4305-b92d-c11a84ef276f\") " pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" Oct 30 16:33:40.439198 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:40.439126 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj7hs\" (UniqueName: \"kubernetes.io/projected/08baad08-cd72-4305-b92d-c11a84ef276f-kube-api-access-vj7hs\") pod \"simulator-w0-f3503b-kserve-76869b4554-tj8hq\" (UID: \"08baad08-cd72-4305-b92d-c11a84ef276f\") " pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" Oct 30 16:33:40.439258 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:40.439204 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08baad08-cd72-4305-b92d-c11a84ef276f-tls-certs\") pod \"simulator-w0-f3503b-kserve-76869b4554-tj8hq\" (UID: \"08baad08-cd72-4305-b92d-c11a84ef276f\") " pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" Oct 30 16:33:40.439258 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:40.439230 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08baad08-cd72-4305-b92d-c11a84ef276f-dshm\") pod \"simulator-w0-f3503b-kserve-76869b4554-tj8hq\" (UID: \"08baad08-cd72-4305-b92d-c11a84ef276f\") " pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" Oct 30 16:33:40.439365 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:40.439271 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08baad08-cd72-4305-b92d-c11a84ef276f-tmp-dir\") pod \"simulator-w0-f3503b-kserve-76869b4554-tj8hq\" (UID: \"08baad08-cd72-4305-b92d-c11a84ef276f\") " pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" Oct 30 16:33:40.539886 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:40.539842 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj7hs\" (UniqueName: \"kubernetes.io/projected/08baad08-cd72-4305-b92d-c11a84ef276f-kube-api-access-vj7hs\") pod \"simulator-w0-f3503b-kserve-76869b4554-tj8hq\" (UID: \"08baad08-cd72-4305-b92d-c11a84ef276f\") " pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" Oct 30 16:33:40.540045 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:40.539947 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08baad08-cd72-4305-b92d-c11a84ef276f-tls-certs\") pod \"simulator-w0-f3503b-kserve-76869b4554-tj8hq\" (UID: \"08baad08-cd72-4305-b92d-c11a84ef276f\") " pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" Oct 30 16:33:40.540045 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:40.539985 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08baad08-cd72-4305-b92d-c11a84ef276f-dshm\") pod \"simulator-w0-f3503b-kserve-76869b4554-tj8hq\" (UID: \"08baad08-cd72-4305-b92d-c11a84ef276f\") " pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" Oct 30 16:33:40.540118 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:40.540043 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08baad08-cd72-4305-b92d-c11a84ef276f-tmp-dir\") pod \"simulator-w0-f3503b-kserve-76869b4554-tj8hq\" (UID: \"08baad08-cd72-4305-b92d-c11a84ef276f\") " pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" Oct 30 16:33:40.540118 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:40.540075 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08baad08-cd72-4305-b92d-c11a84ef276f-model-cache\") pod \"simulator-w0-f3503b-kserve-76869b4554-tj8hq\" (UID: \"08baad08-cd72-4305-b92d-c11a84ef276f\") " pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" Oct 30 16:33:40.540183 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:40.540131 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08baad08-cd72-4305-b92d-c11a84ef276f-home\") pod \"simulator-w0-f3503b-kserve-76869b4554-tj8hq\" (UID: \"08baad08-cd72-4305-b92d-c11a84ef276f\") " pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" Oct 30 16:33:40.540577 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:40.540558 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08baad08-cd72-4305-b92d-c11a84ef276f-home\") pod \"simulator-w0-f3503b-kserve-76869b4554-tj8hq\" (UID: \"08baad08-cd72-4305-b92d-c11a84ef276f\") " pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" Oct 30 16:33:40.540873 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:40.540851 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08baad08-cd72-4305-b92d-c11a84ef276f-tmp-dir\") pod \"simulator-w0-f3503b-kserve-76869b4554-tj8hq\" (UID: \"08baad08-cd72-4305-b92d-c11a84ef276f\") " pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" Oct 30 16:33:40.540985 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:40.540917 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08baad08-cd72-4305-b92d-c11a84ef276f-model-cache\") pod \"simulator-w0-f3503b-kserve-76869b4554-tj8hq\" (UID: \"08baad08-cd72-4305-b92d-c11a84ef276f\") " pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" Oct 30 16:33:40.542622 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:40.542599 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08baad08-cd72-4305-b92d-c11a84ef276f-dshm\") pod \"simulator-w0-f3503b-kserve-76869b4554-tj8hq\" (UID: \"08baad08-cd72-4305-b92d-c11a84ef276f\") " pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" Oct 30 16:33:40.543084 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:40.543063 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08baad08-cd72-4305-b92d-c11a84ef276f-tls-certs\") pod \"simulator-w0-f3503b-kserve-76869b4554-tj8hq\" (UID: \"08baad08-cd72-4305-b92d-c11a84ef276f\") " pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" Oct 30 16:33:40.564325 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:40.564294 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj7hs\" (UniqueName: \"kubernetes.io/projected/08baad08-cd72-4305-b92d-c11a84ef276f-kube-api-access-vj7hs\") pod \"simulator-w0-f3503b-kserve-76869b4554-tj8hq\" (UID: \"08baad08-cd72-4305-b92d-c11a84ef276f\") " pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" Oct 30 16:33:40.601634 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:40.601602 4493 generic.go:356] "Generic (PLEG): container finished" podID="c7ae7018-798b-4a3a-afef-b2ed1a13b05f" containerID="2b851adaac88aed5e816a099ff1adcf586423cae5811e65fa657dd00ed2b18ae" exitCode=0 Oct 30 16:33:40.601805 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:40.601672 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2" event={"ID":"c7ae7018-798b-4a3a-afef-b2ed1a13b05f","Type":"ContainerDied","Data":"2b851adaac88aed5e816a099ff1adcf586423cae5811e65fa657dd00ed2b18ae"} Oct 30 16:33:40.602098 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:40.602084 4493 scope.go:122] "RemoveContainer" containerID="2b851adaac88aed5e816a099ff1adcf586423cae5811e65fa657dd00ed2b18ae" Oct 30 16:33:40.603387 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:40.603357 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:33:40.603475 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:40.603427 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:33:40.603475 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:40.603460 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:33:40.722290 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:40.722260 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" Oct 30 16:33:41.037567 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:41.037490 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq"] Oct 30 16:33:41.041223 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:33:41.041193 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod08baad08_cd72_4305_b92d_c11a84ef276f.slice/crio-df1164b5832f0fd74a25422669164ac01c3e6ebad8ec25d5cc12654a8de3e4b9 WatchSource:0}: Error finding container df1164b5832f0fd74a25422669164ac01c3e6ebad8ec25d5cc12654a8de3e4b9: Status 404 returned error can't find the container with id df1164b5832f0fd74a25422669164ac01c3e6ebad8ec25d5cc12654a8de3e4b9 Oct 30 16:33:41.632320 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:41.632272 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" event={"ID":"08baad08-cd72-4305-b92d-c11a84ef276f","Type":"ContainerStarted","Data":"a0f7895786747598a42513c8e971f1f7741fa9b45b156dbab73bb4052578a447"} Oct 30 16:33:41.632876 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:41.632339 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" event={"ID":"08baad08-cd72-4305-b92d-c11a84ef276f","Type":"ContainerStarted","Data":"df1164b5832f0fd74a25422669164ac01c3e6ebad8ec25d5cc12654a8de3e4b9"} Oct 30 16:33:41.634448 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:41.634418 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2" event={"ID":"c7ae7018-798b-4a3a-afef-b2ed1a13b05f","Type":"ContainerStarted","Data":"77995f75476d1b422cb7d950bac96dace02d103308e06c8c1b9649b5c577b969"} Oct 30 16:33:41.674708 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:41.674477 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" podStartSLOduration=1.6744475159999999 podStartE2EDuration="1.674447516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:33:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:33:41.673501246 +0000 UTC m=+648.900676347" watchObservedRunningTime="2026-10-30 16:33:41.674447516 +0000 UTC m=+648.901622618" Oct 30 16:33:41.729828 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:41.723030 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" Oct 30 16:33:41.931077 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:41.931042 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2" Oct 30 16:33:41.931421 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:41.931398 4493 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 30 16:33:41.931580 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:41.931554 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2" podUID="c7ae7018-798b-4a3a-afef-b2ed1a13b05f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 30 16:33:42.929156 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:42.929118 4493 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 30 16:33:42.934931 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:42.934886 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2" podUID="c7ae7018-798b-4a3a-afef-b2ed1a13b05f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 30 16:33:43.058530 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:43.058491 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4"] Oct 30 16:33:43.061680 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:43.061659 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" Oct 30 16:33:43.076579 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:43.076552 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-420ff3\"/\"openshift-service-ca.crt\"" Oct 30 16:33:43.087776 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:43.087748 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-420ff3\"/\"simulator-w4-420ff3-kserve-self-signed-certs\"" Oct 30 16:33:43.087971 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:43.087839 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-420ff3\"/\"kube-root-ca.crt\"" Oct 30 16:33:43.088285 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:43.088265 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-420ff3\"/\"default-dockercfg-2wjrq\"" Oct 30 16:33:43.093117 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:43.093091 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4"] Oct 30 16:33:43.166301 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:43.166259 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14805045-a727-41ba-8bd0-100d41f9f27d-home\") pod \"simulator-w4-420ff3-kserve-6d994d64b6-frdh4\" (UID: \"14805045-a727-41ba-8bd0-100d41f9f27d\") " pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" Oct 30 16:33:43.166301 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:43.166305 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14805045-a727-41ba-8bd0-100d41f9f27d-dshm\") pod \"simulator-w4-420ff3-kserve-6d994d64b6-frdh4\" (UID: \"14805045-a727-41ba-8bd0-100d41f9f27d\") " pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" Oct 30 16:33:43.166552 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:43.166492 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14805045-a727-41ba-8bd0-100d41f9f27d-model-cache\") pod \"simulator-w4-420ff3-kserve-6d994d64b6-frdh4\" (UID: \"14805045-a727-41ba-8bd0-100d41f9f27d\") " pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" Oct 30 16:33:43.166617 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:43.166561 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkpzd\" (UniqueName: \"kubernetes.io/projected/14805045-a727-41ba-8bd0-100d41f9f27d-kube-api-access-dkpzd\") pod \"simulator-w4-420ff3-kserve-6d994d64b6-frdh4\" (UID: \"14805045-a727-41ba-8bd0-100d41f9f27d\") " pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" Oct 30 16:33:43.166727 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:43.166706 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14805045-a727-41ba-8bd0-100d41f9f27d-tmp-dir\") pod \"simulator-w4-420ff3-kserve-6d994d64b6-frdh4\" (UID: \"14805045-a727-41ba-8bd0-100d41f9f27d\") " pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" Oct 30 16:33:43.166840 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:43.166821 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14805045-a727-41ba-8bd0-100d41f9f27d-tls-certs\") pod \"simulator-w4-420ff3-kserve-6d994d64b6-frdh4\" (UID: \"14805045-a727-41ba-8bd0-100d41f9f27d\") " pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" Oct 30 16:33:43.268139 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:43.268033 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14805045-a727-41ba-8bd0-100d41f9f27d-home\") pod \"simulator-w4-420ff3-kserve-6d994d64b6-frdh4\" (UID: \"14805045-a727-41ba-8bd0-100d41f9f27d\") " pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" Oct 30 16:33:43.268139 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:43.268118 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14805045-a727-41ba-8bd0-100d41f9f27d-dshm\") pod \"simulator-w4-420ff3-kserve-6d994d64b6-frdh4\" (UID: \"14805045-a727-41ba-8bd0-100d41f9f27d\") " pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" Oct 30 16:33:43.268371 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:43.268182 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14805045-a727-41ba-8bd0-100d41f9f27d-model-cache\") pod \"simulator-w4-420ff3-kserve-6d994d64b6-frdh4\" (UID: \"14805045-a727-41ba-8bd0-100d41f9f27d\") " pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" Oct 30 16:33:43.268371 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:43.268221 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkpzd\" (UniqueName: \"kubernetes.io/projected/14805045-a727-41ba-8bd0-100d41f9f27d-kube-api-access-dkpzd\") pod \"simulator-w4-420ff3-kserve-6d994d64b6-frdh4\" (UID: \"14805045-a727-41ba-8bd0-100d41f9f27d\") " pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" Oct 30 16:33:43.268371 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:43.268282 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14805045-a727-41ba-8bd0-100d41f9f27d-tmp-dir\") pod \"simulator-w4-420ff3-kserve-6d994d64b6-frdh4\" (UID: \"14805045-a727-41ba-8bd0-100d41f9f27d\") " pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" Oct 30 16:33:43.268371 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:43.268317 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14805045-a727-41ba-8bd0-100d41f9f27d-tls-certs\") pod \"simulator-w4-420ff3-kserve-6d994d64b6-frdh4\" (UID: \"14805045-a727-41ba-8bd0-100d41f9f27d\") " pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" Oct 30 16:33:43.268589 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:43.268486 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14805045-a727-41ba-8bd0-100d41f9f27d-home\") pod \"simulator-w4-420ff3-kserve-6d994d64b6-frdh4\" (UID: \"14805045-a727-41ba-8bd0-100d41f9f27d\") " pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" Oct 30 16:33:43.268777 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:43.268726 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14805045-a727-41ba-8bd0-100d41f9f27d-tmp-dir\") pod \"simulator-w4-420ff3-kserve-6d994d64b6-frdh4\" (UID: \"14805045-a727-41ba-8bd0-100d41f9f27d\") " pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" Oct 30 16:33:43.268946 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:43.268871 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14805045-a727-41ba-8bd0-100d41f9f27d-model-cache\") pod \"simulator-w4-420ff3-kserve-6d994d64b6-frdh4\" (UID: \"14805045-a727-41ba-8bd0-100d41f9f27d\") " pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" Oct 30 16:33:43.271107 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:43.271080 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14805045-a727-41ba-8bd0-100d41f9f27d-dshm\") pod \"simulator-w4-420ff3-kserve-6d994d64b6-frdh4\" (UID: \"14805045-a727-41ba-8bd0-100d41f9f27d\") " pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" Oct 30 16:33:43.271997 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:43.271968 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14805045-a727-41ba-8bd0-100d41f9f27d-tls-certs\") pod \"simulator-w4-420ff3-kserve-6d994d64b6-frdh4\" (UID: \"14805045-a727-41ba-8bd0-100d41f9f27d\") " pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" Oct 30 16:33:43.292329 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:43.292302 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkpzd\" (UniqueName: \"kubernetes.io/projected/14805045-a727-41ba-8bd0-100d41f9f27d-kube-api-access-dkpzd\") pod \"simulator-w4-420ff3-kserve-6d994d64b6-frdh4\" (UID: \"14805045-a727-41ba-8bd0-100d41f9f27d\") " pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" Oct 30 16:33:43.378083 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:43.378051 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" Oct 30 16:33:43.699997 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:43.699974 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4"] Oct 30 16:33:43.701714 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:33:43.701687 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod14805045_a727_41ba_8bd0_100d41f9f27d.slice/crio-a9f4e488519cdc139ad3e08c8487c3daf9e8667d4c70b44f1c5480757148da2f WatchSource:0}: Error finding container a9f4e488519cdc139ad3e08c8487c3daf9e8667d4c70b44f1c5480757148da2f: Status 404 returned error can't find the container with id a9f4e488519cdc139ad3e08c8487c3daf9e8667d4c70b44f1c5480757148da2f Oct 30 16:33:43.928811 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:43.927114 4493 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 30 16:33:43.928811 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:43.927170 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2" podUID="c7ae7018-798b-4a3a-afef-b2ed1a13b05f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 30 16:33:44.352082 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:44.352045 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2" Oct 30 16:33:44.663504 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:44.663469 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" event={"ID":"14805045-a727-41ba-8bd0-100d41f9f27d","Type":"ContainerStarted","Data":"8a24eac0d5002d9945036ecbb5b8fb68b88425ede0a7b03e14601c9f617cc27f"} Oct 30 16:33:44.663504 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:44.663514 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" event={"ID":"14805045-a727-41ba-8bd0-100d41f9f27d","Type":"ContainerStarted","Data":"a9f4e488519cdc139ad3e08c8487c3daf9e8667d4c70b44f1c5480757148da2f"} Oct 30 16:33:44.928234 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:44.928146 4493 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 30 16:33:44.931918 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:44.931879 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2" podUID="c7ae7018-798b-4a3a-afef-b2ed1a13b05f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 30 16:33:45.378271 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:45.378195 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" Oct 30 16:33:45.930809 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:45.930173 4493 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 30 16:33:45.930809 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:45.930234 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2" podUID="c7ae7018-798b-4a3a-afef-b2ed1a13b05f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 30 16:33:46.204224 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:46.204163 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" podStartSLOduration=3.204146599 podStartE2EDuration="3.204146599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:33:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:33:44.728948628 +0000 UTC m=+651.956123730" watchObservedRunningTime="2026-10-30 16:33:46.204146599 +0000 UTC m=+653.431321700" Oct 30 16:33:46.220084 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:46.220045 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 16:33:46.935140 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:46.935107 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2" Oct 30 16:33:46.936235 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:46.936215 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2" Oct 30 16:33:50.723324 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:50.723291 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" Oct 30 16:33:50.735937 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:50.735914 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" Oct 30 16:33:51.724644 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:51.724616 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" Oct 30 16:33:53.378252 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:53.378170 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" Oct 30 16:33:53.391276 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:53.391249 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" Oct 30 16:33:53.742355 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:53.742323 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" Oct 30 16:33:54.801020 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:54.800986 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s"] Oct 30 16:33:54.807381 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:54.807356 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" Oct 30 16:33:54.826642 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:54.826615 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s"] Oct 30 16:33:54.836284 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:54.835721 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-a77b48\"/\"kube-root-ca.crt\"" Oct 30 16:33:54.836284 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:54.835819 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-a77b48\"/\"openshift-service-ca.crt\"" Oct 30 16:33:54.838694 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:54.838677 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-a77b48\"/\"simulator-w1-a77b48-kserve-self-signed-certs\"" Oct 30 16:33:54.838903 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:54.838723 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-a77b48\"/\"default-dockercfg-pvgxh\"" Oct 30 16:33:54.894018 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:54.893966 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e76172b1-659d-45bd-91c6-1a0ce4062335-tls-certs\") pod \"simulator-w1-a77b48-kserve-569bdfdd67-rd82s\" (UID: \"e76172b1-659d-45bd-91c6-1a0ce4062335\") " pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" Oct 30 16:33:54.894200 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:54.894026 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e76172b1-659d-45bd-91c6-1a0ce4062335-dshm\") pod \"simulator-w1-a77b48-kserve-569bdfdd67-rd82s\" (UID: \"e76172b1-659d-45bd-91c6-1a0ce4062335\") " pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" Oct 30 16:33:54.894200 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:54.894162 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e76172b1-659d-45bd-91c6-1a0ce4062335-home\") pod \"simulator-w1-a77b48-kserve-569bdfdd67-rd82s\" (UID: \"e76172b1-659d-45bd-91c6-1a0ce4062335\") " pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" Oct 30 16:33:54.894200 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:54.894187 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e76172b1-659d-45bd-91c6-1a0ce4062335-model-cache\") pod \"simulator-w1-a77b48-kserve-569bdfdd67-rd82s\" (UID: \"e76172b1-659d-45bd-91c6-1a0ce4062335\") " pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" Oct 30 16:33:54.894382 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:54.894266 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e76172b1-659d-45bd-91c6-1a0ce4062335-tmp-dir\") pod \"simulator-w1-a77b48-kserve-569bdfdd67-rd82s\" (UID: \"e76172b1-659d-45bd-91c6-1a0ce4062335\") " pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" Oct 30 16:33:54.894382 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:54.894297 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gcnl\" (UniqueName: \"kubernetes.io/projected/e76172b1-659d-45bd-91c6-1a0ce4062335-kube-api-access-7gcnl\") pod \"simulator-w1-a77b48-kserve-569bdfdd67-rd82s\" (UID: \"e76172b1-659d-45bd-91c6-1a0ce4062335\") " pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" Oct 30 16:33:54.995634 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:54.995594 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e76172b1-659d-45bd-91c6-1a0ce4062335-home\") pod \"simulator-w1-a77b48-kserve-569bdfdd67-rd82s\" (UID: \"e76172b1-659d-45bd-91c6-1a0ce4062335\") " pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" Oct 30 16:33:54.995634 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:54.995646 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e76172b1-659d-45bd-91c6-1a0ce4062335-model-cache\") pod \"simulator-w1-a77b48-kserve-569bdfdd67-rd82s\" (UID: \"e76172b1-659d-45bd-91c6-1a0ce4062335\") " pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" Oct 30 16:33:54.995912 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:54.995698 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e76172b1-659d-45bd-91c6-1a0ce4062335-tmp-dir\") pod \"simulator-w1-a77b48-kserve-569bdfdd67-rd82s\" (UID: \"e76172b1-659d-45bd-91c6-1a0ce4062335\") " pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" Oct 30 16:33:54.995912 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:54.995726 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gcnl\" (UniqueName: \"kubernetes.io/projected/e76172b1-659d-45bd-91c6-1a0ce4062335-kube-api-access-7gcnl\") pod \"simulator-w1-a77b48-kserve-569bdfdd67-rd82s\" (UID: \"e76172b1-659d-45bd-91c6-1a0ce4062335\") " pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" Oct 30 16:33:54.995912 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:54.995838 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e76172b1-659d-45bd-91c6-1a0ce4062335-tls-certs\") pod \"simulator-w1-a77b48-kserve-569bdfdd67-rd82s\" (UID: \"e76172b1-659d-45bd-91c6-1a0ce4062335\") " pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" Oct 30 16:33:54.995912 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:54.995867 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e76172b1-659d-45bd-91c6-1a0ce4062335-dshm\") pod \"simulator-w1-a77b48-kserve-569bdfdd67-rd82s\" (UID: \"e76172b1-659d-45bd-91c6-1a0ce4062335\") " pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" Oct 30 16:33:54.996171 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:54.996143 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e76172b1-659d-45bd-91c6-1a0ce4062335-model-cache\") pod \"simulator-w1-a77b48-kserve-569bdfdd67-rd82s\" (UID: \"e76172b1-659d-45bd-91c6-1a0ce4062335\") " pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" Oct 30 16:33:54.996450 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:54.996420 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e76172b1-659d-45bd-91c6-1a0ce4062335-tmp-dir\") pod \"simulator-w1-a77b48-kserve-569bdfdd67-rd82s\" (UID: \"e76172b1-659d-45bd-91c6-1a0ce4062335\") " pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" Oct 30 16:33:54.996562 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:54.996524 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e76172b1-659d-45bd-91c6-1a0ce4062335-home\") pod \"simulator-w1-a77b48-kserve-569bdfdd67-rd82s\" (UID: \"e76172b1-659d-45bd-91c6-1a0ce4062335\") " pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" Oct 30 16:33:54.998694 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:54.998664 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e76172b1-659d-45bd-91c6-1a0ce4062335-tls-certs\") pod \"simulator-w1-a77b48-kserve-569bdfdd67-rd82s\" (UID: \"e76172b1-659d-45bd-91c6-1a0ce4062335\") " pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" Oct 30 16:33:54.998835 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:54.998664 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e76172b1-659d-45bd-91c6-1a0ce4062335-dshm\") pod \"simulator-w1-a77b48-kserve-569bdfdd67-rd82s\" (UID: \"e76172b1-659d-45bd-91c6-1a0ce4062335\") " pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" Oct 30 16:33:55.021576 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:55.021543 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gcnl\" (UniqueName: \"kubernetes.io/projected/e76172b1-659d-45bd-91c6-1a0ce4062335-kube-api-access-7gcnl\") pod \"simulator-w1-a77b48-kserve-569bdfdd67-rd82s\" (UID: \"e76172b1-659d-45bd-91c6-1a0ce4062335\") " pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" Oct 30 16:33:55.123928 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:55.123849 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" Oct 30 16:33:55.455440 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:55.455322 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s"] Oct 30 16:33:55.458073 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:33:55.458045 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode76172b1_659d_45bd_91c6_1a0ce4062335.slice/crio-9533242dfa45531743738f256243d2178534a2461f29f39801c2f3bb3e62e84d WatchSource:0}: Error finding container 9533242dfa45531743738f256243d2178534a2461f29f39801c2f3bb3e62e84d: Status 404 returned error can't find the container with id 9533242dfa45531743738f256243d2178534a2461f29f39801c2f3bb3e62e84d Oct 30 16:33:55.743052 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:55.742960 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" event={"ID":"e76172b1-659d-45bd-91c6-1a0ce4062335","Type":"ContainerStarted","Data":"d2fa10a51b115430a6d73a19fa32de800ee2097a3b3d1812b86855f431596134"} Oct 30 16:33:55.743052 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:55.743007 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" event={"ID":"e76172b1-659d-45bd-91c6-1a0ce4062335","Type":"ContainerStarted","Data":"9533242dfa45531743738f256243d2178534a2461f29f39801c2f3bb3e62e84d"} Oct 30 16:33:55.790976 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:55.790927 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" podStartSLOduration=1.7909141229999999 podStartE2EDuration="1.790914123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:33:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:33:55.790267862 +0000 UTC m=+663.017442961" watchObservedRunningTime="2026-10-30 16:33:55.790914123 +0000 UTC m=+663.018089236" Oct 30 16:33:56.124622 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:56.124543 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" Oct 30 16:33:59.109959 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:33:59.109918 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 16:34:00.344388 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:34:00.344344 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 16:34:05.124367 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:05.124279 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" Oct 30 16:34:05.137542 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:05.137515 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" Oct 30 16:34:05.829499 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:05.829465 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" Oct 30 16:34:10.108277 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:10.108170 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642"] Oct 30 16:34:10.112170 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:10.112149 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" Oct 30 16:34:10.135971 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:10.135937 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642"] Oct 30 16:34:10.141001 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:10.140976 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f3503b\"/\"premium-w0-f3503b-kserve-self-signed-certs\"" Oct 30 16:34:10.173605 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:10.173574 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18b72648-f523-405e-b5cd-1266ff13f8f2-tmp-dir\") pod \"premium-w0-f3503b-kserve-7c85fbb6bc-rv642\" (UID: \"18b72648-f523-405e-b5cd-1266ff13f8f2\") " pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" Oct 30 16:34:10.173731 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:10.173616 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18b72648-f523-405e-b5cd-1266ff13f8f2-dshm\") pod \"premium-w0-f3503b-kserve-7c85fbb6bc-rv642\" (UID: \"18b72648-f523-405e-b5cd-1266ff13f8f2\") " pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" Oct 30 16:34:10.173731 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:10.173647 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18b72648-f523-405e-b5cd-1266ff13f8f2-tls-certs\") pod \"premium-w0-f3503b-kserve-7c85fbb6bc-rv642\" (UID: \"18b72648-f523-405e-b5cd-1266ff13f8f2\") " pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" Oct 30 16:34:10.173731 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:10.173679 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18b72648-f523-405e-b5cd-1266ff13f8f2-home\") pod \"premium-w0-f3503b-kserve-7c85fbb6bc-rv642\" (UID: \"18b72648-f523-405e-b5cd-1266ff13f8f2\") " pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" Oct 30 16:34:10.173937 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:10.173760 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbzwm\" (UniqueName: \"kubernetes.io/projected/18b72648-f523-405e-b5cd-1266ff13f8f2-kube-api-access-vbzwm\") pod \"premium-w0-f3503b-kserve-7c85fbb6bc-rv642\" (UID: \"18b72648-f523-405e-b5cd-1266ff13f8f2\") " pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" Oct 30 16:34:10.173937 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:10.173899 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18b72648-f523-405e-b5cd-1266ff13f8f2-model-cache\") pod \"premium-w0-f3503b-kserve-7c85fbb6bc-rv642\" (UID: \"18b72648-f523-405e-b5cd-1266ff13f8f2\") " pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" Oct 30 16:34:10.274747 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:10.274707 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18b72648-f523-405e-b5cd-1266ff13f8f2-tls-certs\") pod \"premium-w0-f3503b-kserve-7c85fbb6bc-rv642\" (UID: \"18b72648-f523-405e-b5cd-1266ff13f8f2\") " pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" Oct 30 16:34:10.274747 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:10.274748 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18b72648-f523-405e-b5cd-1266ff13f8f2-home\") pod \"premium-w0-f3503b-kserve-7c85fbb6bc-rv642\" (UID: \"18b72648-f523-405e-b5cd-1266ff13f8f2\") " pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" Oct 30 16:34:10.275020 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:10.274767 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbzwm\" (UniqueName: \"kubernetes.io/projected/18b72648-f523-405e-b5cd-1266ff13f8f2-kube-api-access-vbzwm\") pod \"premium-w0-f3503b-kserve-7c85fbb6bc-rv642\" (UID: \"18b72648-f523-405e-b5cd-1266ff13f8f2\") " pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" Oct 30 16:34:10.275020 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:10.274837 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18b72648-f523-405e-b5cd-1266ff13f8f2-model-cache\") pod \"premium-w0-f3503b-kserve-7c85fbb6bc-rv642\" (UID: \"18b72648-f523-405e-b5cd-1266ff13f8f2\") " pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" Oct 30 16:34:10.275020 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:10.274919 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18b72648-f523-405e-b5cd-1266ff13f8f2-tmp-dir\") pod \"premium-w0-f3503b-kserve-7c85fbb6bc-rv642\" (UID: \"18b72648-f523-405e-b5cd-1266ff13f8f2\") " pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" Oct 30 16:34:10.275020 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:10.274952 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18b72648-f523-405e-b5cd-1266ff13f8f2-dshm\") pod \"premium-w0-f3503b-kserve-7c85fbb6bc-rv642\" (UID: \"18b72648-f523-405e-b5cd-1266ff13f8f2\") " pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" Oct 30 16:34:10.275232 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:10.275199 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18b72648-f523-405e-b5cd-1266ff13f8f2-home\") pod \"premium-w0-f3503b-kserve-7c85fbb6bc-rv642\" (UID: \"18b72648-f523-405e-b5cd-1266ff13f8f2\") " pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" Oct 30 16:34:10.275292 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:10.275271 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18b72648-f523-405e-b5cd-1266ff13f8f2-tmp-dir\") pod \"premium-w0-f3503b-kserve-7c85fbb6bc-rv642\" (UID: \"18b72648-f523-405e-b5cd-1266ff13f8f2\") " pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" Oct 30 16:34:10.275576 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:10.275557 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18b72648-f523-405e-b5cd-1266ff13f8f2-model-cache\") pod \"premium-w0-f3503b-kserve-7c85fbb6bc-rv642\" (UID: \"18b72648-f523-405e-b5cd-1266ff13f8f2\") " pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" Oct 30 16:34:10.277354 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:10.277331 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18b72648-f523-405e-b5cd-1266ff13f8f2-dshm\") pod \"premium-w0-f3503b-kserve-7c85fbb6bc-rv642\" (UID: \"18b72648-f523-405e-b5cd-1266ff13f8f2\") " pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" Oct 30 16:34:10.278327 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:10.278308 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18b72648-f523-405e-b5cd-1266ff13f8f2-tls-certs\") pod \"premium-w0-f3503b-kserve-7c85fbb6bc-rv642\" (UID: \"18b72648-f523-405e-b5cd-1266ff13f8f2\") " pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" Oct 30 16:34:10.298483 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:10.298450 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbzwm\" (UniqueName: \"kubernetes.io/projected/18b72648-f523-405e-b5cd-1266ff13f8f2-kube-api-access-vbzwm\") pod \"premium-w0-f3503b-kserve-7c85fbb6bc-rv642\" (UID: \"18b72648-f523-405e-b5cd-1266ff13f8f2\") " pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" Oct 30 16:34:10.426109 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:10.426073 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" Oct 30 16:34:10.731834 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:10.731744 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642"] Oct 30 16:34:10.737037 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:34:10.736992 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod18b72648_f523_405e_b5cd_1266ff13f8f2.slice/crio-78379a38696f47352d213ee97a9c0232e0a811be5cafd9248036df3681c8c79d WatchSource:0}: Error finding container 78379a38696f47352d213ee97a9c0232e0a811be5cafd9248036df3681c8c79d: Status 404 returned error can't find the container with id 78379a38696f47352d213ee97a9c0232e0a811be5cafd9248036df3681c8c79d Oct 30 16:34:10.844684 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:10.844650 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" event={"ID":"18b72648-f523-405e-b5cd-1266ff13f8f2","Type":"ContainerStarted","Data":"29896e9cf036eebbca0f710d59c4b8f028c45b9a194c56e479f32a5625dd9fec"} Oct 30 16:34:10.844934 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:10.844692 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" event={"ID":"18b72648-f523-405e-b5cd-1266ff13f8f2","Type":"ContainerStarted","Data":"78379a38696f47352d213ee97a9c0232e0a811be5cafd9248036df3681c8c79d"} Oct 30 16:34:10.894380 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:10.894323 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" podStartSLOduration=0.894309806 podStartE2EDuration="894.309806ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:34:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:34:10.893848384 +0000 UTC m=+678.121023481" watchObservedRunningTime="2026-10-30 16:34:10.894309806 +0000 UTC m=+678.121484905" Oct 30 16:34:11.426200 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:11.426167 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" Oct 30 16:34:14.769432 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:14.769398 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 16:34:20.427126 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:20.427090 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" Oct 30 16:34:20.440816 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:20.440775 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" Oct 30 16:34:20.920739 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:20.920709 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" Oct 30 16:34:26.469723 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:26.469688 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd"] Oct 30 16:34:26.480348 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:26.480324 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" Oct 30 16:34:26.495104 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:26.495071 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd"] Oct 30 16:34:26.503702 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:26.503677 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-a77b48\"/\"premium-w1-a77b48-kserve-self-signed-certs\"" Oct 30 16:34:26.552480 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:26.552442 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt22l\" (UniqueName: \"kubernetes.io/projected/0eadc5d4-98c3-4bf2-b3c2-f73fe249182f-kube-api-access-tt22l\") pod \"premium-w1-a77b48-kserve-844856b7c4-rfxxd\" (UID: \"0eadc5d4-98c3-4bf2-b3c2-f73fe249182f\") " pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" Oct 30 16:34:26.552662 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:26.552584 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0eadc5d4-98c3-4bf2-b3c2-f73fe249182f-tls-certs\") pod \"premium-w1-a77b48-kserve-844856b7c4-rfxxd\" (UID: \"0eadc5d4-98c3-4bf2-b3c2-f73fe249182f\") " pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" Oct 30 16:34:26.552662 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:26.552650 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0eadc5d4-98c3-4bf2-b3c2-f73fe249182f-home\") pod \"premium-w1-a77b48-kserve-844856b7c4-rfxxd\" (UID: \"0eadc5d4-98c3-4bf2-b3c2-f73fe249182f\") " pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" Oct 30 16:34:26.552748 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:26.552726 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0eadc5d4-98c3-4bf2-b3c2-f73fe249182f-model-cache\") pod \"premium-w1-a77b48-kserve-844856b7c4-rfxxd\" (UID: \"0eadc5d4-98c3-4bf2-b3c2-f73fe249182f\") " pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" Oct 30 16:34:26.552829 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:26.552760 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0eadc5d4-98c3-4bf2-b3c2-f73fe249182f-dshm\") pod \"premium-w1-a77b48-kserve-844856b7c4-rfxxd\" (UID: \"0eadc5d4-98c3-4bf2-b3c2-f73fe249182f\") " pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" Oct 30 16:34:26.552899 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:26.552835 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0eadc5d4-98c3-4bf2-b3c2-f73fe249182f-tmp-dir\") pod \"premium-w1-a77b48-kserve-844856b7c4-rfxxd\" (UID: \"0eadc5d4-98c3-4bf2-b3c2-f73fe249182f\") " pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" Oct 30 16:34:26.654234 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:26.654184 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0eadc5d4-98c3-4bf2-b3c2-f73fe249182f-tls-certs\") pod \"premium-w1-a77b48-kserve-844856b7c4-rfxxd\" (UID: \"0eadc5d4-98c3-4bf2-b3c2-f73fe249182f\") " pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" Oct 30 16:34:26.654434 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:26.654256 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0eadc5d4-98c3-4bf2-b3c2-f73fe249182f-home\") pod \"premium-w1-a77b48-kserve-844856b7c4-rfxxd\" (UID: \"0eadc5d4-98c3-4bf2-b3c2-f73fe249182f\") " pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" Oct 30 16:34:26.654434 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:26.654300 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0eadc5d4-98c3-4bf2-b3c2-f73fe249182f-model-cache\") pod \"premium-w1-a77b48-kserve-844856b7c4-rfxxd\" (UID: \"0eadc5d4-98c3-4bf2-b3c2-f73fe249182f\") " pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" Oct 30 16:34:26.654434 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:26.654328 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0eadc5d4-98c3-4bf2-b3c2-f73fe249182f-dshm\") pod \"premium-w1-a77b48-kserve-844856b7c4-rfxxd\" (UID: \"0eadc5d4-98c3-4bf2-b3c2-f73fe249182f\") " pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" Oct 30 16:34:26.654434 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:26.654382 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0eadc5d4-98c3-4bf2-b3c2-f73fe249182f-tmp-dir\") pod \"premium-w1-a77b48-kserve-844856b7c4-rfxxd\" (UID: \"0eadc5d4-98c3-4bf2-b3c2-f73fe249182f\") " pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" Oct 30 16:34:26.654717 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:26.654470 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt22l\" (UniqueName: \"kubernetes.io/projected/0eadc5d4-98c3-4bf2-b3c2-f73fe249182f-kube-api-access-tt22l\") pod \"premium-w1-a77b48-kserve-844856b7c4-rfxxd\" (UID: \"0eadc5d4-98c3-4bf2-b3c2-f73fe249182f\") " pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" Oct 30 16:34:26.655504 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:26.655472 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0eadc5d4-98c3-4bf2-b3c2-f73fe249182f-tmp-dir\") pod \"premium-w1-a77b48-kserve-844856b7c4-rfxxd\" (UID: \"0eadc5d4-98c3-4bf2-b3c2-f73fe249182f\") " pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" Oct 30 16:34:26.655773 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:26.655744 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0eadc5d4-98c3-4bf2-b3c2-f73fe249182f-home\") pod \"premium-w1-a77b48-kserve-844856b7c4-rfxxd\" (UID: \"0eadc5d4-98c3-4bf2-b3c2-f73fe249182f\") " pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" Oct 30 16:34:26.655973 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:26.655951 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0eadc5d4-98c3-4bf2-b3c2-f73fe249182f-model-cache\") pod \"premium-w1-a77b48-kserve-844856b7c4-rfxxd\" (UID: \"0eadc5d4-98c3-4bf2-b3c2-f73fe249182f\") " pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" Oct 30 16:34:26.657340 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:26.657315 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0eadc5d4-98c3-4bf2-b3c2-f73fe249182f-dshm\") pod \"premium-w1-a77b48-kserve-844856b7c4-rfxxd\" (UID: \"0eadc5d4-98c3-4bf2-b3c2-f73fe249182f\") " pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" Oct 30 16:34:26.657608 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:26.657588 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0eadc5d4-98c3-4bf2-b3c2-f73fe249182f-tls-certs\") pod \"premium-w1-a77b48-kserve-844856b7c4-rfxxd\" (UID: \"0eadc5d4-98c3-4bf2-b3c2-f73fe249182f\") " pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" Oct 30 16:34:26.677616 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:26.677587 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt22l\" (UniqueName: \"kubernetes.io/projected/0eadc5d4-98c3-4bf2-b3c2-f73fe249182f-kube-api-access-tt22l\") pod \"premium-w1-a77b48-kserve-844856b7c4-rfxxd\" (UID: \"0eadc5d4-98c3-4bf2-b3c2-f73fe249182f\") " pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" Oct 30 16:34:26.795621 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:26.795530 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" Oct 30 16:34:27.115144 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:27.115114 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd"] Oct 30 16:34:27.117493 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:34:27.117463 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0eadc5d4_98c3_4bf2_b3c2_f73fe249182f.slice/crio-ed579b24529bb9a3949947f4de558316e24e67288fb3b5684bdee90762eea507 WatchSource:0}: Error finding container ed579b24529bb9a3949947f4de558316e24e67288fb3b5684bdee90762eea507: Status 404 returned error can't find the container with id ed579b24529bb9a3949947f4de558316e24e67288fb3b5684bdee90762eea507 Oct 30 16:34:27.976806 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:27.976357 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" event={"ID":"0eadc5d4-98c3-4bf2-b3c2-f73fe249182f","Type":"ContainerStarted","Data":"e6f1fd9815791f9f13012d2fb5965ab06f886f5e8993baf69e5419f175b5512a"} Oct 30 16:34:27.976806 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:27.976408 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" event={"ID":"0eadc5d4-98c3-4bf2-b3c2-f73fe249182f","Type":"ContainerStarted","Data":"ed579b24529bb9a3949947f4de558316e24e67288fb3b5684bdee90762eea507"} Oct 30 16:34:28.035540 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:28.035460 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" podStartSLOduration=2.03544107 podStartE2EDuration="2.03544107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:34:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:34:28.023234724 +0000 UTC m=+695.250409825" watchObservedRunningTime="2026-10-30 16:34:28.03544107 +0000 UTC m=+695.262616171" Oct 30 16:34:28.796381 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:28.796346 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" Oct 30 16:34:30.341506 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:34:30.341474 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 16:34:35.803151 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:35.803112 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 16:34:36.796012 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:36.795967 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" Oct 30 16:34:36.811985 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:36.811953 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" Oct 30 16:34:37.050356 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:37.050263 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" Oct 30 16:34:44.798761 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:44.798721 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47f5178c-v8mkn"] Oct 30 16:34:44.803235 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:44.803215 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f5178c-v8mkn" Oct 30 16:34:44.817635 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:44.817569 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:34:44.817843 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:44.817828 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 16:34:44.833162 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:44.833133 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47f5178c-v8mkn"] Oct 30 16:34:44.867128 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:44.867054 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d3e0d8b-2248-472d-8bbf-90504460b614-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47f5178c-v8mkn\" (UID: \"8d3e0d8b-2248-472d-8bbf-90504460b614\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f5178c-v8mkn" Oct 30 16:34:44.867311 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:44.867195 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d3e0d8b-2248-472d-8bbf-90504460b614-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47f5178c-v8mkn\" (UID: \"8d3e0d8b-2248-472d-8bbf-90504460b614\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f5178c-v8mkn" Oct 30 16:34:44.867411 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:44.867385 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trxnv\" (UniqueName: \"kubernetes.io/projected/8d3e0d8b-2248-472d-8bbf-90504460b614-kube-api-access-trxnv\") pod \"maas-api-revoke-sub-47f5178c-v8mkn\" (UID: \"8d3e0d8b-2248-472d-8bbf-90504460b614\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f5178c-v8mkn" Oct 30 16:34:44.917673 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:44.917642 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:34:44.917851 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:44.917692 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:34:44.971807 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:44.969445 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trxnv\" (UniqueName: \"kubernetes.io/projected/8d3e0d8b-2248-472d-8bbf-90504460b614-kube-api-access-trxnv\") pod \"maas-api-revoke-sub-47f5178c-v8mkn\" (UID: \"8d3e0d8b-2248-472d-8bbf-90504460b614\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f5178c-v8mkn" Oct 30 16:34:44.971807 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:44.969556 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d3e0d8b-2248-472d-8bbf-90504460b614-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47f5178c-v8mkn\" (UID: \"8d3e0d8b-2248-472d-8bbf-90504460b614\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f5178c-v8mkn" Oct 30 16:34:44.971807 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:44.969623 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d3e0d8b-2248-472d-8bbf-90504460b614-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47f5178c-v8mkn\" (UID: \"8d3e0d8b-2248-472d-8bbf-90504460b614\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f5178c-v8mkn" Oct 30 16:34:44.972747 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:44.972706 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d3e0d8b-2248-472d-8bbf-90504460b614-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47f5178c-v8mkn\" (UID: \"8d3e0d8b-2248-472d-8bbf-90504460b614\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f5178c-v8mkn" Oct 30 16:34:44.977864 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:44.977839 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d3e0d8b-2248-472d-8bbf-90504460b614-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47f5178c-v8mkn\" (UID: \"8d3e0d8b-2248-472d-8bbf-90504460b614\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f5178c-v8mkn" Oct 30 16:34:44.994659 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:44.994639 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trxnv\" (UniqueName: \"kubernetes.io/projected/8d3e0d8b-2248-472d-8bbf-90504460b614-kube-api-access-trxnv\") pod \"maas-api-revoke-sub-47f5178c-v8mkn\" (UID: \"8d3e0d8b-2248-472d-8bbf-90504460b614\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f5178c-v8mkn" Oct 30 16:34:45.119480 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:45.119406 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f5178c-v8mkn" Oct 30 16:34:45.437555 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:45.437524 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47f5178c-v8mkn"] Oct 30 16:34:45.440421 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:34:45.440380 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d3e0d8b_2248_472d_8bbf_90504460b614.slice/crio-09a3edf39748d6802441ae682acae39b0181bd67acc19ca02c0178324626b2aa WatchSource:0}: Error finding container 09a3edf39748d6802441ae682acae39b0181bd67acc19ca02c0178324626b2aa: Status 404 returned error can't find the container with id 09a3edf39748d6802441ae682acae39b0181bd67acc19ca02c0178324626b2aa Oct 30 16:34:46.105667 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:46.105573 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f5178c-v8mkn" event={"ID":"8d3e0d8b-2248-472d-8bbf-90504460b614","Type":"ContainerStarted","Data":"b2ca2aba6b860ae3720203ab373e415d3281529b8839249465b2a1df759ca9fe"} Oct 30 16:34:46.105667 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:46.105645 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f5178c-v8mkn" event={"ID":"8d3e0d8b-2248-472d-8bbf-90504460b614","Type":"ContainerStarted","Data":"09a3edf39748d6802441ae682acae39b0181bd67acc19ca02c0178324626b2aa"} Oct 30 16:34:46.146519 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:46.146461 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f5178c-v8mkn" podStartSLOduration=2.146443535 podStartE2EDuration="2.146443535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:34:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:34:46.145536787 +0000 UTC m=+713.372711888" watchObservedRunningTime="2026-10-30 16:34:46.146443535 +0000 UTC m=+713.373618634" Oct 30 16:34:47.117275 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:47.117234 4493 generic.go:356] "Generic (PLEG): container finished" podID="8d3e0d8b-2248-472d-8bbf-90504460b614" containerID="b2ca2aba6b860ae3720203ab373e415d3281529b8839249465b2a1df759ca9fe" exitCode=0 Oct 30 16:34:47.117683 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:47.117324 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f5178c-v8mkn" event={"ID":"8d3e0d8b-2248-472d-8bbf-90504460b614","Type":"ContainerDied","Data":"b2ca2aba6b860ae3720203ab373e415d3281529b8839249465b2a1df759ca9fe"} Oct 30 16:34:48.273448 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:48.273422 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f5178c-v8mkn" Oct 30 16:34:48.405345 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:48.405315 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d3e0d8b-2248-472d-8bbf-90504460b614-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d3e0d8b-2248-472d-8bbf-90504460b614-maas-api-xks-ca\") pod \"8d3e0d8b-2248-472d-8bbf-90504460b614\" (UID: \"8d3e0d8b-2248-472d-8bbf-90504460b614\") " Oct 30 16:34:48.405529 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:48.405378 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d3e0d8b-2248-472d-8bbf-90504460b614-kube-api-access-trxnv\" (UniqueName: \"kubernetes.io/projected/8d3e0d8b-2248-472d-8bbf-90504460b614-kube-api-access-trxnv\") pod \"8d3e0d8b-2248-472d-8bbf-90504460b614\" (UID: \"8d3e0d8b-2248-472d-8bbf-90504460b614\") " Oct 30 16:34:48.405529 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:48.405459 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d3e0d8b-2248-472d-8bbf-90504460b614-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d3e0d8b-2248-472d-8bbf-90504460b614-maas-api-openshift-ca\") pod \"8d3e0d8b-2248-472d-8bbf-90504460b614\" (UID: \"8d3e0d8b-2248-472d-8bbf-90504460b614\") " Oct 30 16:34:48.405850 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:48.405821 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d3e0d8b-2248-472d-8bbf-90504460b614-maas-api-openshift-ca" pod "8d3e0d8b-2248-472d-8bbf-90504460b614" (UID: "8d3e0d8b-2248-472d-8bbf-90504460b614"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:34:48.407626 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:48.407592 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d3e0d8b-2248-472d-8bbf-90504460b614-maas-api-xks-ca" pod "8d3e0d8b-2248-472d-8bbf-90504460b614" (UID: "8d3e0d8b-2248-472d-8bbf-90504460b614"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:34:48.407731 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:48.407673 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d3e0d8b-2248-472d-8bbf-90504460b614-kube-api-access-trxnv" pod "8d3e0d8b-2248-472d-8bbf-90504460b614" (UID: "8d3e0d8b-2248-472d-8bbf-90504460b614"). InnerVolumeSpecName "kube-api-access-trxnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:34:48.507194 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:48.507159 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d3e0d8b-2248-472d-8bbf-90504460b614-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:34:48.507329 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:48.507196 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d3e0d8b-2248-472d-8bbf-90504460b614-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:34:48.507329 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:48.507227 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trxnv\" (UniqueName: \"kubernetes.io/projected/8d3e0d8b-2248-472d-8bbf-90504460b614-kube-api-access-trxnv\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:34:49.133879 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:49.133843 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f5178c-v8mkn" event={"ID":"8d3e0d8b-2248-472d-8bbf-90504460b614","Type":"ContainerDied","Data":"09a3edf39748d6802441ae682acae39b0181bd67acc19ca02c0178324626b2aa"} Oct 30 16:34:49.133879 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:49.133870 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f5178c-v8mkn" Oct 30 16:34:49.134105 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:49.133905 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09a3edf39748d6802441ae682acae39b0181bd67acc19ca02c0178324626b2aa" Oct 30 16:34:49.690207 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:49.690166 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 16:34:50.916194 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:50.916136 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e296dac2-l6sr7"] Oct 30 16:34:50.916877 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:50.916856 4493 cpu_manager.go:414] "containerMap: removing container" podUID="8d3e0d8b-2248-472d-8bbf-90504460b614" containerName="revoke-keys" Oct 30 16:34:50.916967 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:50.916879 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d3e0d8b-2248-472d-8bbf-90504460b614" containerName="revoke-keys" Oct 30 16:34:50.917070 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:50.917055 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d3e0d8b-2248-472d-8bbf-90504460b614" containerName="revoke-keys" Oct 30 16:34:50.921825 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:50.921772 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e296dac2-l6sr7" Oct 30 16:34:50.939737 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:50.939712 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e296dac2-l6sr7"] Oct 30 16:34:50.950662 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:50.949092 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:34:50.950662 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:50.949180 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:34:50.950662 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:50.949211 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:34:50.950662 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:50.949348 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 16:34:51.052168 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:51.052131 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb7182a3-151d-42f6-ad4e-9ddce1ce44fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e296dac2-l6sr7\" (UID: \"eb7182a3-151d-42f6-ad4e-9ddce1ce44fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e296dac2-l6sr7" Oct 30 16:34:51.052348 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:51.052215 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8xr4\" (UniqueName: \"kubernetes.io/projected/eb7182a3-151d-42f6-ad4e-9ddce1ce44fe-kube-api-access-g8xr4\") pod \"maas-api-revoke-sub-e296dac2-l6sr7\" (UID: \"eb7182a3-151d-42f6-ad4e-9ddce1ce44fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e296dac2-l6sr7" Oct 30 16:34:51.052403 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:51.052379 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb7182a3-151d-42f6-ad4e-9ddce1ce44fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e296dac2-l6sr7\" (UID: \"eb7182a3-151d-42f6-ad4e-9ddce1ce44fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e296dac2-l6sr7" Oct 30 16:34:51.153192 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:51.153153 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb7182a3-151d-42f6-ad4e-9ddce1ce44fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e296dac2-l6sr7\" (UID: \"eb7182a3-151d-42f6-ad4e-9ddce1ce44fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e296dac2-l6sr7" Oct 30 16:34:51.153390 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:51.153229 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb7182a3-151d-42f6-ad4e-9ddce1ce44fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e296dac2-l6sr7\" (UID: \"eb7182a3-151d-42f6-ad4e-9ddce1ce44fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e296dac2-l6sr7" Oct 30 16:34:51.153390 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:51.153262 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8xr4\" (UniqueName: \"kubernetes.io/projected/eb7182a3-151d-42f6-ad4e-9ddce1ce44fe-kube-api-access-g8xr4\") pod \"maas-api-revoke-sub-e296dac2-l6sr7\" (UID: \"eb7182a3-151d-42f6-ad4e-9ddce1ce44fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e296dac2-l6sr7" Oct 30 16:34:51.154080 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:51.154053 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb7182a3-151d-42f6-ad4e-9ddce1ce44fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e296dac2-l6sr7\" (UID: \"eb7182a3-151d-42f6-ad4e-9ddce1ce44fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e296dac2-l6sr7" Oct 30 16:34:51.155897 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:51.155874 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb7182a3-151d-42f6-ad4e-9ddce1ce44fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e296dac2-l6sr7\" (UID: \"eb7182a3-151d-42f6-ad4e-9ddce1ce44fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e296dac2-l6sr7" Oct 30 16:34:51.177338 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:51.177217 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8xr4\" (UniqueName: \"kubernetes.io/projected/eb7182a3-151d-42f6-ad4e-9ddce1ce44fe-kube-api-access-g8xr4\") pod \"maas-api-revoke-sub-e296dac2-l6sr7\" (UID: \"eb7182a3-151d-42f6-ad4e-9ddce1ce44fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e296dac2-l6sr7" Oct 30 16:34:51.250758 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:51.250722 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e296dac2-l6sr7" Oct 30 16:34:51.565232 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:51.565142 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e296dac2-l6sr7"] Oct 30 16:34:52.158653 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:52.158614 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e296dac2-l6sr7" event={"ID":"eb7182a3-151d-42f6-ad4e-9ddce1ce44fe","Type":"ContainerStarted","Data":"6c052f30460205f0b4c9bb45d4698085e5ecd45e2068fcf6f4b9ab3f9f5a5c25"} Oct 30 16:34:52.159093 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:52.158672 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e296dac2-l6sr7" event={"ID":"eb7182a3-151d-42f6-ad4e-9ddce1ce44fe","Type":"ContainerStarted","Data":"d0d96c8dc20d7ac190d6c95f3b3b351ec848b4617adfebf2efddc12f7b01b2de"} Oct 30 16:34:52.197933 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:52.197864 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e296dac2-l6sr7" podStartSLOduration=2.197844241 podStartE2EDuration="2.197844241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:34:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:34:52.197627432 +0000 UTC m=+719.424802553" watchObservedRunningTime="2026-10-30 16:34:52.197844241 +0000 UTC m=+719.425019342" Oct 30 16:34:53.166166 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:53.166133 4493 generic.go:356] "Generic (PLEG): container finished" podID="eb7182a3-151d-42f6-ad4e-9ddce1ce44fe" containerID="6c052f30460205f0b4c9bb45d4698085e5ecd45e2068fcf6f4b9ab3f9f5a5c25" exitCode=0 Oct 30 16:34:53.166570 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:53.166223 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e296dac2-l6sr7" event={"ID":"eb7182a3-151d-42f6-ad4e-9ddce1ce44fe","Type":"ContainerDied","Data":"6c052f30460205f0b4c9bb45d4698085e5ecd45e2068fcf6f4b9ab3f9f5a5c25"} Oct 30 16:34:54.315627 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:54.315601 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e296dac2-l6sr7" Oct 30 16:34:54.394309 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:54.394278 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb7182a3-151d-42f6-ad4e-9ddce1ce44fe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb7182a3-151d-42f6-ad4e-9ddce1ce44fe-maas-api-openshift-ca\") pod \"eb7182a3-151d-42f6-ad4e-9ddce1ce44fe\" (UID: \"eb7182a3-151d-42f6-ad4e-9ddce1ce44fe\") " Oct 30 16:34:54.394469 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:54.394375 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb7182a3-151d-42f6-ad4e-9ddce1ce44fe-kube-api-access-g8xr4\" (UniqueName: \"kubernetes.io/projected/eb7182a3-151d-42f6-ad4e-9ddce1ce44fe-kube-api-access-g8xr4\") pod \"eb7182a3-151d-42f6-ad4e-9ddce1ce44fe\" (UID: \"eb7182a3-151d-42f6-ad4e-9ddce1ce44fe\") " Oct 30 16:34:54.394519 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:54.394482 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb7182a3-151d-42f6-ad4e-9ddce1ce44fe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb7182a3-151d-42f6-ad4e-9ddce1ce44fe-maas-api-xks-ca\") pod \"eb7182a3-151d-42f6-ad4e-9ddce1ce44fe\" (UID: \"eb7182a3-151d-42f6-ad4e-9ddce1ce44fe\") " Oct 30 16:34:54.394673 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:54.394643 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb7182a3-151d-42f6-ad4e-9ddce1ce44fe-maas-api-openshift-ca" pod "eb7182a3-151d-42f6-ad4e-9ddce1ce44fe" (UID: "eb7182a3-151d-42f6-ad4e-9ddce1ce44fe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:34:54.396523 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:54.396493 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb7182a3-151d-42f6-ad4e-9ddce1ce44fe-maas-api-xks-ca" pod "eb7182a3-151d-42f6-ad4e-9ddce1ce44fe" (UID: "eb7182a3-151d-42f6-ad4e-9ddce1ce44fe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:34:54.396631 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:54.396520 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb7182a3-151d-42f6-ad4e-9ddce1ce44fe-kube-api-access-g8xr4" pod "eb7182a3-151d-42f6-ad4e-9ddce1ce44fe" (UID: "eb7182a3-151d-42f6-ad4e-9ddce1ce44fe"). InnerVolumeSpecName "kube-api-access-g8xr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:34:54.496264 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:54.496185 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8xr4\" (UniqueName: \"kubernetes.io/projected/eb7182a3-151d-42f6-ad4e-9ddce1ce44fe-kube-api-access-g8xr4\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:34:54.496264 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:54.496213 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb7182a3-151d-42f6-ad4e-9ddce1ce44fe-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:34:54.496264 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:54.496224 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb7182a3-151d-42f6-ad4e-9ddce1ce44fe-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:34:55.178859 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:55.178819 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e296dac2-l6sr7" Oct 30 16:34:55.178859 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:55.178840 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e296dac2-l6sr7" event={"ID":"eb7182a3-151d-42f6-ad4e-9ddce1ce44fe","Type":"ContainerDied","Data":"d0d96c8dc20d7ac190d6c95f3b3b351ec848b4617adfebf2efddc12f7b01b2de"} Oct 30 16:34:55.179071 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:55.178893 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0d96c8dc20d7ac190d6c95f3b3b351ec848b4617adfebf2efddc12f7b01b2de" Oct 30 16:34:55.224168 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:55.224095 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ca6bcd5-t8r4g"] Oct 30 16:34:55.224833 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:55.224814 4493 cpu_manager.go:414] "containerMap: removing container" podUID="eb7182a3-151d-42f6-ad4e-9ddce1ce44fe" containerName="revoke-keys" Oct 30 16:34:55.224877 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:55.224836 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb7182a3-151d-42f6-ad4e-9ddce1ce44fe" containerName="revoke-keys" Oct 30 16:34:55.225024 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:55.225013 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb7182a3-151d-42f6-ad4e-9ddce1ce44fe" containerName="revoke-keys" Oct 30 16:34:55.228697 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:55.228670 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ca6bcd5-t8r4g" Oct 30 16:34:55.250246 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:55.250210 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ca6bcd5-t8r4g"] Oct 30 16:34:55.250553 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:55.250325 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:34:55.250553 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:55.250417 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 16:34:55.250553 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:55.250451 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:34:55.250766 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:55.250457 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:34:55.405063 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:55.405029 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72ed491e-d203-4084-ac8a-dc459c564583-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ca6bcd5-t8r4g\" (UID: \"72ed491e-d203-4084-ac8a-dc459c564583\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ca6bcd5-t8r4g" Oct 30 16:34:55.405517 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:55.405084 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8bg6\" (UniqueName: \"kubernetes.io/projected/72ed491e-d203-4084-ac8a-dc459c564583-kube-api-access-t8bg6\") pod \"maas-api-revoke-sub-7ca6bcd5-t8r4g\" (UID: \"72ed491e-d203-4084-ac8a-dc459c564583\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ca6bcd5-t8r4g" Oct 30 16:34:55.405517 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:55.405260 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72ed491e-d203-4084-ac8a-dc459c564583-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ca6bcd5-t8r4g\" (UID: \"72ed491e-d203-4084-ac8a-dc459c564583\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ca6bcd5-t8r4g" Oct 30 16:34:55.506448 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:55.506347 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72ed491e-d203-4084-ac8a-dc459c564583-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ca6bcd5-t8r4g\" (UID: \"72ed491e-d203-4084-ac8a-dc459c564583\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ca6bcd5-t8r4g" Oct 30 16:34:55.506626 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:55.506454 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72ed491e-d203-4084-ac8a-dc459c564583-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ca6bcd5-t8r4g\" (UID: \"72ed491e-d203-4084-ac8a-dc459c564583\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ca6bcd5-t8r4g" Oct 30 16:34:55.506626 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:55.506522 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8bg6\" (UniqueName: \"kubernetes.io/projected/72ed491e-d203-4084-ac8a-dc459c564583-kube-api-access-t8bg6\") pod \"maas-api-revoke-sub-7ca6bcd5-t8r4g\" (UID: \"72ed491e-d203-4084-ac8a-dc459c564583\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ca6bcd5-t8r4g" Oct 30 16:34:55.507089 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:55.507068 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72ed491e-d203-4084-ac8a-dc459c564583-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ca6bcd5-t8r4g\" (UID: \"72ed491e-d203-4084-ac8a-dc459c564583\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ca6bcd5-t8r4g" Oct 30 16:34:55.508941 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:55.508918 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72ed491e-d203-4084-ac8a-dc459c564583-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ca6bcd5-t8r4g\" (UID: \"72ed491e-d203-4084-ac8a-dc459c564583\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ca6bcd5-t8r4g" Oct 30 16:34:55.530180 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:55.530155 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8bg6\" (UniqueName: \"kubernetes.io/projected/72ed491e-d203-4084-ac8a-dc459c564583-kube-api-access-t8bg6\") pod \"maas-api-revoke-sub-7ca6bcd5-t8r4g\" (UID: \"72ed491e-d203-4084-ac8a-dc459c564583\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ca6bcd5-t8r4g" Oct 30 16:34:55.544021 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:55.543988 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ca6bcd5-t8r4g" Oct 30 16:34:55.851318 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:55.851295 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ca6bcd5-t8r4g"] Oct 30 16:34:55.854237 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:34:55.854207 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72ed491e_d203_4084_ac8a_dc459c564583.slice/crio-12682a5dad251f8b9ae7f072c4bc531e0a11101957493c365d9c85b361abf1fe WatchSource:0}: Error finding container 12682a5dad251f8b9ae7f072c4bc531e0a11101957493c365d9c85b361abf1fe: Status 404 returned error can't find the container with id 12682a5dad251f8b9ae7f072c4bc531e0a11101957493c365d9c85b361abf1fe Oct 30 16:34:56.186979 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:56.186947 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ca6bcd5-t8r4g" event={"ID":"72ed491e-d203-4084-ac8a-dc459c564583","Type":"ContainerStarted","Data":"74a246e88ffc9741d9f0371caab8edfcedae2c9c80d93d63ca3c91a1dda7ded0"} Oct 30 16:34:56.187157 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:56.187028 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ca6bcd5-t8r4g" event={"ID":"72ed491e-d203-4084-ac8a-dc459c564583","Type":"ContainerStarted","Data":"12682a5dad251f8b9ae7f072c4bc531e0a11101957493c365d9c85b361abf1fe"} Oct 30 16:34:56.231220 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:56.231155 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ca6bcd5-t8r4g" podStartSLOduration=1.231137719 podStartE2EDuration="1.231137719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:34:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:34:56.229687873 +0000 UTC m=+723.456862974" watchObservedRunningTime="2026-10-30 16:34:56.231137719 +0000 UTC m=+723.458312856" Oct 30 16:34:57.194894 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:57.194861 4493 generic.go:356] "Generic (PLEG): container finished" podID="72ed491e-d203-4084-ac8a-dc459c564583" containerID="74a246e88ffc9741d9f0371caab8edfcedae2c9c80d93d63ca3c91a1dda7ded0" exitCode=0 Oct 30 16:34:57.195410 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:57.194949 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ca6bcd5-t8r4g" event={"ID":"72ed491e-d203-4084-ac8a-dc459c564583","Type":"ContainerDied","Data":"74a246e88ffc9741d9f0371caab8edfcedae2c9c80d93d63ca3c91a1dda7ded0"} Oct 30 16:34:58.345008 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:58.344983 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ca6bcd5-t8r4g" Oct 30 16:34:58.442364 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:58.442337 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72ed491e-d203-4084-ac8a-dc459c564583-kube-api-access-t8bg6\" (UniqueName: \"kubernetes.io/projected/72ed491e-d203-4084-ac8a-dc459c564583-kube-api-access-t8bg6\") pod \"72ed491e-d203-4084-ac8a-dc459c564583\" (UID: \"72ed491e-d203-4084-ac8a-dc459c564583\") " Oct 30 16:34:58.442532 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:58.442375 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72ed491e-d203-4084-ac8a-dc459c564583-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72ed491e-d203-4084-ac8a-dc459c564583-maas-api-openshift-ca\") pod \"72ed491e-d203-4084-ac8a-dc459c564583\" (UID: \"72ed491e-d203-4084-ac8a-dc459c564583\") " Oct 30 16:34:58.442532 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:58.442429 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72ed491e-d203-4084-ac8a-dc459c564583-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72ed491e-d203-4084-ac8a-dc459c564583-maas-api-xks-ca\") pod \"72ed491e-d203-4084-ac8a-dc459c564583\" (UID: \"72ed491e-d203-4084-ac8a-dc459c564583\") " Oct 30 16:34:58.442804 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:58.442753 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72ed491e-d203-4084-ac8a-dc459c564583-maas-api-openshift-ca" pod "72ed491e-d203-4084-ac8a-dc459c564583" (UID: "72ed491e-d203-4084-ac8a-dc459c564583"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:34:58.444638 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:58.444610 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72ed491e-d203-4084-ac8a-dc459c564583-kube-api-access-t8bg6" pod "72ed491e-d203-4084-ac8a-dc459c564583" (UID: "72ed491e-d203-4084-ac8a-dc459c564583"). InnerVolumeSpecName "kube-api-access-t8bg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:34:58.444750 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:58.444689 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72ed491e-d203-4084-ac8a-dc459c564583-maas-api-xks-ca" pod "72ed491e-d203-4084-ac8a-dc459c564583" (UID: "72ed491e-d203-4084-ac8a-dc459c564583"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:34:58.543494 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:58.543420 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8bg6\" (UniqueName: \"kubernetes.io/projected/72ed491e-d203-4084-ac8a-dc459c564583-kube-api-access-t8bg6\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:34:58.543494 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:58.543448 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72ed491e-d203-4084-ac8a-dc459c564583-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:34:58.543494 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:58.543461 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72ed491e-d203-4084-ac8a-dc459c564583-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:34:59.214961 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:59.214887 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ca6bcd5-t8r4g" Oct 30 16:34:59.214961 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:59.214917 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ca6bcd5-t8r4g" event={"ID":"72ed491e-d203-4084-ac8a-dc459c564583","Type":"ContainerDied","Data":"12682a5dad251f8b9ae7f072c4bc531e0a11101957493c365d9c85b361abf1fe"} Oct 30 16:34:59.215136 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:34:59.214965 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12682a5dad251f8b9ae7f072c4bc531e0a11101957493c365d9c85b361abf1fe" Oct 30 16:35:00.142907 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:00.142871 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04fa06a9-wqm7n"] Oct 30 16:35:00.144520 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:00.144491 4493 cpu_manager.go:414] "containerMap: removing container" podUID="72ed491e-d203-4084-ac8a-dc459c564583" containerName="revoke-keys" Oct 30 16:35:00.144687 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:00.144673 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72ed491e-d203-4084-ac8a-dc459c564583" containerName="revoke-keys" Oct 30 16:35:00.145052 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:00.145035 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="72ed491e-d203-4084-ac8a-dc459c564583" containerName="revoke-keys" Oct 30 16:35:00.149737 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:00.149715 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04fa06a9-wqm7n" Oct 30 16:35:00.158188 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:00.158163 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmmxp\" (UniqueName: \"kubernetes.io/projected/f3f871ac-8b1f-4dc5-bb98-2c0f09507776-kube-api-access-pmmxp\") pod \"maas-api-revoke-sub-04fa06a9-wqm7n\" (UID: \"f3f871ac-8b1f-4dc5-bb98-2c0f09507776\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04fa06a9-wqm7n" Oct 30 16:35:00.158302 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:00.158205 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3f871ac-8b1f-4dc5-bb98-2c0f09507776-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04fa06a9-wqm7n\" (UID: \"f3f871ac-8b1f-4dc5-bb98-2c0f09507776\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04fa06a9-wqm7n" Oct 30 16:35:00.158302 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:00.158240 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3f871ac-8b1f-4dc5-bb98-2c0f09507776-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04fa06a9-wqm7n\" (UID: \"f3f871ac-8b1f-4dc5-bb98-2c0f09507776\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04fa06a9-wqm7n" Oct 30 16:35:00.175141 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:00.175113 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04fa06a9-wqm7n"] Oct 30 16:35:00.192237 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:00.192207 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:35:00.192562 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:00.192543 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 16:35:00.192718 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:00.192704 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:35:00.192986 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:00.192964 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:35:00.258807 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:00.258754 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmmxp\" (UniqueName: \"kubernetes.io/projected/f3f871ac-8b1f-4dc5-bb98-2c0f09507776-kube-api-access-pmmxp\") pod \"maas-api-revoke-sub-04fa06a9-wqm7n\" (UID: \"f3f871ac-8b1f-4dc5-bb98-2c0f09507776\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04fa06a9-wqm7n" Oct 30 16:35:00.258993 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:00.258825 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3f871ac-8b1f-4dc5-bb98-2c0f09507776-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04fa06a9-wqm7n\" (UID: \"f3f871ac-8b1f-4dc5-bb98-2c0f09507776\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04fa06a9-wqm7n" Oct 30 16:35:00.258993 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:00.258867 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3f871ac-8b1f-4dc5-bb98-2c0f09507776-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04fa06a9-wqm7n\" (UID: \"f3f871ac-8b1f-4dc5-bb98-2c0f09507776\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04fa06a9-wqm7n" Oct 30 16:35:00.259546 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:00.259522 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3f871ac-8b1f-4dc5-bb98-2c0f09507776-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04fa06a9-wqm7n\" (UID: \"f3f871ac-8b1f-4dc5-bb98-2c0f09507776\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04fa06a9-wqm7n" Oct 30 16:35:00.261232 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:00.261208 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3f871ac-8b1f-4dc5-bb98-2c0f09507776-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04fa06a9-wqm7n\" (UID: \"f3f871ac-8b1f-4dc5-bb98-2c0f09507776\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04fa06a9-wqm7n" Oct 30 16:35:00.282145 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:00.282116 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmmxp\" (UniqueName: \"kubernetes.io/projected/f3f871ac-8b1f-4dc5-bb98-2c0f09507776-kube-api-access-pmmxp\") pod \"maas-api-revoke-sub-04fa06a9-wqm7n\" (UID: \"f3f871ac-8b1f-4dc5-bb98-2c0f09507776\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04fa06a9-wqm7n" Oct 30 16:35:00.344433 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:35:00.344395 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 16:35:00.463959 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:00.463876 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04fa06a9-wqm7n" Oct 30 16:35:00.768693 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:00.768600 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04fa06a9-wqm7n"] Oct 30 16:35:00.772208 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:35:00.772175 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3f871ac_8b1f_4dc5_bb98_2c0f09507776.slice/crio-53bc0f57cce27bcf7df0deede624cde3e8a21656f9a74452a269e7e0dcb1189a WatchSource:0}: Error finding container 53bc0f57cce27bcf7df0deede624cde3e8a21656f9a74452a269e7e0dcb1189a: Status 404 returned error can't find the container with id 53bc0f57cce27bcf7df0deede624cde3e8a21656f9a74452a269e7e0dcb1189a Oct 30 16:35:01.232661 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:01.232623 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04fa06a9-wqm7n" event={"ID":"f3f871ac-8b1f-4dc5-bb98-2c0f09507776","Type":"ContainerStarted","Data":"5cf4f5028b4a353d38a9d8c0a7a3e79816f6e9aad7811844621d213526734126"} Oct 30 16:35:01.233090 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:01.232691 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04fa06a9-wqm7n" event={"ID":"f3f871ac-8b1f-4dc5-bb98-2c0f09507776","Type":"ContainerStarted","Data":"53bc0f57cce27bcf7df0deede624cde3e8a21656f9a74452a269e7e0dcb1189a"} Oct 30 16:35:01.275938 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:01.275847 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04fa06a9-wqm7n" podStartSLOduration=1.275832388 podStartE2EDuration="1.275832388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:35:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:35:01.273385585 +0000 UTC m=+728.500560690" watchObservedRunningTime="2026-10-30 16:35:01.275832388 +0000 UTC m=+728.503007488" Oct 30 16:35:01.839849 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:01.839817 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6040abad-gwb7g"] Oct 30 16:35:01.844277 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:01.844253 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6040abad-gwb7g" Oct 30 16:35:01.865162 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:01.865036 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6040abad-gwb7g"] Oct 30 16:35:01.872360 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:01.871259 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec7bcd15-b4d3-4e0f-beec-a8d843f006ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6040abad-gwb7g\" (UID: \"ec7bcd15-b4d3-4e0f-beec-a8d843f006ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6040abad-gwb7g" Oct 30 16:35:01.872360 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:01.871300 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd2nm\" (UniqueName: \"kubernetes.io/projected/ec7bcd15-b4d3-4e0f-beec-a8d843f006ac-kube-api-access-bd2nm\") pod \"maas-api-revoke-sub-6040abad-gwb7g\" (UID: \"ec7bcd15-b4d3-4e0f-beec-a8d843f006ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6040abad-gwb7g" Oct 30 16:35:01.872360 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:01.871485 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec7bcd15-b4d3-4e0f-beec-a8d843f006ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6040abad-gwb7g\" (UID: \"ec7bcd15-b4d3-4e0f-beec-a8d843f006ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6040abad-gwb7g" Oct 30 16:35:01.972848 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:01.972809 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec7bcd15-b4d3-4e0f-beec-a8d843f006ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6040abad-gwb7g\" (UID: \"ec7bcd15-b4d3-4e0f-beec-a8d843f006ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6040abad-gwb7g" Oct 30 16:35:01.972848 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:01.972852 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd2nm\" (UniqueName: \"kubernetes.io/projected/ec7bcd15-b4d3-4e0f-beec-a8d843f006ac-kube-api-access-bd2nm\") pod \"maas-api-revoke-sub-6040abad-gwb7g\" (UID: \"ec7bcd15-b4d3-4e0f-beec-a8d843f006ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6040abad-gwb7g" Oct 30 16:35:01.973060 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:01.972951 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec7bcd15-b4d3-4e0f-beec-a8d843f006ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6040abad-gwb7g\" (UID: \"ec7bcd15-b4d3-4e0f-beec-a8d843f006ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6040abad-gwb7g" Oct 30 16:35:01.973562 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:01.973537 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec7bcd15-b4d3-4e0f-beec-a8d843f006ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6040abad-gwb7g\" (UID: \"ec7bcd15-b4d3-4e0f-beec-a8d843f006ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6040abad-gwb7g" Oct 30 16:35:01.975264 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:01.975246 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec7bcd15-b4d3-4e0f-beec-a8d843f006ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6040abad-gwb7g\" (UID: \"ec7bcd15-b4d3-4e0f-beec-a8d843f006ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6040abad-gwb7g" Oct 30 16:35:01.996941 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:01.996915 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd2nm\" (UniqueName: \"kubernetes.io/projected/ec7bcd15-b4d3-4e0f-beec-a8d843f006ac-kube-api-access-bd2nm\") pod \"maas-api-revoke-sub-6040abad-gwb7g\" (UID: \"ec7bcd15-b4d3-4e0f-beec-a8d843f006ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6040abad-gwb7g" Oct 30 16:35:02.180716 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:02.180647 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6040abad-gwb7g" Oct 30 16:35:02.252759 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:02.246641 4493 generic.go:356] "Generic (PLEG): container finished" podID="f3f871ac-8b1f-4dc5-bb98-2c0f09507776" containerID="5cf4f5028b4a353d38a9d8c0a7a3e79816f6e9aad7811844621d213526734126" exitCode=0 Oct 30 16:35:02.252759 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:02.246738 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04fa06a9-wqm7n" event={"ID":"f3f871ac-8b1f-4dc5-bb98-2c0f09507776","Type":"ContainerDied","Data":"5cf4f5028b4a353d38a9d8c0a7a3e79816f6e9aad7811844621d213526734126"} Oct 30 16:35:02.488204 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:02.488179 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6040abad-gwb7g"] Oct 30 16:35:02.490046 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:35:02.490011 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec7bcd15_b4d3_4e0f_beec_a8d843f006ac.slice/crio-340a4968840bebf762e442ea4ea90fd5345d57a37887a6b6681a705087dc6dd9 WatchSource:0}: Error finding container 340a4968840bebf762e442ea4ea90fd5345d57a37887a6b6681a705087dc6dd9: Status 404 returned error can't find the container with id 340a4968840bebf762e442ea4ea90fd5345d57a37887a6b6681a705087dc6dd9 Oct 30 16:35:03.256671 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:03.256630 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6040abad-gwb7g" event={"ID":"ec7bcd15-b4d3-4e0f-beec-a8d843f006ac","Type":"ContainerStarted","Data":"fc18f23133a0447c7cff008ffaf82fabd47600cb400ab0cdbc0ab134cd631f82"} Oct 30 16:35:03.257138 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:03.256721 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6040abad-gwb7g" event={"ID":"ec7bcd15-b4d3-4e0f-beec-a8d843f006ac","Type":"ContainerStarted","Data":"340a4968840bebf762e442ea4ea90fd5345d57a37887a6b6681a705087dc6dd9"} Oct 30 16:35:03.309370 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:03.309309 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6040abad-gwb7g" podStartSLOduration=2.309290286 podStartE2EDuration="2.309290286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:35:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:35:03.306709203 +0000 UTC m=+730.533884304" watchObservedRunningTime="2026-10-30 16:35:03.309290286 +0000 UTC m=+730.536465387" Oct 30 16:35:03.407883 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:03.407838 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04fa06a9-wqm7n" Oct 30 16:35:03.491281 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:03.491255 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3f871ac-8b1f-4dc5-bb98-2c0f09507776-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3f871ac-8b1f-4dc5-bb98-2c0f09507776-maas-api-xks-ca\") pod \"f3f871ac-8b1f-4dc5-bb98-2c0f09507776\" (UID: \"f3f871ac-8b1f-4dc5-bb98-2c0f09507776\") " Oct 30 16:35:03.491410 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:03.491309 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3f871ac-8b1f-4dc5-bb98-2c0f09507776-kube-api-access-pmmxp\" (UniqueName: \"kubernetes.io/projected/f3f871ac-8b1f-4dc5-bb98-2c0f09507776-kube-api-access-pmmxp\") pod \"f3f871ac-8b1f-4dc5-bb98-2c0f09507776\" (UID: \"f3f871ac-8b1f-4dc5-bb98-2c0f09507776\") " Oct 30 16:35:03.491460 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:03.491442 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3f871ac-8b1f-4dc5-bb98-2c0f09507776-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3f871ac-8b1f-4dc5-bb98-2c0f09507776-maas-api-openshift-ca\") pod \"f3f871ac-8b1f-4dc5-bb98-2c0f09507776\" (UID: \"f3f871ac-8b1f-4dc5-bb98-2c0f09507776\") " Oct 30 16:35:03.491813 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:03.491770 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3f871ac-8b1f-4dc5-bb98-2c0f09507776-maas-api-openshift-ca" pod "f3f871ac-8b1f-4dc5-bb98-2c0f09507776" (UID: "f3f871ac-8b1f-4dc5-bb98-2c0f09507776"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:35:03.493520 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:03.493492 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3f871ac-8b1f-4dc5-bb98-2c0f09507776-maas-api-xks-ca" pod "f3f871ac-8b1f-4dc5-bb98-2c0f09507776" (UID: "f3f871ac-8b1f-4dc5-bb98-2c0f09507776"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:35:03.493638 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:03.493578 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3f871ac-8b1f-4dc5-bb98-2c0f09507776-kube-api-access-pmmxp" pod "f3f871ac-8b1f-4dc5-bb98-2c0f09507776" (UID: "f3f871ac-8b1f-4dc5-bb98-2c0f09507776"). InnerVolumeSpecName "kube-api-access-pmmxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:35:03.593087 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:03.592978 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3f871ac-8b1f-4dc5-bb98-2c0f09507776-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:35:03.593087 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:03.593014 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3f871ac-8b1f-4dc5-bb98-2c0f09507776-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:35:03.593087 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:03.593031 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmmxp\" (UniqueName: \"kubernetes.io/projected/f3f871ac-8b1f-4dc5-bb98-2c0f09507776-kube-api-access-pmmxp\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:35:04.263993 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:04.263963 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04fa06a9-wqm7n" Oct 30 16:35:04.264392 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:04.263971 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04fa06a9-wqm7n" event={"ID":"f3f871ac-8b1f-4dc5-bb98-2c0f09507776","Type":"ContainerDied","Data":"53bc0f57cce27bcf7df0deede624cde3e8a21656f9a74452a269e7e0dcb1189a"} Oct 30 16:35:04.264392 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:04.264081 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53bc0f57cce27bcf7df0deede624cde3e8a21656f9a74452a269e7e0dcb1189a" Oct 30 16:35:04.265509 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:04.265481 4493 generic.go:356] "Generic (PLEG): container finished" podID="ec7bcd15-b4d3-4e0f-beec-a8d843f006ac" containerID="fc18f23133a0447c7cff008ffaf82fabd47600cb400ab0cdbc0ab134cd631f82" exitCode=0 Oct 30 16:35:04.265622 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:04.265603 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6040abad-gwb7g" event={"ID":"ec7bcd15-b4d3-4e0f-beec-a8d843f006ac","Type":"ContainerDied","Data":"fc18f23133a0447c7cff008ffaf82fabd47600cb400ab0cdbc0ab134cd631f82"} Oct 30 16:35:05.342322 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:05.342293 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe1b2b03-v2nfc"] Oct 30 16:35:05.343047 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:05.343029 4493 cpu_manager.go:414] "containerMap: removing container" podUID="f3f871ac-8b1f-4dc5-bb98-2c0f09507776" containerName="revoke-keys" Oct 30 16:35:05.343108 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:05.343050 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3f871ac-8b1f-4dc5-bb98-2c0f09507776" containerName="revoke-keys" Oct 30 16:35:05.343168 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:05.343157 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3f871ac-8b1f-4dc5-bb98-2c0f09507776" containerName="revoke-keys" Oct 30 16:35:05.348800 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:05.348759 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1b2b03-v2nfc" Oct 30 16:35:05.371052 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:05.368414 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe1b2b03-v2nfc"] Oct 30 16:35:05.412153 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:05.412123 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjln9\" (UniqueName: \"kubernetes.io/projected/c866da74-9e9e-4e46-a33c-5f641f2dbb90-kube-api-access-kjln9\") pod \"maas-api-revoke-sub-fe1b2b03-v2nfc\" (UID: \"c866da74-9e9e-4e46-a33c-5f641f2dbb90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1b2b03-v2nfc" Oct 30 16:35:05.412308 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:05.412175 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c866da74-9e9e-4e46-a33c-5f641f2dbb90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe1b2b03-v2nfc\" (UID: \"c866da74-9e9e-4e46-a33c-5f641f2dbb90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1b2b03-v2nfc" Oct 30 16:35:05.412308 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:05.412241 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c866da74-9e9e-4e46-a33c-5f641f2dbb90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe1b2b03-v2nfc\" (UID: \"c866da74-9e9e-4e46-a33c-5f641f2dbb90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1b2b03-v2nfc" Oct 30 16:35:05.425447 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:05.425424 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6040abad-gwb7g" Oct 30 16:35:05.512799 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:05.512761 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec7bcd15-b4d3-4e0f-beec-a8d843f006ac-kube-api-access-bd2nm\" (UniqueName: \"kubernetes.io/projected/ec7bcd15-b4d3-4e0f-beec-a8d843f006ac-kube-api-access-bd2nm\") pod \"ec7bcd15-b4d3-4e0f-beec-a8d843f006ac\" (UID: \"ec7bcd15-b4d3-4e0f-beec-a8d843f006ac\") " Oct 30 16:35:05.512964 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:05.512856 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec7bcd15-b4d3-4e0f-beec-a8d843f006ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec7bcd15-b4d3-4e0f-beec-a8d843f006ac-maas-api-openshift-ca\") pod \"ec7bcd15-b4d3-4e0f-beec-a8d843f006ac\" (UID: \"ec7bcd15-b4d3-4e0f-beec-a8d843f006ac\") " Oct 30 16:35:05.512964 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:05.512937 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec7bcd15-b4d3-4e0f-beec-a8d843f006ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec7bcd15-b4d3-4e0f-beec-a8d843f006ac-maas-api-xks-ca\") pod \"ec7bcd15-b4d3-4e0f-beec-a8d843f006ac\" (UID: \"ec7bcd15-b4d3-4e0f-beec-a8d843f006ac\") " Oct 30 16:35:05.513139 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:05.513121 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjln9\" (UniqueName: \"kubernetes.io/projected/c866da74-9e9e-4e46-a33c-5f641f2dbb90-kube-api-access-kjln9\") pod \"maas-api-revoke-sub-fe1b2b03-v2nfc\" (UID: \"c866da74-9e9e-4e46-a33c-5f641f2dbb90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1b2b03-v2nfc" Oct 30 16:35:05.513190 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:05.513169 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c866da74-9e9e-4e46-a33c-5f641f2dbb90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe1b2b03-v2nfc\" (UID: \"c866da74-9e9e-4e46-a33c-5f641f2dbb90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1b2b03-v2nfc" Oct 30 16:35:05.513239 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:05.513192 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c866da74-9e9e-4e46-a33c-5f641f2dbb90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe1b2b03-v2nfc\" (UID: \"c866da74-9e9e-4e46-a33c-5f641f2dbb90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1b2b03-v2nfc" Oct 30 16:35:05.513354 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:05.513299 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec7bcd15-b4d3-4e0f-beec-a8d843f006ac-maas-api-openshift-ca" pod "ec7bcd15-b4d3-4e0f-beec-a8d843f006ac" (UID: "ec7bcd15-b4d3-4e0f-beec-a8d843f006ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:35:05.514170 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:05.514142 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c866da74-9e9e-4e46-a33c-5f641f2dbb90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe1b2b03-v2nfc\" (UID: \"c866da74-9e9e-4e46-a33c-5f641f2dbb90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1b2b03-v2nfc" Oct 30 16:35:05.515176 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:05.515146 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec7bcd15-b4d3-4e0f-beec-a8d843f006ac-kube-api-access-bd2nm" pod "ec7bcd15-b4d3-4e0f-beec-a8d843f006ac" (UID: "ec7bcd15-b4d3-4e0f-beec-a8d843f006ac"). InnerVolumeSpecName "kube-api-access-bd2nm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:35:05.515281 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:05.515238 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec7bcd15-b4d3-4e0f-beec-a8d843f006ac-maas-api-xks-ca" pod "ec7bcd15-b4d3-4e0f-beec-a8d843f006ac" (UID: "ec7bcd15-b4d3-4e0f-beec-a8d843f006ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:35:05.515533 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:05.515514 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c866da74-9e9e-4e46-a33c-5f641f2dbb90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe1b2b03-v2nfc\" (UID: \"c866da74-9e9e-4e46-a33c-5f641f2dbb90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1b2b03-v2nfc" Oct 30 16:35:05.536536 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:05.536487 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjln9\" (UniqueName: \"kubernetes.io/projected/c866da74-9e9e-4e46-a33c-5f641f2dbb90-kube-api-access-kjln9\") pod \"maas-api-revoke-sub-fe1b2b03-v2nfc\" (UID: \"c866da74-9e9e-4e46-a33c-5f641f2dbb90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1b2b03-v2nfc" Oct 30 16:35:05.614427 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:05.614398 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec7bcd15-b4d3-4e0f-beec-a8d843f006ac-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:35:05.614427 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:05.614424 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bd2nm\" (UniqueName: \"kubernetes.io/projected/ec7bcd15-b4d3-4e0f-beec-a8d843f006ac-kube-api-access-bd2nm\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:35:05.614629 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:05.614434 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec7bcd15-b4d3-4e0f-beec-a8d843f006ac-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:35:05.665453 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:05.665421 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1b2b03-v2nfc" Oct 30 16:35:05.973083 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:05.973043 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe1b2b03-v2nfc"] Oct 30 16:35:05.974731 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:35:05.974704 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc866da74_9e9e_4e46_a33c_5f641f2dbb90.slice/crio-e7f0b648cd267ff6e8278a50573d14d3a9ba4ace81a9665292ed387bade14f8b WatchSource:0}: Error finding container e7f0b648cd267ff6e8278a50573d14d3a9ba4ace81a9665292ed387bade14f8b: Status 404 returned error can't find the container with id e7f0b648cd267ff6e8278a50573d14d3a9ba4ace81a9665292ed387bade14f8b Oct 30 16:35:06.281300 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:06.281264 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1b2b03-v2nfc" event={"ID":"c866da74-9e9e-4e46-a33c-5f641f2dbb90","Type":"ContainerStarted","Data":"e7f0b648cd267ff6e8278a50573d14d3a9ba4ace81a9665292ed387bade14f8b"} Oct 30 16:35:06.282586 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:06.282561 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6040abad-gwb7g" event={"ID":"ec7bcd15-b4d3-4e0f-beec-a8d843f006ac","Type":"ContainerDied","Data":"340a4968840bebf762e442ea4ea90fd5345d57a37887a6b6681a705087dc6dd9"} Oct 30 16:35:06.282708 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:06.282608 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="340a4968840bebf762e442ea4ea90fd5345d57a37887a6b6681a705087dc6dd9" Oct 30 16:35:06.282708 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:06.282565 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6040abad-gwb7g" Oct 30 16:35:07.290370 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:07.290286 4493 generic.go:356] "Generic (PLEG): container finished" podID="c866da74-9e9e-4e46-a33c-5f641f2dbb90" containerID="3490bd6648a1e9d6c90087151a9eda007f84fa620eb7a7120bbf7131fd3b4325" exitCode=0 Oct 30 16:35:07.290734 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:07.290430 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1b2b03-v2nfc" event={"ID":"c866da74-9e9e-4e46-a33c-5f641f2dbb90","Type":"ContainerDied","Data":"3490bd6648a1e9d6c90087151a9eda007f84fa620eb7a7120bbf7131fd3b4325"} Oct 30 16:35:08.437871 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:08.437843 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1b2b03-v2nfc" Oct 30 16:35:08.547449 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:08.547356 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c866da74-9e9e-4e46-a33c-5f641f2dbb90-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c866da74-9e9e-4e46-a33c-5f641f2dbb90-maas-api-openshift-ca\") pod \"c866da74-9e9e-4e46-a33c-5f641f2dbb90\" (UID: \"c866da74-9e9e-4e46-a33c-5f641f2dbb90\") " Oct 30 16:35:08.547606 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:08.547460 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c866da74-9e9e-4e46-a33c-5f641f2dbb90-kube-api-access-kjln9\" (UniqueName: \"kubernetes.io/projected/c866da74-9e9e-4e46-a33c-5f641f2dbb90-kube-api-access-kjln9\") pod \"c866da74-9e9e-4e46-a33c-5f641f2dbb90\" (UID: \"c866da74-9e9e-4e46-a33c-5f641f2dbb90\") " Oct 30 16:35:08.547606 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:08.547509 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c866da74-9e9e-4e46-a33c-5f641f2dbb90-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c866da74-9e9e-4e46-a33c-5f641f2dbb90-maas-api-xks-ca\") pod \"c866da74-9e9e-4e46-a33c-5f641f2dbb90\" (UID: \"c866da74-9e9e-4e46-a33c-5f641f2dbb90\") " Oct 30 16:35:08.547742 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:08.547715 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c866da74-9e9e-4e46-a33c-5f641f2dbb90-maas-api-openshift-ca" pod "c866da74-9e9e-4e46-a33c-5f641f2dbb90" (UID: "c866da74-9e9e-4e46-a33c-5f641f2dbb90"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:35:08.549712 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:08.549687 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c866da74-9e9e-4e46-a33c-5f641f2dbb90-kube-api-access-kjln9" pod "c866da74-9e9e-4e46-a33c-5f641f2dbb90" (UID: "c866da74-9e9e-4e46-a33c-5f641f2dbb90"). InnerVolumeSpecName "kube-api-access-kjln9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:35:08.549874 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:08.549823 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c866da74-9e9e-4e46-a33c-5f641f2dbb90-maas-api-xks-ca" pod "c866da74-9e9e-4e46-a33c-5f641f2dbb90" (UID: "c866da74-9e9e-4e46-a33c-5f641f2dbb90"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:35:08.649190 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:08.649161 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c866da74-9e9e-4e46-a33c-5f641f2dbb90-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:35:08.649190 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:08.649188 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c866da74-9e9e-4e46-a33c-5f641f2dbb90-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:35:08.649378 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:08.649202 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjln9\" (UniqueName: \"kubernetes.io/projected/c866da74-9e9e-4e46-a33c-5f641f2dbb90-kube-api-access-kjln9\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:35:09.306032 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:09.305948 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1b2b03-v2nfc" Oct 30 16:35:09.306180 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:09.305947 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1b2b03-v2nfc" event={"ID":"c866da74-9e9e-4e46-a33c-5f641f2dbb90","Type":"ContainerDied","Data":"e7f0b648cd267ff6e8278a50573d14d3a9ba4ace81a9665292ed387bade14f8b"} Oct 30 16:35:09.306180 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:09.306091 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7f0b648cd267ff6e8278a50573d14d3a9ba4ace81a9665292ed387bade14f8b" Oct 30 16:35:13.071772 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:13.071734 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn"] Oct 30 16:35:13.072210 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:13.072174 4493 cpu_manager.go:414] "containerMap: removing container" podUID="ec7bcd15-b4d3-4e0f-beec-a8d843f006ac" containerName="revoke-keys" Oct 30 16:35:13.072210 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:13.072187 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec7bcd15-b4d3-4e0f-beec-a8d843f006ac" containerName="revoke-keys" Oct 30 16:35:13.072210 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:13.072205 4493 cpu_manager.go:414] "containerMap: removing container" podUID="c866da74-9e9e-4e46-a33c-5f641f2dbb90" containerName="revoke-keys" Oct 30 16:35:13.072210 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:13.072211 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c866da74-9e9e-4e46-a33c-5f641f2dbb90" containerName="revoke-keys" Oct 30 16:35:13.072336 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:13.072294 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec7bcd15-b4d3-4e0f-beec-a8d843f006ac" containerName="revoke-keys" Oct 30 16:35:13.072336 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:13.072304 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="c866da74-9e9e-4e46-a33c-5f641f2dbb90" containerName="revoke-keys" Oct 30 16:35:13.075699 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:13.075679 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn" Oct 30 16:35:13.090757 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:13.090734 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58f6b168-jvvbc"] Oct 30 16:35:13.093896 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:13.093878 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-18519c-openshift-default-dockercfg-nqftk\"" Oct 30 16:35:13.094211 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:13.094196 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f6b168-jvvbc" Oct 30 16:35:13.108662 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:13.108639 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:35:13.108769 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:13.108737 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:35:13.110086 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:13.110058 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn"] Oct 30 16:35:13.121387 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:13.121352 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 16:35:13.121627 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:13.121611 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:35:13.123098 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:13.123080 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58f6b168-jvvbc"] Oct 30 16:35:13.192559 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:13.192525 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84009ea2-157b-45c6-b4bd-6df55fc7c72d-istio-data\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn\" (UID: \"84009ea2-157b-45c6-b4bd-6df55fc7c72d\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn" Oct 30 16:35:13.192753 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:13.192580 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84009ea2-157b-45c6-b4bd-6df55fc7c72d-credential-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn\" (UID: \"84009ea2-157b-45c6-b4bd-6df55fc7c72d\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn" Oct 30 16:35:13.192753 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:13.192679 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84009ea2-157b-45c6-b4bd-6df55fc7c72d-workload-certs\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn\" (UID: \"84009ea2-157b-45c6-b4bd-6df55fc7c72d\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn" Oct 30 16:35:13.192753 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:13.192723 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84009ea2-157b-45c6-b4bd-6df55fc7c72d-istiod-ca-cert\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn\" (UID: \"84009ea2-157b-45c6-b4bd-6df55fc7c72d\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn" Oct 30 16:35:13.192946 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:13.192760 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84009ea2-157b-45c6-b4bd-6df55fc7c72d-istio-token\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn\" (UID: \"84009ea2-157b-45c6-b4bd-6df55fc7c72d\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn" Oct 30 16:35:13.192946 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:13.192777 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lq5x\" (UniqueName: \"kubernetes.io/projected/bed50be1-2f29-48f7-b2a2-2a02be02535c-kube-api-access-2lq5x\") pod \"maas-api-revoke-sub-58f6b168-jvvbc\" (UID: \"bed50be1-2f29-48f7-b2a2-2a02be02535c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f6b168-jvvbc" Oct 30 16:35:13.192946 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:13.192906 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kl95m\" (UniqueName: \"kubernetes.io/projected/84009ea2-157b-45c6-b4bd-6df55fc7c72d-kube-api-access-kl95m\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn\" (UID: \"84009ea2-157b-45c6-b4bd-6df55fc7c72d\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn" Oct 30 16:35:13.193077 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:13.192945 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bed50be1-2f29-48f7-b2a2-2a02be02535c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58f6b168-jvvbc\" (UID: \"bed50be1-2f29-48f7-b2a2-2a02be02535c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f6b168-jvvbc" Oct 30 16:35:13.193077 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:13.193031 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84009ea2-157b-45c6-b4bd-6df55fc7c72d-workload-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn\" (UID: \"84009ea2-157b-45c6-b4bd-6df55fc7c72d\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn" Oct 30 16:35:13.193077 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:13.193062 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84009ea2-157b-45c6-b4bd-6df55fc7c72d-istio-envoy\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn\" (UID: \"84009ea2-157b-45c6-b4bd-6df55fc7c72d\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn" Oct 30 16:35:13.193180 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:13.193110 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bed50be1-2f29-48f7-b2a2-2a02be02535c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58f6b168-jvvbc\" (UID: \"bed50be1-2f29-48f7-b2a2-2a02be02535c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f6b168-jvvbc" Oct 30 16:35:13.193180 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:13.193141 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84009ea2-157b-45c6-b4bd-6df55fc7c72d-istio-podinfo\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn\" (UID: \"84009ea2-157b-45c6-b4bd-6df55fc7c72d\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn" Oct 30 16:35:13.294136 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:13.294102 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84009ea2-157b-45c6-b4bd-6df55fc7c72d-workload-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn\" (UID: \"84009ea2-157b-45c6-b4bd-6df55fc7c72d\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn" Oct 30 16:35:13.294136 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:13.294138 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84009ea2-157b-45c6-b4bd-6df55fc7c72d-istio-envoy\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn\" (UID: \"84009ea2-157b-45c6-b4bd-6df55fc7c72d\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn" Oct 30 16:35:13.294396 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:13.294165 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bed50be1-2f29-48f7-b2a2-2a02be02535c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58f6b168-jvvbc\" (UID: \"bed50be1-2f29-48f7-b2a2-2a02be02535c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f6b168-jvvbc" Oct 30 16:35:13.294396 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:13.294189 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84009ea2-157b-45c6-b4bd-6df55fc7c72d-istio-podinfo\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn\" (UID: \"84009ea2-157b-45c6-b4bd-6df55fc7c72d\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn" Oct 30 16:35:13.294396 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:13.294264 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84009ea2-157b-45c6-b4bd-6df55fc7c72d-istio-data\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn\" (UID: \"84009ea2-157b-45c6-b4bd-6df55fc7c72d\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn" Oct 30 16:35:13.294560 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:13.294396 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84009ea2-157b-45c6-b4bd-6df55fc7c72d-credential-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn\" (UID: \"84009ea2-157b-45c6-b4bd-6df55fc7c72d\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn" Oct 30 16:35:13.294560 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:13.294444 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84009ea2-157b-45c6-b4bd-6df55fc7c72d-workload-certs\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn\" (UID: \"84009ea2-157b-45c6-b4bd-6df55fc7c72d\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn" Oct 30 16:35:13.294691 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:13.294666 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84009ea2-157b-45c6-b4bd-6df55fc7c72d-workload-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn\" (UID: \"84009ea2-157b-45c6-b4bd-6df55fc7c72d\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn" Oct 30 16:35:13.294771 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:13.294697 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84009ea2-157b-45c6-b4bd-6df55fc7c72d-istio-data\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn\" (UID: \"84009ea2-157b-45c6-b4bd-6df55fc7c72d\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn" Oct 30 16:35:13.294771 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:13.294722 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84009ea2-157b-45c6-b4bd-6df55fc7c72d-istiod-ca-cert\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn\" (UID: \"84009ea2-157b-45c6-b4bd-6df55fc7c72d\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn" Oct 30 16:35:13.294771 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:13.294744 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84009ea2-157b-45c6-b4bd-6df55fc7c72d-credential-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn\" (UID: \"84009ea2-157b-45c6-b4bd-6df55fc7c72d\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn" Oct 30 16:35:13.294771 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:13.294768 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84009ea2-157b-45c6-b4bd-6df55fc7c72d-istio-token\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn\" (UID: \"84009ea2-157b-45c6-b4bd-6df55fc7c72d\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn" Oct 30 16:35:13.295014 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:13.294818 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lq5x\" (UniqueName: \"kubernetes.io/projected/bed50be1-2f29-48f7-b2a2-2a02be02535c-kube-api-access-2lq5x\") pod \"maas-api-revoke-sub-58f6b168-jvvbc\" (UID: \"bed50be1-2f29-48f7-b2a2-2a02be02535c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f6b168-jvvbc" Oct 30 16:35:13.295014 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:13.294865 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl95m\" (UniqueName: \"kubernetes.io/projected/84009ea2-157b-45c6-b4bd-6df55fc7c72d-kube-api-access-kl95m\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn\" (UID: \"84009ea2-157b-45c6-b4bd-6df55fc7c72d\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn" Oct 30 16:35:13.295014 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:13.294890 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bed50be1-2f29-48f7-b2a2-2a02be02535c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58f6b168-jvvbc\" (UID: \"bed50be1-2f29-48f7-b2a2-2a02be02535c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f6b168-jvvbc" Oct 30 16:35:13.295343 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:13.294765 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84009ea2-157b-45c6-b4bd-6df55fc7c72d-workload-certs\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn\" (UID: \"84009ea2-157b-45c6-b4bd-6df55fc7c72d\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn" Oct 30 16:35:13.295557 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:13.295536 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84009ea2-157b-45c6-b4bd-6df55fc7c72d-istiod-ca-cert\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn\" (UID: \"84009ea2-157b-45c6-b4bd-6df55fc7c72d\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn" Oct 30 16:35:13.295724 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:13.295701 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bed50be1-2f29-48f7-b2a2-2a02be02535c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58f6b168-jvvbc\" (UID: \"bed50be1-2f29-48f7-b2a2-2a02be02535c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f6b168-jvvbc" Oct 30 16:35:13.296679 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:13.296651 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bed50be1-2f29-48f7-b2a2-2a02be02535c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58f6b168-jvvbc\" (UID: \"bed50be1-2f29-48f7-b2a2-2a02be02535c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f6b168-jvvbc" Oct 30 16:35:13.296809 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:13.296765 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84009ea2-157b-45c6-b4bd-6df55fc7c72d-istio-envoy\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn\" (UID: \"84009ea2-157b-45c6-b4bd-6df55fc7c72d\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn" Oct 30 16:35:13.296931 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:13.296913 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84009ea2-157b-45c6-b4bd-6df55fc7c72d-istio-podinfo\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn\" (UID: \"84009ea2-157b-45c6-b4bd-6df55fc7c72d\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn" Oct 30 16:35:13.318823 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:13.318768 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lq5x\" (UniqueName: \"kubernetes.io/projected/bed50be1-2f29-48f7-b2a2-2a02be02535c-kube-api-access-2lq5x\") pod \"maas-api-revoke-sub-58f6b168-jvvbc\" (UID: \"bed50be1-2f29-48f7-b2a2-2a02be02535c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f6b168-jvvbc" Oct 30 16:35:13.318968 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:13.318770 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl95m\" (UniqueName: \"kubernetes.io/projected/84009ea2-157b-45c6-b4bd-6df55fc7c72d-kube-api-access-kl95m\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn\" (UID: \"84009ea2-157b-45c6-b4bd-6df55fc7c72d\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn" Oct 30 16:35:13.323005 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:13.322949 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84009ea2-157b-45c6-b4bd-6df55fc7c72d-istio-token\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn\" (UID: \"84009ea2-157b-45c6-b4bd-6df55fc7c72d\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn" Oct 30 16:35:13.389026 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:13.389001 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn" Oct 30 16:35:13.407502 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:13.407475 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f6b168-jvvbc" Oct 30 16:35:13.722758 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:13.722726 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58f6b168-jvvbc"] Oct 30 16:35:13.723392 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:35:13.723366 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbed50be1_2f29_48f7_b2a2_2a02be02535c.slice/crio-8144b58866596f9b27938da3b316fec77de3bd13379bda3c06242b354432aade WatchSource:0}: Error finding container 8144b58866596f9b27938da3b316fec77de3bd13379bda3c06242b354432aade: Status 404 returned error can't find the container with id 8144b58866596f9b27938da3b316fec77de3bd13379bda3c06242b354432aade Oct 30 16:35:13.726319 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:13.726298 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn"] Oct 30 16:35:13.735687 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:35:13.735659 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84009ea2_157b_45c6_b4bd_6df55fc7c72d.slice/crio-3fa2d42ff03e1f548a6d0ea01bdf19510d2f3afa2fe170640c9bcc4322a34ca9 WatchSource:0}: Error finding container 3fa2d42ff03e1f548a6d0ea01bdf19510d2f3afa2fe170640c9bcc4322a34ca9: Status 404 returned error can't find the container with id 3fa2d42ff03e1f548a6d0ea01bdf19510d2f3afa2fe170640c9bcc4322a34ca9 Oct 30 16:35:13.738070 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:13.738036 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:35:13.738169 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:13.738113 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:35:13.738169 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:13.738154 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:35:14.341630 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:14.341531 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn" event={"ID":"84009ea2-157b-45c6-b4bd-6df55fc7c72d","Type":"ContainerStarted","Data":"8ebf076f93f55dae5e773e774ad3dc10673d4b61201803708b086c5436da423e"} Oct 30 16:35:14.341630 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:14.341591 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn" event={"ID":"84009ea2-157b-45c6-b4bd-6df55fc7c72d","Type":"ContainerStarted","Data":"3fa2d42ff03e1f548a6d0ea01bdf19510d2f3afa2fe170640c9bcc4322a34ca9"} Oct 30 16:35:14.343261 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:14.343232 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f6b168-jvvbc" event={"ID":"bed50be1-2f29-48f7-b2a2-2a02be02535c","Type":"ContainerStarted","Data":"b014b6db563656f1959b1e034e7c5433f2dbce62b1650d57521666a0035d5611"} Oct 30 16:35:14.343388 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:14.343282 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f6b168-jvvbc" event={"ID":"bed50be1-2f29-48f7-b2a2-2a02be02535c","Type":"ContainerStarted","Data":"8144b58866596f9b27938da3b316fec77de3bd13379bda3c06242b354432aade"} Oct 30 16:35:14.389845 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:14.389818 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn" Oct 30 16:35:14.389845 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:14.389820 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn" podStartSLOduration=1.3898013040000001 podStartE2EDuration="1.389801304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:35:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:35:14.386220155 +0000 UTC m=+741.613395254" watchObservedRunningTime="2026-10-30 16:35:14.389801304 +0000 UTC m=+741.616976406" Oct 30 16:35:14.391670 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:14.391644 4493 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 16:35:14.391825 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:14.391697 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn" podUID="84009ea2-157b-45c6-b4bd-6df55fc7c72d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 16:35:14.434810 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:14.433775 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f6b168-jvvbc" podStartSLOduration=1.433760837 podStartE2EDuration="1.433760837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:35:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:35:14.432001191 +0000 UTC m=+741.659176292" watchObservedRunningTime="2026-10-30 16:35:14.433760837 +0000 UTC m=+741.660935939" Oct 30 16:35:15.352938 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:15.352903 4493 generic.go:356] "Generic (PLEG): container finished" podID="bed50be1-2f29-48f7-b2a2-2a02be02535c" containerID="b014b6db563656f1959b1e034e7c5433f2dbce62b1650d57521666a0035d5611" exitCode=0 Oct 30 16:35:15.353338 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:15.352957 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f6b168-jvvbc" event={"ID":"bed50be1-2f29-48f7-b2a2-2a02be02535c","Type":"ContainerDied","Data":"b014b6db563656f1959b1e034e7c5433f2dbce62b1650d57521666a0035d5611"} Oct 30 16:35:15.389757 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:15.389727 4493 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 16:35:15.389901 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:15.389810 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn" podUID="84009ea2-157b-45c6-b4bd-6df55fc7c72d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 16:35:16.390410 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:16.390274 4493 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 16:35:16.390410 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:16.390349 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn" podUID="84009ea2-157b-45c6-b4bd-6df55fc7c72d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 16:35:16.562195 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:16.562170 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f6b168-jvvbc" Oct 30 16:35:16.667853 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:16.667816 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bed50be1-2f29-48f7-b2a2-2a02be02535c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bed50be1-2f29-48f7-b2a2-2a02be02535c-maas-api-openshift-ca\") pod \"bed50be1-2f29-48f7-b2a2-2a02be02535c\" (UID: \"bed50be1-2f29-48f7-b2a2-2a02be02535c\") " Oct 30 16:35:16.669230 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:16.669205 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bed50be1-2f29-48f7-b2a2-2a02be02535c-kube-api-access-2lq5x\" (UniqueName: \"kubernetes.io/projected/bed50be1-2f29-48f7-b2a2-2a02be02535c-kube-api-access-2lq5x\") pod \"bed50be1-2f29-48f7-b2a2-2a02be02535c\" (UID: \"bed50be1-2f29-48f7-b2a2-2a02be02535c\") " Oct 30 16:35:16.669884 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:16.669863 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bed50be1-2f29-48f7-b2a2-2a02be02535c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bed50be1-2f29-48f7-b2a2-2a02be02535c-maas-api-xks-ca\") pod \"bed50be1-2f29-48f7-b2a2-2a02be02535c\" (UID: \"bed50be1-2f29-48f7-b2a2-2a02be02535c\") " Oct 30 16:35:16.670422 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:16.669086 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bed50be1-2f29-48f7-b2a2-2a02be02535c-maas-api-openshift-ca" pod "bed50be1-2f29-48f7-b2a2-2a02be02535c" (UID: "bed50be1-2f29-48f7-b2a2-2a02be02535c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:35:16.670890 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:16.670869 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bed50be1-2f29-48f7-b2a2-2a02be02535c-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:35:16.673596 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:16.673573 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bed50be1-2f29-48f7-b2a2-2a02be02535c-maas-api-xks-ca" pod "bed50be1-2f29-48f7-b2a2-2a02be02535c" (UID: "bed50be1-2f29-48f7-b2a2-2a02be02535c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:35:16.673722 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:16.673668 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bed50be1-2f29-48f7-b2a2-2a02be02535c-kube-api-access-2lq5x" pod "bed50be1-2f29-48f7-b2a2-2a02be02535c" (UID: "bed50be1-2f29-48f7-b2a2-2a02be02535c"). InnerVolumeSpecName "kube-api-access-2lq5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:35:16.771718 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:16.771667 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lq5x\" (UniqueName: \"kubernetes.io/projected/bed50be1-2f29-48f7-b2a2-2a02be02535c-kube-api-access-2lq5x\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:35:16.771718 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:16.771693 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bed50be1-2f29-48f7-b2a2-2a02be02535c-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:35:17.370270 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:17.370188 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f6b168-jvvbc" event={"ID":"bed50be1-2f29-48f7-b2a2-2a02be02535c","Type":"ContainerDied","Data":"8144b58866596f9b27938da3b316fec77de3bd13379bda3c06242b354432aade"} Oct 30 16:35:17.370270 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:17.370245 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8144b58866596f9b27938da3b316fec77de3bd13379bda3c06242b354432aade" Oct 30 16:35:17.370566 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:17.370281 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58f6b168-jvvbc" Oct 30 16:35:17.389540 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:17.389509 4493 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 16:35:17.389685 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:17.389563 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn" podUID="84009ea2-157b-45c6-b4bd-6df55fc7c72d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 16:35:18.389984 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:18.389943 4493 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 16:35:18.390376 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:18.390011 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn" podUID="84009ea2-157b-45c6-b4bd-6df55fc7c72d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 16:35:19.389612 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:19.389564 4493 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 16:35:19.389841 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:19.389635 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn" podUID="84009ea2-157b-45c6-b4bd-6df55fc7c72d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 16:35:20.434342 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:20.434307 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn" Oct 30 16:35:20.435745 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:20.435720 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn" Oct 30 16:35:20.436872 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:20.436850 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn" Oct 30 16:35:20.783584 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:20.783496 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 16:35:23.522984 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:23.522945 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b57f3178-4jjhq"] Oct 30 16:35:23.523632 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:23.523608 4493 cpu_manager.go:414] "containerMap: removing container" podUID="bed50be1-2f29-48f7-b2a2-2a02be02535c" containerName="revoke-keys" Oct 30 16:35:23.523738 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:23.523634 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bed50be1-2f29-48f7-b2a2-2a02be02535c" containerName="revoke-keys" Oct 30 16:35:23.523831 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:23.523819 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="bed50be1-2f29-48f7-b2a2-2a02be02535c" containerName="revoke-keys" Oct 30 16:35:23.528385 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:23.528363 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b57f3178-4jjhq" Oct 30 16:35:23.557068 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:23.557045 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:35:23.557301 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:23.557275 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:35:23.558736 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:23.558720 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:35:23.559039 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:23.559010 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 16:35:23.561752 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:23.561723 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b57f3178-4jjhq"] Oct 30 16:35:23.661064 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:23.661021 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39f93bd8-f7dc-4ce9-9e91-5e012b71cc5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b57f3178-4jjhq\" (UID: \"39f93bd8-f7dc-4ce9-9e91-5e012b71cc5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b57f3178-4jjhq" Oct 30 16:35:23.661291 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:23.661198 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdppx\" (UniqueName: \"kubernetes.io/projected/39f93bd8-f7dc-4ce9-9e91-5e012b71cc5c-kube-api-access-fdppx\") pod \"maas-api-revoke-sub-b57f3178-4jjhq\" (UID: \"39f93bd8-f7dc-4ce9-9e91-5e012b71cc5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b57f3178-4jjhq" Oct 30 16:35:23.661291 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:23.661238 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39f93bd8-f7dc-4ce9-9e91-5e012b71cc5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b57f3178-4jjhq\" (UID: \"39f93bd8-f7dc-4ce9-9e91-5e012b71cc5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b57f3178-4jjhq" Oct 30 16:35:23.761991 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:23.761953 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdppx\" (UniqueName: \"kubernetes.io/projected/39f93bd8-f7dc-4ce9-9e91-5e012b71cc5c-kube-api-access-fdppx\") pod \"maas-api-revoke-sub-b57f3178-4jjhq\" (UID: \"39f93bd8-f7dc-4ce9-9e91-5e012b71cc5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b57f3178-4jjhq" Oct 30 16:35:23.761991 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:23.762001 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39f93bd8-f7dc-4ce9-9e91-5e012b71cc5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b57f3178-4jjhq\" (UID: \"39f93bd8-f7dc-4ce9-9e91-5e012b71cc5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b57f3178-4jjhq" Oct 30 16:35:23.762276 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:23.762054 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39f93bd8-f7dc-4ce9-9e91-5e012b71cc5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b57f3178-4jjhq\" (UID: \"39f93bd8-f7dc-4ce9-9e91-5e012b71cc5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b57f3178-4jjhq" Oct 30 16:35:23.762955 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:23.762929 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39f93bd8-f7dc-4ce9-9e91-5e012b71cc5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b57f3178-4jjhq\" (UID: \"39f93bd8-f7dc-4ce9-9e91-5e012b71cc5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b57f3178-4jjhq" Oct 30 16:35:23.765163 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:23.765142 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39f93bd8-f7dc-4ce9-9e91-5e012b71cc5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b57f3178-4jjhq\" (UID: \"39f93bd8-f7dc-4ce9-9e91-5e012b71cc5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b57f3178-4jjhq" Oct 30 16:35:23.785205 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:23.785146 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdppx\" (UniqueName: \"kubernetes.io/projected/39f93bd8-f7dc-4ce9-9e91-5e012b71cc5c-kube-api-access-fdppx\") pod \"maas-api-revoke-sub-b57f3178-4jjhq\" (UID: \"39f93bd8-f7dc-4ce9-9e91-5e012b71cc5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b57f3178-4jjhq" Oct 30 16:35:23.853170 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:23.853128 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b57f3178-4jjhq" Oct 30 16:35:24.170260 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:24.170234 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b57f3178-4jjhq"] Oct 30 16:35:24.172204 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:35:24.172170 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39f93bd8_f7dc_4ce9_9e91_5e012b71cc5c.slice/crio-6f763158e90b9381c583203424e655e5e6584842ace94b244415e47b20f2eb1f WatchSource:0}: Error finding container 6f763158e90b9381c583203424e655e5e6584842ace94b244415e47b20f2eb1f: Status 404 returned error can't find the container with id 6f763158e90b9381c583203424e655e5e6584842ace94b244415e47b20f2eb1f Oct 30 16:35:24.422316 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:24.422095 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b57f3178-4jjhq" event={"ID":"39f93bd8-f7dc-4ce9-9e91-5e012b71cc5c","Type":"ContainerStarted","Data":"6f763158e90b9381c583203424e655e5e6584842ace94b244415e47b20f2eb1f"} Oct 30 16:35:25.430162 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:25.430130 4493 generic.go:356] "Generic (PLEG): container finished" podID="39f93bd8-f7dc-4ce9-9e91-5e012b71cc5c" containerID="627984d734054ea6bf006e5a196d7fa6bb0df24a8dab8de33af980ee578c421c" exitCode=0 Oct 30 16:35:25.430573 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:25.430222 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b57f3178-4jjhq" event={"ID":"39f93bd8-f7dc-4ce9-9e91-5e012b71cc5c","Type":"ContainerDied","Data":"627984d734054ea6bf006e5a196d7fa6bb0df24a8dab8de33af980ee578c421c"} Oct 30 16:35:26.587798 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:26.587766 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b57f3178-4jjhq" Oct 30 16:35:26.692577 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:26.692491 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39f93bd8-f7dc-4ce9-9e91-5e012b71cc5c-kube-api-access-fdppx\" (UniqueName: \"kubernetes.io/projected/39f93bd8-f7dc-4ce9-9e91-5e012b71cc5c-kube-api-access-fdppx\") pod \"39f93bd8-f7dc-4ce9-9e91-5e012b71cc5c\" (UID: \"39f93bd8-f7dc-4ce9-9e91-5e012b71cc5c\") " Oct 30 16:35:26.692577 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:26.692556 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39f93bd8-f7dc-4ce9-9e91-5e012b71cc5c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39f93bd8-f7dc-4ce9-9e91-5e012b71cc5c-maas-api-xks-ca\") pod \"39f93bd8-f7dc-4ce9-9e91-5e012b71cc5c\" (UID: \"39f93bd8-f7dc-4ce9-9e91-5e012b71cc5c\") " Oct 30 16:35:26.692815 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:26.692592 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39f93bd8-f7dc-4ce9-9e91-5e012b71cc5c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39f93bd8-f7dc-4ce9-9e91-5e012b71cc5c-maas-api-openshift-ca\") pod \"39f93bd8-f7dc-4ce9-9e91-5e012b71cc5c\" (UID: \"39f93bd8-f7dc-4ce9-9e91-5e012b71cc5c\") " Oct 30 16:35:26.692921 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:26.692893 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39f93bd8-f7dc-4ce9-9e91-5e012b71cc5c-maas-api-openshift-ca" pod "39f93bd8-f7dc-4ce9-9e91-5e012b71cc5c" (UID: "39f93bd8-f7dc-4ce9-9e91-5e012b71cc5c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:35:26.693087 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:26.693068 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39f93bd8-f7dc-4ce9-9e91-5e012b71cc5c-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:35:26.694822 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:26.694777 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39f93bd8-f7dc-4ce9-9e91-5e012b71cc5c-maas-api-xks-ca" pod "39f93bd8-f7dc-4ce9-9e91-5e012b71cc5c" (UID: "39f93bd8-f7dc-4ce9-9e91-5e012b71cc5c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:35:26.694929 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:26.694844 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39f93bd8-f7dc-4ce9-9e91-5e012b71cc5c-kube-api-access-fdppx" pod "39f93bd8-f7dc-4ce9-9e91-5e012b71cc5c" (UID: "39f93bd8-f7dc-4ce9-9e91-5e012b71cc5c"). InnerVolumeSpecName "kube-api-access-fdppx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:35:26.793845 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:26.793779 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdppx\" (UniqueName: \"kubernetes.io/projected/39f93bd8-f7dc-4ce9-9e91-5e012b71cc5c-kube-api-access-fdppx\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:35:26.793845 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:26.793844 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39f93bd8-f7dc-4ce9-9e91-5e012b71cc5c-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:35:26.987512 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:26.987438 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-d9rhc"] Oct 30 16:35:26.987913 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:26.987900 4493 cpu_manager.go:414] "containerMap: removing container" podUID="39f93bd8-f7dc-4ce9-9e91-5e012b71cc5c" containerName="revoke-keys" Oct 30 16:35:26.987961 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:26.987914 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39f93bd8-f7dc-4ce9-9e91-5e012b71cc5c" containerName="revoke-keys" Oct 30 16:35:26.988005 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:26.987997 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="39f93bd8-f7dc-4ce9-9e91-5e012b71cc5c" containerName="revoke-keys" Oct 30 16:35:26.991308 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:26.991291 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-d9rhc" Oct 30 16:35:27.027082 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:27.027051 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-d9rhc"] Oct 30 16:35:27.097506 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:27.097471 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0476eb7d-91e3-4f24-bfbc-245775c8e93b-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-d9rhc\" (UID: \"0476eb7d-91e3-4f24-bfbc-245775c8e93b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-d9rhc" Oct 30 16:35:27.097506 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:27.097505 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0476eb7d-91e3-4f24-bfbc-245775c8e93b-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-d9rhc\" (UID: \"0476eb7d-91e3-4f24-bfbc-245775c8e93b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-d9rhc" Oct 30 16:35:27.097699 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:27.097537 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0476eb7d-91e3-4f24-bfbc-245775c8e93b-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-d9rhc\" (UID: \"0476eb7d-91e3-4f24-bfbc-245775c8e93b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-d9rhc" Oct 30 16:35:27.097735 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:27.097714 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0476eb7d-91e3-4f24-bfbc-245775c8e93b-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-d9rhc\" (UID: \"0476eb7d-91e3-4f24-bfbc-245775c8e93b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-d9rhc" Oct 30 16:35:27.097815 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:27.097778 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0476eb7d-91e3-4f24-bfbc-245775c8e93b-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-d9rhc\" (UID: \"0476eb7d-91e3-4f24-bfbc-245775c8e93b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-d9rhc" Oct 30 16:35:27.097866 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:27.097828 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0476eb7d-91e3-4f24-bfbc-245775c8e93b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-d9rhc\" (UID: \"0476eb7d-91e3-4f24-bfbc-245775c8e93b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-d9rhc" Oct 30 16:35:27.097934 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:27.097921 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0476eb7d-91e3-4f24-bfbc-245775c8e93b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-d9rhc\" (UID: \"0476eb7d-91e3-4f24-bfbc-245775c8e93b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-d9rhc" Oct 30 16:35:27.097966 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:27.097945 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0476eb7d-91e3-4f24-bfbc-245775c8e93b-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-d9rhc\" (UID: \"0476eb7d-91e3-4f24-bfbc-245775c8e93b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-d9rhc" Oct 30 16:35:27.098029 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:27.097965 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkhtm\" (UniqueName: \"kubernetes.io/projected/0476eb7d-91e3-4f24-bfbc-245775c8e93b-kube-api-access-fkhtm\") pod \"maas-default-gateway-openshift-default-7854949c9-d9rhc\" (UID: \"0476eb7d-91e3-4f24-bfbc-245775c8e93b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-d9rhc" Oct 30 16:35:27.199273 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:27.199241 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0476eb7d-91e3-4f24-bfbc-245775c8e93b-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-d9rhc\" (UID: \"0476eb7d-91e3-4f24-bfbc-245775c8e93b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-d9rhc" Oct 30 16:35:27.199273 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:27.199277 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0476eb7d-91e3-4f24-bfbc-245775c8e93b-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-d9rhc\" (UID: \"0476eb7d-91e3-4f24-bfbc-245775c8e93b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-d9rhc" Oct 30 16:35:27.199508 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:27.199295 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0476eb7d-91e3-4f24-bfbc-245775c8e93b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-d9rhc\" (UID: \"0476eb7d-91e3-4f24-bfbc-245775c8e93b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-d9rhc" Oct 30 16:35:27.199508 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:27.199323 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0476eb7d-91e3-4f24-bfbc-245775c8e93b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-d9rhc\" (UID: \"0476eb7d-91e3-4f24-bfbc-245775c8e93b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-d9rhc" Oct 30 16:35:27.199508 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:27.199340 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0476eb7d-91e3-4f24-bfbc-245775c8e93b-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-d9rhc\" (UID: \"0476eb7d-91e3-4f24-bfbc-245775c8e93b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-d9rhc" Oct 30 16:35:27.199508 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:27.199415 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkhtm\" (UniqueName: \"kubernetes.io/projected/0476eb7d-91e3-4f24-bfbc-245775c8e93b-kube-api-access-fkhtm\") pod \"maas-default-gateway-openshift-default-7854949c9-d9rhc\" (UID: \"0476eb7d-91e3-4f24-bfbc-245775c8e93b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-d9rhc" Oct 30 16:35:27.199721 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:27.199546 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0476eb7d-91e3-4f24-bfbc-245775c8e93b-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-d9rhc\" (UID: \"0476eb7d-91e3-4f24-bfbc-245775c8e93b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-d9rhc" Oct 30 16:35:27.199721 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:27.199577 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0476eb7d-91e3-4f24-bfbc-245775c8e93b-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-d9rhc\" (UID: \"0476eb7d-91e3-4f24-bfbc-245775c8e93b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-d9rhc" Oct 30 16:35:27.199721 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:27.199624 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0476eb7d-91e3-4f24-bfbc-245775c8e93b-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-d9rhc\" (UID: \"0476eb7d-91e3-4f24-bfbc-245775c8e93b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-d9rhc" Oct 30 16:35:27.199927 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:27.199742 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0476eb7d-91e3-4f24-bfbc-245775c8e93b-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-d9rhc\" (UID: \"0476eb7d-91e3-4f24-bfbc-245775c8e93b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-d9rhc" Oct 30 16:35:27.199927 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:27.199828 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0476eb7d-91e3-4f24-bfbc-245775c8e93b-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-d9rhc\" (UID: \"0476eb7d-91e3-4f24-bfbc-245775c8e93b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-d9rhc" Oct 30 16:35:27.200035 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:27.199999 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0476eb7d-91e3-4f24-bfbc-245775c8e93b-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-d9rhc\" (UID: \"0476eb7d-91e3-4f24-bfbc-245775c8e93b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-d9rhc" Oct 30 16:35:27.200273 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:27.200252 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0476eb7d-91e3-4f24-bfbc-245775c8e93b-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-d9rhc\" (UID: \"0476eb7d-91e3-4f24-bfbc-245775c8e93b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-d9rhc" Oct 30 16:35:27.200768 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:27.200747 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0476eb7d-91e3-4f24-bfbc-245775c8e93b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-d9rhc\" (UID: \"0476eb7d-91e3-4f24-bfbc-245775c8e93b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-d9rhc" Oct 30 16:35:27.201746 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:27.201717 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0476eb7d-91e3-4f24-bfbc-245775c8e93b-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-d9rhc\" (UID: \"0476eb7d-91e3-4f24-bfbc-245775c8e93b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-d9rhc" Oct 30 16:35:27.201989 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:27.201966 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0476eb7d-91e3-4f24-bfbc-245775c8e93b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-d9rhc\" (UID: \"0476eb7d-91e3-4f24-bfbc-245775c8e93b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-d9rhc" Oct 30 16:35:27.222937 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:27.222912 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0476eb7d-91e3-4f24-bfbc-245775c8e93b-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-d9rhc\" (UID: \"0476eb7d-91e3-4f24-bfbc-245775c8e93b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-d9rhc" Oct 30 16:35:27.223054 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:27.223022 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkhtm\" (UniqueName: \"kubernetes.io/projected/0476eb7d-91e3-4f24-bfbc-245775c8e93b-kube-api-access-fkhtm\") pod \"maas-default-gateway-openshift-default-7854949c9-d9rhc\" (UID: \"0476eb7d-91e3-4f24-bfbc-245775c8e93b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-d9rhc" Oct 30 16:35:27.304994 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:27.304908 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-d9rhc" Oct 30 16:35:27.444860 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:27.444827 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b57f3178-4jjhq" Oct 30 16:35:27.445000 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:27.444855 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b57f3178-4jjhq" event={"ID":"39f93bd8-f7dc-4ce9-9e91-5e012b71cc5c","Type":"ContainerDied","Data":"6f763158e90b9381c583203424e655e5e6584842ace94b244415e47b20f2eb1f"} Oct 30 16:35:27.445000 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:27.444907 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f763158e90b9381c583203424e655e5e6584842ace94b244415e47b20f2eb1f" Oct 30 16:35:27.626360 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:27.626329 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-d9rhc"] Oct 30 16:35:27.628441 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:35:27.628400 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0476eb7d_91e3_4f24_bfbc_245775c8e93b.slice/crio-6e33692044b6afe333d583e9f6f9bafef51cf3261a33bbc33fa06592dee278b3 WatchSource:0}: Error finding container 6e33692044b6afe333d583e9f6f9bafef51cf3261a33bbc33fa06592dee278b3: Status 404 returned error can't find the container with id 6e33692044b6afe333d583e9f6f9bafef51cf3261a33bbc33fa06592dee278b3 Oct 30 16:35:27.631009 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:27.630970 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:35:27.631128 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:27.631055 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:35:27.631128 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:27.631106 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:35:28.452470 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:28.452434 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-d9rhc" event={"ID":"0476eb7d-91e3-4f24-bfbc-245775c8e93b","Type":"ContainerStarted","Data":"97a8e4b2ca4d74a421648cb3113f545ac021c7a7ca3deead0f96b8539a6f5358"} Oct 30 16:35:28.452634 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:28.452488 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-d9rhc" event={"ID":"0476eb7d-91e3-4f24-bfbc-245775c8e93b","Type":"ContainerStarted","Data":"6e33692044b6afe333d583e9f6f9bafef51cf3261a33bbc33fa06592dee278b3"} Oct 30 16:35:28.529458 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:28.529381 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-d9rhc" podStartSLOduration=2.5293587459999998 podStartE2EDuration="2.529358746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:35:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:35:28.508401469 +0000 UTC m=+755.735576571" watchObservedRunningTime="2026-10-30 16:35:28.529358746 +0000 UTC m=+755.756533847" Oct 30 16:35:28.990114 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:28.990085 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl"] Oct 30 16:35:28.994215 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:28.994193 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl" Oct 30 16:35:29.037845 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:29.029863 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-f3503b-openshift-default-dockercfg-xgg9f\"" Oct 30 16:35:29.046817 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:29.043148 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl"] Oct 30 16:35:29.136196 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:29.136162 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c933c090-d160-449b-a3e3-75aec1c2ce36-istio-podinfo\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl\" (UID: \"c933c090-d160-449b-a3e3-75aec1c2ce36\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl" Oct 30 16:35:29.136409 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:29.136209 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c933c090-d160-449b-a3e3-75aec1c2ce36-credential-socket\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl\" (UID: \"c933c090-d160-449b-a3e3-75aec1c2ce36\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl" Oct 30 16:35:29.136409 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:29.136267 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c933c090-d160-449b-a3e3-75aec1c2ce36-istio-envoy\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl\" (UID: \"c933c090-d160-449b-a3e3-75aec1c2ce36\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl" Oct 30 16:35:29.136409 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:29.136390 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c933c090-d160-449b-a3e3-75aec1c2ce36-istio-data\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl\" (UID: \"c933c090-d160-449b-a3e3-75aec1c2ce36\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl" Oct 30 16:35:29.136591 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:29.136422 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c933c090-d160-449b-a3e3-75aec1c2ce36-istiod-ca-cert\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl\" (UID: \"c933c090-d160-449b-a3e3-75aec1c2ce36\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl" Oct 30 16:35:29.136591 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:29.136448 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c933c090-d160-449b-a3e3-75aec1c2ce36-istio-token\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl\" (UID: \"c933c090-d160-449b-a3e3-75aec1c2ce36\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl" Oct 30 16:35:29.136591 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:29.136480 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c933c090-d160-449b-a3e3-75aec1c2ce36-workload-certs\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl\" (UID: \"c933c090-d160-449b-a3e3-75aec1c2ce36\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl" Oct 30 16:35:29.136747 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:29.136678 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c933c090-d160-449b-a3e3-75aec1c2ce36-workload-socket\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl\" (UID: \"c933c090-d160-449b-a3e3-75aec1c2ce36\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl" Oct 30 16:35:29.136747 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:29.136714 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jmfw\" (UniqueName: \"kubernetes.io/projected/c933c090-d160-449b-a3e3-75aec1c2ce36-kube-api-access-6jmfw\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl\" (UID: \"c933c090-d160-449b-a3e3-75aec1c2ce36\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl" Oct 30 16:35:29.237507 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:29.237463 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c933c090-d160-449b-a3e3-75aec1c2ce36-istio-envoy\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl\" (UID: \"c933c090-d160-449b-a3e3-75aec1c2ce36\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl" Oct 30 16:35:29.237677 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:29.237531 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c933c090-d160-449b-a3e3-75aec1c2ce36-istio-data\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl\" (UID: \"c933c090-d160-449b-a3e3-75aec1c2ce36\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl" Oct 30 16:35:29.237677 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:29.237562 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c933c090-d160-449b-a3e3-75aec1c2ce36-istiod-ca-cert\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl\" (UID: \"c933c090-d160-449b-a3e3-75aec1c2ce36\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl" Oct 30 16:35:29.237677 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:29.237589 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c933c090-d160-449b-a3e3-75aec1c2ce36-istio-token\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl\" (UID: \"c933c090-d160-449b-a3e3-75aec1c2ce36\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl" Oct 30 16:35:29.237677 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:29.237620 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c933c090-d160-449b-a3e3-75aec1c2ce36-workload-certs\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl\" (UID: \"c933c090-d160-449b-a3e3-75aec1c2ce36\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl" Oct 30 16:35:29.237947 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:29.237689 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c933c090-d160-449b-a3e3-75aec1c2ce36-workload-socket\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl\" (UID: \"c933c090-d160-449b-a3e3-75aec1c2ce36\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl" Oct 30 16:35:29.237947 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:29.237717 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jmfw\" (UniqueName: \"kubernetes.io/projected/c933c090-d160-449b-a3e3-75aec1c2ce36-kube-api-access-6jmfw\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl\" (UID: \"c933c090-d160-449b-a3e3-75aec1c2ce36\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl" Oct 30 16:35:29.237947 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:29.237830 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c933c090-d160-449b-a3e3-75aec1c2ce36-istio-podinfo\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl\" (UID: \"c933c090-d160-449b-a3e3-75aec1c2ce36\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl" Oct 30 16:35:29.237947 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:29.237862 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c933c090-d160-449b-a3e3-75aec1c2ce36-credential-socket\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl\" (UID: \"c933c090-d160-449b-a3e3-75aec1c2ce36\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl" Oct 30 16:35:29.238335 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:29.238311 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c933c090-d160-449b-a3e3-75aec1c2ce36-credential-socket\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl\" (UID: \"c933c090-d160-449b-a3e3-75aec1c2ce36\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl" Oct 30 16:35:29.238739 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:29.238717 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c933c090-d160-449b-a3e3-75aec1c2ce36-workload-socket\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl\" (UID: \"c933c090-d160-449b-a3e3-75aec1c2ce36\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl" Oct 30 16:35:29.239141 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:29.239113 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c933c090-d160-449b-a3e3-75aec1c2ce36-workload-certs\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl\" (UID: \"c933c090-d160-449b-a3e3-75aec1c2ce36\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl" Oct 30 16:35:29.239260 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:29.239212 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c933c090-d160-449b-a3e3-75aec1c2ce36-istiod-ca-cert\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl\" (UID: \"c933c090-d160-449b-a3e3-75aec1c2ce36\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl" Oct 30 16:35:29.239329 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:29.239272 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c933c090-d160-449b-a3e3-75aec1c2ce36-istio-data\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl\" (UID: \"c933c090-d160-449b-a3e3-75aec1c2ce36\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl" Oct 30 16:35:29.241313 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:29.241248 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c933c090-d160-449b-a3e3-75aec1c2ce36-istio-envoy\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl\" (UID: \"c933c090-d160-449b-a3e3-75aec1c2ce36\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl" Oct 30 16:35:29.241495 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:29.241473 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c933c090-d160-449b-a3e3-75aec1c2ce36-istio-podinfo\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl\" (UID: \"c933c090-d160-449b-a3e3-75aec1c2ce36\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl" Oct 30 16:35:29.261145 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:29.261120 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c933c090-d160-449b-a3e3-75aec1c2ce36-istio-token\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl\" (UID: \"c933c090-d160-449b-a3e3-75aec1c2ce36\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl" Oct 30 16:35:29.261338 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:29.261317 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jmfw\" (UniqueName: \"kubernetes.io/projected/c933c090-d160-449b-a3e3-75aec1c2ce36-kube-api-access-6jmfw\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl\" (UID: \"c933c090-d160-449b-a3e3-75aec1c2ce36\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl" Oct 30 16:35:29.315808 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:29.315093 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-d9rhc" Oct 30 16:35:29.315808 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:29.315241 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-d9rhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 16:35:29.315808 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:29.315285 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-d9rhc" podUID="0476eb7d-91e3-4f24-bfbc-245775c8e93b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 16:35:29.335249 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:29.335091 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl" Oct 30 16:35:29.662712 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:29.662689 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl"] Oct 30 16:35:29.663883 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:35:29.663857 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc933c090_d160_449b_a3e3_75aec1c2ce36.slice/crio-c064721bf582f63b1bedf86448b2cebbd3740734a984b0f2e4cae3eda00a4525 WatchSource:0}: Error finding container c064721bf582f63b1bedf86448b2cebbd3740734a984b0f2e4cae3eda00a4525: Status 404 returned error can't find the container with id c064721bf582f63b1bedf86448b2cebbd3740734a984b0f2e4cae3eda00a4525 Oct 30 16:35:29.666191 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:29.666158 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:35:29.666247 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:29.666219 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:35:29.666295 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:29.666248 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:35:30.311812 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:30.310186 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-d9rhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 16:35:30.311812 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:30.310246 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-d9rhc" podUID="0476eb7d-91e3-4f24-bfbc-245775c8e93b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 16:35:30.360502 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:35:30.358355 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 16:35:30.469528 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:30.469492 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl" event={"ID":"c933c090-d160-449b-a3e3-75aec1c2ce36","Type":"ContainerStarted","Data":"59ee73106ddcb99e1eabac289f0ffbed1eb433e7c4f5b90bc5dd6d83ead195e8"} Oct 30 16:35:30.469746 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:30.469554 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl" event={"ID":"c933c090-d160-449b-a3e3-75aec1c2ce36","Type":"ContainerStarted","Data":"c064721bf582f63b1bedf86448b2cebbd3740734a984b0f2e4cae3eda00a4525"} Oct 30 16:35:30.531822 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:30.531743 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl" podStartSLOduration=2.531725037 podStartE2EDuration="2.531725037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:35:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:35:30.53026015 +0000 UTC m=+757.757435274" watchObservedRunningTime="2026-10-30 16:35:30.531725037 +0000 UTC m=+757.758900138" Oct 30 16:35:31.312810 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:31.308169 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-d9rhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 16:35:31.312810 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:31.308233 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-d9rhc" podUID="0476eb7d-91e3-4f24-bfbc-245775c8e93b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 16:35:31.338973 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:31.338932 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl" Oct 30 16:35:31.339368 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:31.339347 4493 patch_prober.go:29] interesting pod/e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 16:35:31.339566 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:31.339530 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl" podUID="c933c090-d160-449b-a3e3-75aec1c2ce36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 16:35:32.308824 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:32.308169 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-d9rhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 16:35:32.308824 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:32.308232 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-d9rhc" podUID="0476eb7d-91e3-4f24-bfbc-245775c8e93b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 16:35:32.338186 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:32.338152 4493 patch_prober.go:29] interesting pod/e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 16:35:32.338844 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:32.338758 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl" podUID="c933c090-d160-449b-a3e3-75aec1c2ce36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 16:35:33.308206 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:33.308167 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-d9rhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 16:35:33.308631 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:33.308599 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-d9rhc" podUID="0476eb7d-91e3-4f24-bfbc-245775c8e93b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 16:35:33.338184 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:33.338148 4493 patch_prober.go:29] interesting pod/e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 16:35:33.338440 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:33.338414 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl" podUID="c933c090-d160-449b-a3e3-75aec1c2ce36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 16:35:34.309863 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:34.308884 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-d9rhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 16:35:34.309863 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:34.308942 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-d9rhc" podUID="0476eb7d-91e3-4f24-bfbc-245775c8e93b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 16:35:34.337810 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:34.337265 4493 patch_prober.go:29] interesting pod/e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 16:35:34.337810 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:34.337337 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl" podUID="c933c090-d160-449b-a3e3-75aec1c2ce36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 16:35:35.311811 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:35.308155 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-d9rhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 16:35:35.311811 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:35.308218 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-d9rhc" podUID="0476eb7d-91e3-4f24-bfbc-245775c8e93b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 16:35:35.348927 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:35.348890 4493 patch_prober.go:29] interesting pod/e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 16:35:35.349190 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:35.349161 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl" podUID="c933c090-d160-449b-a3e3-75aec1c2ce36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 16:35:36.335581 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:36.335538 4493 patch_prober.go:29] interesting pod/e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 16:35:36.336114 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:36.335605 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl" podUID="c933c090-d160-449b-a3e3-75aec1c2ce36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 16:35:37.337687 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:37.337652 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-d9rhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:35:37.338296 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:37.338261 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-d9rhc" podUID="0476eb7d-91e3-4f24-bfbc-245775c8e93b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:35:37.373810 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:37.373112 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-d9rhc" Oct 30 16:35:37.378806 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:37.374460 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-d9rhc" Oct 30 16:35:37.384911 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:37.383485 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-d9rhc" Oct 30 16:35:38.310173 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:38.310138 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl" Oct 30 16:35:38.310821 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:38.310757 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl" Oct 30 16:35:38.312011 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:38.311989 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl" Oct 30 16:35:39.517319 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:39.517285 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a8fd3e5-rmwfd"] Oct 30 16:35:39.521111 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:39.521088 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a8fd3e5-rmwfd" Oct 30 16:35:39.535641 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:39.535619 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:35:39.553203 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:39.553174 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a8fd3e5-rmwfd"] Oct 30 16:35:39.553363 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:39.553293 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:35:39.553438 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:39.553378 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:35:39.553438 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:39.553428 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 16:35:39.571249 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:39.571207 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0768bd59-6767-4d80-9982-abed675e7c27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a8fd3e5-rmwfd\" (UID: \"0768bd59-6767-4d80-9982-abed675e7c27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a8fd3e5-rmwfd" Oct 30 16:35:39.571554 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:39.571537 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0768bd59-6767-4d80-9982-abed675e7c27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a8fd3e5-rmwfd\" (UID: \"0768bd59-6767-4d80-9982-abed675e7c27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a8fd3e5-rmwfd" Oct 30 16:35:39.571833 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:39.571807 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmpg8\" (UniqueName: \"kubernetes.io/projected/0768bd59-6767-4d80-9982-abed675e7c27-kube-api-access-qmpg8\") pod \"maas-api-revoke-sub-7a8fd3e5-rmwfd\" (UID: \"0768bd59-6767-4d80-9982-abed675e7c27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a8fd3e5-rmwfd" Oct 30 16:35:39.672695 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:39.672664 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmpg8\" (UniqueName: \"kubernetes.io/projected/0768bd59-6767-4d80-9982-abed675e7c27-kube-api-access-qmpg8\") pod \"maas-api-revoke-sub-7a8fd3e5-rmwfd\" (UID: \"0768bd59-6767-4d80-9982-abed675e7c27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a8fd3e5-rmwfd" Oct 30 16:35:39.672912 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:39.672749 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0768bd59-6767-4d80-9982-abed675e7c27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a8fd3e5-rmwfd\" (UID: \"0768bd59-6767-4d80-9982-abed675e7c27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a8fd3e5-rmwfd" Oct 30 16:35:39.672912 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:39.672854 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0768bd59-6767-4d80-9982-abed675e7c27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a8fd3e5-rmwfd\" (UID: \"0768bd59-6767-4d80-9982-abed675e7c27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a8fd3e5-rmwfd" Oct 30 16:35:39.673496 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:39.673471 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0768bd59-6767-4d80-9982-abed675e7c27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a8fd3e5-rmwfd\" (UID: \"0768bd59-6767-4d80-9982-abed675e7c27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a8fd3e5-rmwfd" Oct 30 16:35:39.675352 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:39.675329 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0768bd59-6767-4d80-9982-abed675e7c27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a8fd3e5-rmwfd\" (UID: \"0768bd59-6767-4d80-9982-abed675e7c27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a8fd3e5-rmwfd" Oct 30 16:35:39.696510 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:39.696487 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmpg8\" (UniqueName: \"kubernetes.io/projected/0768bd59-6767-4d80-9982-abed675e7c27-kube-api-access-qmpg8\") pod \"maas-api-revoke-sub-7a8fd3e5-rmwfd\" (UID: \"0768bd59-6767-4d80-9982-abed675e7c27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a8fd3e5-rmwfd" Oct 30 16:35:39.836075 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:39.835988 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a8fd3e5-rmwfd" Oct 30 16:35:40.147539 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:40.147508 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a8fd3e5-rmwfd"] Oct 30 16:35:40.149402 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:35:40.149372 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0768bd59_6767_4d80_9982_abed675e7c27.slice/crio-8ebd9b2f68479312ef1be1c5fdb64d2aacc5c39341bccc1d4a736b8e1ccc6b76 WatchSource:0}: Error finding container 8ebd9b2f68479312ef1be1c5fdb64d2aacc5c39341bccc1d4a736b8e1ccc6b76: Status 404 returned error can't find the container with id 8ebd9b2f68479312ef1be1c5fdb64d2aacc5c39341bccc1d4a736b8e1ccc6b76 Oct 30 16:35:40.607581 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:40.607529 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a8fd3e5-rmwfd" event={"ID":"0768bd59-6767-4d80-9982-abed675e7c27","Type":"ContainerStarted","Data":"484bc224d48b1f63815c3d0a06cf09c1ab3e30dd251ebfcbab34c7911eafa91e"} Oct 30 16:35:40.608052 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:40.607599 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a8fd3e5-rmwfd" event={"ID":"0768bd59-6767-4d80-9982-abed675e7c27","Type":"ContainerStarted","Data":"8ebd9b2f68479312ef1be1c5fdb64d2aacc5c39341bccc1d4a736b8e1ccc6b76"} Oct 30 16:35:40.649358 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:40.649305 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a8fd3e5-rmwfd" podStartSLOduration=1.649288656 podStartE2EDuration="1.649288656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:35:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:35:40.647036499 +0000 UTC m=+767.874211600" watchObservedRunningTime="2026-10-30 16:35:40.649288656 +0000 UTC m=+767.876463755" Oct 30 16:35:41.618630 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:41.618596 4493 generic.go:356] "Generic (PLEG): container finished" podID="0768bd59-6767-4d80-9982-abed675e7c27" containerID="484bc224d48b1f63815c3d0a06cf09c1ab3e30dd251ebfcbab34c7911eafa91e" exitCode=0 Oct 30 16:35:41.619049 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:41.618691 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a8fd3e5-rmwfd" event={"ID":"0768bd59-6767-4d80-9982-abed675e7c27","Type":"ContainerDied","Data":"484bc224d48b1f63815c3d0a06cf09c1ab3e30dd251ebfcbab34c7911eafa91e"} Oct 30 16:35:42.075826 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:42.075763 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6w2sv"] Oct 30 16:35:42.080003 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:42.079977 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6w2sv" Oct 30 16:35:42.113031 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:42.113002 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6w2sv"] Oct 30 16:35:42.187448 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:42.187412 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 16:35:42.198308 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:42.198279 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59ae5769-a6cf-444d-947e-313ff39785a5-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-6w2sv\" (UID: \"59ae5769-a6cf-444d-947e-313ff39785a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6w2sv" Oct 30 16:35:42.198308 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:42.198308 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59ae5769-a6cf-444d-947e-313ff39785a5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-6w2sv\" (UID: \"59ae5769-a6cf-444d-947e-313ff39785a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6w2sv" Oct 30 16:35:42.198474 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:42.198434 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59ae5769-a6cf-444d-947e-313ff39785a5-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-6w2sv\" (UID: \"59ae5769-a6cf-444d-947e-313ff39785a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6w2sv" Oct 30 16:35:42.198510 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:42.198478 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59ae5769-a6cf-444d-947e-313ff39785a5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-6w2sv\" (UID: \"59ae5769-a6cf-444d-947e-313ff39785a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6w2sv" Oct 30 16:35:42.198548 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:42.198530 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs98z\" (UniqueName: \"kubernetes.io/projected/59ae5769-a6cf-444d-947e-313ff39785a5-kube-api-access-zs98z\") pod \"maas-default-gateway-openshift-default-7854949c9-6w2sv\" (UID: \"59ae5769-a6cf-444d-947e-313ff39785a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6w2sv" Oct 30 16:35:42.198589 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:42.198572 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59ae5769-a6cf-444d-947e-313ff39785a5-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-6w2sv\" (UID: \"59ae5769-a6cf-444d-947e-313ff39785a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6w2sv" Oct 30 16:35:42.198638 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:42.198625 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59ae5769-a6cf-444d-947e-313ff39785a5-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-6w2sv\" (UID: \"59ae5769-a6cf-444d-947e-313ff39785a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6w2sv" Oct 30 16:35:42.198687 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:42.198675 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59ae5769-a6cf-444d-947e-313ff39785a5-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-6w2sv\" (UID: \"59ae5769-a6cf-444d-947e-313ff39785a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6w2sv" Oct 30 16:35:42.198721 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:42.198695 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59ae5769-a6cf-444d-947e-313ff39785a5-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-6w2sv\" (UID: \"59ae5769-a6cf-444d-947e-313ff39785a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6w2sv" Oct 30 16:35:42.299708 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:42.299664 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59ae5769-a6cf-444d-947e-313ff39785a5-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-6w2sv\" (UID: \"59ae5769-a6cf-444d-947e-313ff39785a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6w2sv" Oct 30 16:35:42.299926 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:42.299734 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59ae5769-a6cf-444d-947e-313ff39785a5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-6w2sv\" (UID: \"59ae5769-a6cf-444d-947e-313ff39785a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6w2sv" Oct 30 16:35:42.299926 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:42.299763 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs98z\" (UniqueName: \"kubernetes.io/projected/59ae5769-a6cf-444d-947e-313ff39785a5-kube-api-access-zs98z\") pod \"maas-default-gateway-openshift-default-7854949c9-6w2sv\" (UID: \"59ae5769-a6cf-444d-947e-313ff39785a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6w2sv" Oct 30 16:35:42.299926 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:42.299836 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59ae5769-a6cf-444d-947e-313ff39785a5-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-6w2sv\" (UID: \"59ae5769-a6cf-444d-947e-313ff39785a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6w2sv" Oct 30 16:35:42.299926 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:42.299866 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59ae5769-a6cf-444d-947e-313ff39785a5-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-6w2sv\" (UID: \"59ae5769-a6cf-444d-947e-313ff39785a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6w2sv" Oct 30 16:35:42.300158 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:42.299934 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59ae5769-a6cf-444d-947e-313ff39785a5-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-6w2sv\" (UID: \"59ae5769-a6cf-444d-947e-313ff39785a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6w2sv" Oct 30 16:35:42.300158 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:42.299959 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59ae5769-a6cf-444d-947e-313ff39785a5-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-6w2sv\" (UID: \"59ae5769-a6cf-444d-947e-313ff39785a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6w2sv" Oct 30 16:35:42.300158 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:42.300065 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59ae5769-a6cf-444d-947e-313ff39785a5-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-6w2sv\" (UID: \"59ae5769-a6cf-444d-947e-313ff39785a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6w2sv" Oct 30 16:35:42.300158 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:42.300089 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59ae5769-a6cf-444d-947e-313ff39785a5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-6w2sv\" (UID: \"59ae5769-a6cf-444d-947e-313ff39785a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6w2sv" Oct 30 16:35:42.300359 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:42.300234 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59ae5769-a6cf-444d-947e-313ff39785a5-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-6w2sv\" (UID: \"59ae5769-a6cf-444d-947e-313ff39785a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6w2sv" Oct 30 16:35:42.300879 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:42.300851 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59ae5769-a6cf-444d-947e-313ff39785a5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-6w2sv\" (UID: \"59ae5769-a6cf-444d-947e-313ff39785a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6w2sv" Oct 30 16:35:42.301002 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:42.300910 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59ae5769-a6cf-444d-947e-313ff39785a5-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-6w2sv\" (UID: \"59ae5769-a6cf-444d-947e-313ff39785a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6w2sv" Oct 30 16:35:42.301190 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:42.301169 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59ae5769-a6cf-444d-947e-313ff39785a5-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-6w2sv\" (UID: \"59ae5769-a6cf-444d-947e-313ff39785a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6w2sv" Oct 30 16:35:42.301548 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:42.301523 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59ae5769-a6cf-444d-947e-313ff39785a5-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-6w2sv\" (UID: \"59ae5769-a6cf-444d-947e-313ff39785a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6w2sv" Oct 30 16:35:42.303749 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:42.303725 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59ae5769-a6cf-444d-947e-313ff39785a5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-6w2sv\" (UID: \"59ae5769-a6cf-444d-947e-313ff39785a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6w2sv" Oct 30 16:35:42.303884 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:42.303745 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59ae5769-a6cf-444d-947e-313ff39785a5-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-6w2sv\" (UID: \"59ae5769-a6cf-444d-947e-313ff39785a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6w2sv" Oct 30 16:35:42.324260 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:42.324228 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59ae5769-a6cf-444d-947e-313ff39785a5-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-6w2sv\" (UID: \"59ae5769-a6cf-444d-947e-313ff39785a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6w2sv" Oct 30 16:35:42.324472 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:42.324452 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs98z\" (UniqueName: \"kubernetes.io/projected/59ae5769-a6cf-444d-947e-313ff39785a5-kube-api-access-zs98z\") pod \"maas-default-gateway-openshift-default-7854949c9-6w2sv\" (UID: \"59ae5769-a6cf-444d-947e-313ff39785a5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6w2sv" Oct 30 16:35:42.394745 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:42.394710 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6w2sv" Oct 30 16:35:42.731286 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:42.731260 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6w2sv"] Oct 30 16:35:42.734548 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:42.734511 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:35:42.734649 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:42.734578 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:35:42.734649 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:42.734604 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:35:42.853021 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:42.848961 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a8fd3e5-rmwfd" Oct 30 16:35:42.928810 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:42.920808 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0768bd59-6767-4d80-9982-abed675e7c27-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0768bd59-6767-4d80-9982-abed675e7c27-maas-api-openshift-ca\") pod \"0768bd59-6767-4d80-9982-abed675e7c27\" (UID: \"0768bd59-6767-4d80-9982-abed675e7c27\") " Oct 30 16:35:42.928810 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:42.920858 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0768bd59-6767-4d80-9982-abed675e7c27-kube-api-access-qmpg8\" (UniqueName: \"kubernetes.io/projected/0768bd59-6767-4d80-9982-abed675e7c27-kube-api-access-qmpg8\") pod \"0768bd59-6767-4d80-9982-abed675e7c27\" (UID: \"0768bd59-6767-4d80-9982-abed675e7c27\") " Oct 30 16:35:42.928810 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:42.920969 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0768bd59-6767-4d80-9982-abed675e7c27-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0768bd59-6767-4d80-9982-abed675e7c27-maas-api-xks-ca\") pod \"0768bd59-6767-4d80-9982-abed675e7c27\" (UID: \"0768bd59-6767-4d80-9982-abed675e7c27\") " Oct 30 16:35:42.928810 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:42.922122 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0768bd59-6767-4d80-9982-abed675e7c27-maas-api-openshift-ca" pod "0768bd59-6767-4d80-9982-abed675e7c27" (UID: "0768bd59-6767-4d80-9982-abed675e7c27"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:35:42.934052 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:42.933170 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0768bd59-6767-4d80-9982-abed675e7c27-kube-api-access-qmpg8" pod "0768bd59-6767-4d80-9982-abed675e7c27" (UID: "0768bd59-6767-4d80-9982-abed675e7c27"). InnerVolumeSpecName "kube-api-access-qmpg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:35:42.940486 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:42.937678 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0768bd59-6767-4d80-9982-abed675e7c27-maas-api-xks-ca" pod "0768bd59-6767-4d80-9982-abed675e7c27" (UID: "0768bd59-6767-4d80-9982-abed675e7c27"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:35:43.028858 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:43.022646 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0768bd59-6767-4d80-9982-abed675e7c27-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:35:43.028858 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:43.022687 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0768bd59-6767-4d80-9982-abed675e7c27-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:35:43.028858 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:43.022702 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmpg8\" (UniqueName: \"kubernetes.io/projected/0768bd59-6767-4d80-9982-abed675e7c27-kube-api-access-qmpg8\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:35:43.668810 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:43.668677 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a8fd3e5-rmwfd" event={"ID":"0768bd59-6767-4d80-9982-abed675e7c27","Type":"ContainerDied","Data":"8ebd9b2f68479312ef1be1c5fdb64d2aacc5c39341bccc1d4a736b8e1ccc6b76"} Oct 30 16:35:43.668810 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:43.668743 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ebd9b2f68479312ef1be1c5fdb64d2aacc5c39341bccc1d4a736b8e1ccc6b76" Oct 30 16:35:43.669065 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:43.668829 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a8fd3e5-rmwfd" Oct 30 16:35:43.678563 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:43.676887 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6w2sv" event={"ID":"59ae5769-a6cf-444d-947e-313ff39785a5","Type":"ContainerStarted","Data":"03d8bac861db98541fa5cdd6bcd270caad21afee80b1a02209b5fc450831a391"} Oct 30 16:35:43.678563 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:43.676970 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6w2sv" event={"ID":"59ae5769-a6cf-444d-947e-313ff39785a5","Type":"ContainerStarted","Data":"0a7e7a1c47b9e844d6ad020ce2a8d7735cff95aa2283cafe73407182be41223d"} Oct 30 16:35:43.740002 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:43.739930 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6w2sv" podStartSLOduration=1.739910402 podStartE2EDuration="1.739910402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:35:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:35:43.739359234 +0000 UTC m=+770.966534335" watchObservedRunningTime="2026-10-30 16:35:43.739910402 +0000 UTC m=+770.967085501" Oct 30 16:35:44.395249 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:44.395202 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6w2sv" Oct 30 16:35:44.397239 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:44.397210 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-6w2sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 16:35:44.397381 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:44.397267 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6w2sv" podUID="59ae5769-a6cf-444d-947e-313ff39785a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 16:35:45.396192 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:45.396147 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-6w2sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 16:35:45.396673 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:45.396208 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6w2sv" podUID="59ae5769-a6cf-444d-947e-313ff39785a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 16:35:46.395916 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:46.395884 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-6w2sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 16:35:46.396085 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:46.395934 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6w2sv" podUID="59ae5769-a6cf-444d-947e-313ff39785a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 16:35:47.396010 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:47.395973 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-6w2sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 16:35:47.396476 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:47.396033 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6w2sv" podUID="59ae5769-a6cf-444d-947e-313ff39785a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 16:35:48.395668 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:48.395628 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-6w2sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 16:35:48.395890 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:48.395685 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6w2sv" podUID="59ae5769-a6cf-444d-947e-313ff39785a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 16:35:49.395806 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:49.395763 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-6w2sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 16:35:49.396191 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:49.395836 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6w2sv" podUID="59ae5769-a6cf-444d-947e-313ff39785a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 16:35:50.395468 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:50.395433 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-6w2sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 16:35:50.395687 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:50.395495 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6w2sv" podUID="59ae5769-a6cf-444d-947e-313ff39785a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 16:35:51.904505 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:51.904469 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6w2sv" Oct 30 16:35:51.905073 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:51.905051 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6w2sv" Oct 30 16:35:51.905859 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:51.905834 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6w2sv" Oct 30 16:35:58.227996 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:58.227956 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8"] Oct 30 16:35:58.228473 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:58.228459 4493 cpu_manager.go:414] "containerMap: removing container" podUID="0768bd59-6767-4d80-9982-abed675e7c27" containerName="revoke-keys" Oct 30 16:35:58.228526 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:58.228475 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0768bd59-6767-4d80-9982-abed675e7c27" containerName="revoke-keys" Oct 30 16:35:58.228583 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:58.228574 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="0768bd59-6767-4d80-9982-abed675e7c27" containerName="revoke-keys" Oct 30 16:35:58.233748 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:58.233730 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8" Oct 30 16:35:58.268116 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:58.268083 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8"] Oct 30 16:35:58.301488 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:58.301451 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2c02109-c263-42aa-bdad-a995f3577d99-istio-data\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8\" (UID: \"e2c02109-c263-42aa-bdad-a995f3577d99\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8" Oct 30 16:35:58.301659 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:58.301571 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2c02109-c263-42aa-bdad-a995f3577d99-workload-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8\" (UID: \"e2c02109-c263-42aa-bdad-a995f3577d99\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8" Oct 30 16:35:58.301659 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:58.301607 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2c02109-c263-42aa-bdad-a995f3577d99-istiod-ca-cert\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8\" (UID: \"e2c02109-c263-42aa-bdad-a995f3577d99\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8" Oct 30 16:35:58.301807 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:58.301721 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2c02109-c263-42aa-bdad-a995f3577d99-istio-podinfo\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8\" (UID: \"e2c02109-c263-42aa-bdad-a995f3577d99\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8" Oct 30 16:35:58.301807 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:58.301754 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2c02109-c263-42aa-bdad-a995f3577d99-workload-certs\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8\" (UID: \"e2c02109-c263-42aa-bdad-a995f3577d99\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8" Oct 30 16:35:58.301894 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:58.301876 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2c02109-c263-42aa-bdad-a995f3577d99-credential-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8\" (UID: \"e2c02109-c263-42aa-bdad-a995f3577d99\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8" Oct 30 16:35:58.301942 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:58.301901 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2c02109-c263-42aa-bdad-a995f3577d99-istio-token\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8\" (UID: \"e2c02109-c263-42aa-bdad-a995f3577d99\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8" Oct 30 16:35:58.301942 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:58.301923 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2c02109-c263-42aa-bdad-a995f3577d99-istio-envoy\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8\" (UID: \"e2c02109-c263-42aa-bdad-a995f3577d99\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8" Oct 30 16:35:58.302061 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:58.302009 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmnfj\" (UniqueName: \"kubernetes.io/projected/e2c02109-c263-42aa-bdad-a995f3577d99-kube-api-access-dmnfj\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8\" (UID: \"e2c02109-c263-42aa-bdad-a995f3577d99\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8" Oct 30 16:35:58.403272 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:58.403233 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmnfj\" (UniqueName: \"kubernetes.io/projected/e2c02109-c263-42aa-bdad-a995f3577d99-kube-api-access-dmnfj\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8\" (UID: \"e2c02109-c263-42aa-bdad-a995f3577d99\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8" Oct 30 16:35:58.403452 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:58.403315 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2c02109-c263-42aa-bdad-a995f3577d99-istio-data\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8\" (UID: \"e2c02109-c263-42aa-bdad-a995f3577d99\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8" Oct 30 16:35:58.403452 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:58.403391 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2c02109-c263-42aa-bdad-a995f3577d99-workload-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8\" (UID: \"e2c02109-c263-42aa-bdad-a995f3577d99\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8" Oct 30 16:35:58.403452 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:58.403428 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2c02109-c263-42aa-bdad-a995f3577d99-istiod-ca-cert\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8\" (UID: \"e2c02109-c263-42aa-bdad-a995f3577d99\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8" Oct 30 16:35:58.403618 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:58.403482 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2c02109-c263-42aa-bdad-a995f3577d99-istio-podinfo\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8\" (UID: \"e2c02109-c263-42aa-bdad-a995f3577d99\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8" Oct 30 16:35:58.403618 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:58.403507 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2c02109-c263-42aa-bdad-a995f3577d99-workload-certs\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8\" (UID: \"e2c02109-c263-42aa-bdad-a995f3577d99\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8" Oct 30 16:35:58.403618 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:58.403566 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2c02109-c263-42aa-bdad-a995f3577d99-credential-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8\" (UID: \"e2c02109-c263-42aa-bdad-a995f3577d99\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8" Oct 30 16:35:58.403618 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:58.403592 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2c02109-c263-42aa-bdad-a995f3577d99-istio-token\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8\" (UID: \"e2c02109-c263-42aa-bdad-a995f3577d99\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8" Oct 30 16:35:58.403857 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:58.403625 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2c02109-c263-42aa-bdad-a995f3577d99-istio-envoy\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8\" (UID: \"e2c02109-c263-42aa-bdad-a995f3577d99\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8" Oct 30 16:35:58.403857 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:58.403842 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2c02109-c263-42aa-bdad-a995f3577d99-istio-data\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8\" (UID: \"e2c02109-c263-42aa-bdad-a995f3577d99\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8" Oct 30 16:35:58.403974 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:58.403856 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2c02109-c263-42aa-bdad-a995f3577d99-workload-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8\" (UID: \"e2c02109-c263-42aa-bdad-a995f3577d99\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8" Oct 30 16:35:58.404540 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:58.404098 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2c02109-c263-42aa-bdad-a995f3577d99-credential-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8\" (UID: \"e2c02109-c263-42aa-bdad-a995f3577d99\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8" Oct 30 16:35:58.404540 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:58.404240 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2c02109-c263-42aa-bdad-a995f3577d99-istiod-ca-cert\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8\" (UID: \"e2c02109-c263-42aa-bdad-a995f3577d99\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8" Oct 30 16:35:58.404540 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:58.404495 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2c02109-c263-42aa-bdad-a995f3577d99-workload-certs\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8\" (UID: \"e2c02109-c263-42aa-bdad-a995f3577d99\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8" Oct 30 16:35:58.406251 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:58.406226 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2c02109-c263-42aa-bdad-a995f3577d99-istio-envoy\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8\" (UID: \"e2c02109-c263-42aa-bdad-a995f3577d99\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8" Oct 30 16:35:58.406386 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:58.406232 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2c02109-c263-42aa-bdad-a995f3577d99-istio-podinfo\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8\" (UID: \"e2c02109-c263-42aa-bdad-a995f3577d99\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8" Oct 30 16:35:58.426562 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:58.426531 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2c02109-c263-42aa-bdad-a995f3577d99-istio-token\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8\" (UID: \"e2c02109-c263-42aa-bdad-a995f3577d99\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8" Oct 30 16:35:58.426715 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:58.426671 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmnfj\" (UniqueName: \"kubernetes.io/projected/e2c02109-c263-42aa-bdad-a995f3577d99-kube-api-access-dmnfj\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8\" (UID: \"e2c02109-c263-42aa-bdad-a995f3577d99\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8" Oct 30 16:35:58.549176 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:58.549074 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8" Oct 30 16:35:58.893434 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:58.893388 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8"] Oct 30 16:35:58.894682 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:35:58.894658 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2c02109_c263_42aa_bdad_a995f3577d99.slice/crio-4746ca4171044ae934f02bd8c484ea4e5f8f8869820e04ef9e8f75e556d9be64 WatchSource:0}: Error finding container 4746ca4171044ae934f02bd8c484ea4e5f8f8869820e04ef9e8f75e556d9be64: Status 404 returned error can't find the container with id 4746ca4171044ae934f02bd8c484ea4e5f8f8869820e04ef9e8f75e556d9be64 Oct 30 16:35:58.897097 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:58.897066 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:35:58.897159 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:58.897131 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:35:58.897192 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:58.897162 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:35:58.936223 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:58.936191 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8" event={"ID":"e2c02109-c263-42aa-bdad-a995f3577d99","Type":"ContainerStarted","Data":"4746ca4171044ae934f02bd8c484ea4e5f8f8869820e04ef9e8f75e556d9be64"} Oct 30 16:35:59.947935 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:35:59.947898 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8" event={"ID":"e2c02109-c263-42aa-bdad-a995f3577d99","Type":"ContainerStarted","Data":"e2dbe35a076a50a71f243b174960bfd91b291ca96001f7ddf20c71fd56fa4ccf"} Oct 30 16:36:00.016853 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:00.007247 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8" podStartSLOduration=2.007227515 podStartE2EDuration="2.007227515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:35:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:36:00.005916742 +0000 UTC m=+787.233091861" watchObservedRunningTime="2026-10-30 16:36:00.007227515 +0000 UTC m=+787.234402620" Oct 30 16:36:00.380151 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:36:00.380072 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 16:36:00.552058 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:00.551948 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8" Oct 30 16:36:00.553395 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:00.553363 4493 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 16:36:00.554069 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:00.554011 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8" podUID="e2c02109-c263-42aa-bdad-a995f3577d99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 16:36:01.427812 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:01.425343 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 16:36:01.550494 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:01.550454 4493 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 16:36:01.550683 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:01.550529 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8" podUID="e2c02109-c263-42aa-bdad-a995f3577d99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 16:36:02.554810 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:02.554256 4493 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 16:36:02.554810 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:02.554336 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8" podUID="e2c02109-c263-42aa-bdad-a995f3577d99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 16:36:03.558833 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:03.554502 4493 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 16:36:03.558833 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:03.554561 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8" podUID="e2c02109-c263-42aa-bdad-a995f3577d99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 16:36:04.819807 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:04.816889 4493 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 16:36:04.819807 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:04.816974 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8" podUID="e2c02109-c263-42aa-bdad-a995f3577d99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 16:36:05.550817 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:05.550242 4493 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 16:36:05.550817 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:05.550327 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8" podUID="e2c02109-c263-42aa-bdad-a995f3577d99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 16:36:06.555812 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:06.549486 4493 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 16:36:06.555812 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:06.549546 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8" podUID="e2c02109-c263-42aa-bdad-a995f3577d99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 16:36:07.554155 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:07.554121 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8" Oct 30 16:36:07.554538 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:07.554508 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8" Oct 30 16:36:07.555315 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:07.555293 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8" Oct 30 16:36:17.821810 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:17.820879 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-6n8m6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.22:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:36:17.821810 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:17.820975 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6n8m6" podUID="5903f4b2-9604-4eab-adf5-e0f7452ffbba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.22:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:36:19.489195 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:19.489161 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a49e5776-qv4xq"] Oct 30 16:36:19.493648 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:19.493626 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49e5776-qv4xq" Oct 30 16:36:19.512121 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:19.512097 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72b50e48-9b41-4994-b2e2-4e328586edb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a49e5776-qv4xq\" (UID: \"72b50e48-9b41-4994-b2e2-4e328586edb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49e5776-qv4xq" Oct 30 16:36:19.512251 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:19.512157 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mr4vm\" (UniqueName: \"kubernetes.io/projected/72b50e48-9b41-4994-b2e2-4e328586edb2-kube-api-access-mr4vm\") pod \"maas-api-revoke-sub-a49e5776-qv4xq\" (UID: \"72b50e48-9b41-4994-b2e2-4e328586edb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49e5776-qv4xq" Oct 30 16:36:19.512422 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:19.512401 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72b50e48-9b41-4994-b2e2-4e328586edb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a49e5776-qv4xq\" (UID: \"72b50e48-9b41-4994-b2e2-4e328586edb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49e5776-qv4xq" Oct 30 16:36:19.524810 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:19.519961 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:36:19.524810 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:19.520056 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:36:19.524810 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:19.520060 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 16:36:19.524810 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:19.520228 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:36:19.525400 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:19.525377 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a49e5776-qv4xq"] Oct 30 16:36:19.613264 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:19.613236 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72b50e48-9b41-4994-b2e2-4e328586edb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a49e5776-qv4xq\" (UID: \"72b50e48-9b41-4994-b2e2-4e328586edb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49e5776-qv4xq" Oct 30 16:36:19.613264 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:19.613265 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mr4vm\" (UniqueName: \"kubernetes.io/projected/72b50e48-9b41-4994-b2e2-4e328586edb2-kube-api-access-mr4vm\") pod \"maas-api-revoke-sub-a49e5776-qv4xq\" (UID: \"72b50e48-9b41-4994-b2e2-4e328586edb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49e5776-qv4xq" Oct 30 16:36:19.613534 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:19.613362 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72b50e48-9b41-4994-b2e2-4e328586edb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a49e5776-qv4xq\" (UID: \"72b50e48-9b41-4994-b2e2-4e328586edb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49e5776-qv4xq" Oct 30 16:36:19.613928 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:19.613910 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72b50e48-9b41-4994-b2e2-4e328586edb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a49e5776-qv4xq\" (UID: \"72b50e48-9b41-4994-b2e2-4e328586edb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49e5776-qv4xq" Oct 30 16:36:19.615815 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:19.615767 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72b50e48-9b41-4994-b2e2-4e328586edb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a49e5776-qv4xq\" (UID: \"72b50e48-9b41-4994-b2e2-4e328586edb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49e5776-qv4xq" Oct 30 16:36:19.636939 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:19.636915 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mr4vm\" (UniqueName: \"kubernetes.io/projected/72b50e48-9b41-4994-b2e2-4e328586edb2-kube-api-access-mr4vm\") pod \"maas-api-revoke-sub-a49e5776-qv4xq\" (UID: \"72b50e48-9b41-4994-b2e2-4e328586edb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49e5776-qv4xq" Oct 30 16:36:19.810776 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:19.810691 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49e5776-qv4xq" Oct 30 16:36:20.119021 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:20.118987 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a49e5776-qv4xq"] Oct 30 16:36:20.119531 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:36:20.119507 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72b50e48_9b41_4994_b2e2_4e328586edb2.slice/crio-9bc742f7ff39ab0412834c3d69374c5bf999e166521f8091ac8bbe3958eddaa6 WatchSource:0}: Error finding container 9bc742f7ff39ab0412834c3d69374c5bf999e166521f8091ac8bbe3958eddaa6: Status 404 returned error can't find the container with id 9bc742f7ff39ab0412834c3d69374c5bf999e166521f8091ac8bbe3958eddaa6 Oct 30 16:36:20.299294 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:20.299259 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49e5776-qv4xq" event={"ID":"72b50e48-9b41-4994-b2e2-4e328586edb2","Type":"ContainerStarted","Data":"9bc742f7ff39ab0412834c3d69374c5bf999e166521f8091ac8bbe3958eddaa6"} Oct 30 16:36:21.308430 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:21.308333 4493 generic.go:356] "Generic (PLEG): container finished" podID="72b50e48-9b41-4994-b2e2-4e328586edb2" containerID="8ccf2e70590438e94ae105d822644cc483dfe19aefd3c7793d31400caae51c09" exitCode=0 Oct 30 16:36:21.308430 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:21.308389 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49e5776-qv4xq" event={"ID":"72b50e48-9b41-4994-b2e2-4e328586edb2","Type":"ContainerDied","Data":"8ccf2e70590438e94ae105d822644cc483dfe19aefd3c7793d31400caae51c09"} Oct 30 16:36:22.469658 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:22.469631 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49e5776-qv4xq" Oct 30 16:36:22.547564 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:22.547532 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72b50e48-9b41-4994-b2e2-4e328586edb2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72b50e48-9b41-4994-b2e2-4e328586edb2-maas-api-openshift-ca\") pod \"72b50e48-9b41-4994-b2e2-4e328586edb2\" (UID: \"72b50e48-9b41-4994-b2e2-4e328586edb2\") " Oct 30 16:36:22.547738 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:22.547658 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72b50e48-9b41-4994-b2e2-4e328586edb2-kube-api-access-mr4vm\" (UniqueName: \"kubernetes.io/projected/72b50e48-9b41-4994-b2e2-4e328586edb2-kube-api-access-mr4vm\") pod \"72b50e48-9b41-4994-b2e2-4e328586edb2\" (UID: \"72b50e48-9b41-4994-b2e2-4e328586edb2\") " Oct 30 16:36:22.547738 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:22.547715 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72b50e48-9b41-4994-b2e2-4e328586edb2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72b50e48-9b41-4994-b2e2-4e328586edb2-maas-api-xks-ca\") pod \"72b50e48-9b41-4994-b2e2-4e328586edb2\" (UID: \"72b50e48-9b41-4994-b2e2-4e328586edb2\") " Oct 30 16:36:22.547982 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:22.547955 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72b50e48-9b41-4994-b2e2-4e328586edb2-maas-api-openshift-ca" pod "72b50e48-9b41-4994-b2e2-4e328586edb2" (UID: "72b50e48-9b41-4994-b2e2-4e328586edb2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:36:22.548166 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:22.548149 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72b50e48-9b41-4994-b2e2-4e328586edb2-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:36:22.550732 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:22.550710 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72b50e48-9b41-4994-b2e2-4e328586edb2-maas-api-xks-ca" pod "72b50e48-9b41-4994-b2e2-4e328586edb2" (UID: "72b50e48-9b41-4994-b2e2-4e328586edb2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:36:22.550903 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:22.550806 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72b50e48-9b41-4994-b2e2-4e328586edb2-kube-api-access-mr4vm" pod "72b50e48-9b41-4994-b2e2-4e328586edb2" (UID: "72b50e48-9b41-4994-b2e2-4e328586edb2"). InnerVolumeSpecName "kube-api-access-mr4vm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:36:22.649324 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:22.649285 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72b50e48-9b41-4994-b2e2-4e328586edb2-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:36:22.649324 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:22.649315 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mr4vm\" (UniqueName: \"kubernetes.io/projected/72b50e48-9b41-4994-b2e2-4e328586edb2-kube-api-access-mr4vm\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:36:23.010430 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:23.010395 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 16:36:23.327122 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:23.327029 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49e5776-qv4xq" event={"ID":"72b50e48-9b41-4994-b2e2-4e328586edb2","Type":"ContainerDied","Data":"9bc742f7ff39ab0412834c3d69374c5bf999e166521f8091ac8bbe3958eddaa6"} Oct 30 16:36:23.327122 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:23.327094 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9bc742f7ff39ab0412834c3d69374c5bf999e166521f8091ac8bbe3958eddaa6" Oct 30 16:36:23.327122 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:23.327053 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49e5776-qv4xq" Oct 30 16:36:27.808272 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:27.808242 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t"] Oct 30 16:36:27.808747 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:27.808734 4493 cpu_manager.go:414] "containerMap: removing container" podUID="72b50e48-9b41-4994-b2e2-4e328586edb2" containerName="revoke-keys" Oct 30 16:36:27.808833 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:27.808748 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72b50e48-9b41-4994-b2e2-4e328586edb2" containerName="revoke-keys" Oct 30 16:36:27.808882 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:27.808846 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="72b50e48-9b41-4994-b2e2-4e328586edb2" containerName="revoke-keys" Oct 30 16:36:27.812728 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:27.812708 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t" Oct 30 16:36:27.827088 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:27.827069 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-2d365c-openshift-default-dockercfg-gctmp\"" Oct 30 16:36:27.843482 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:27.843450 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t"] Oct 30 16:36:27.914220 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:27.914170 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad417aab-5686-4ae2-9163-91af4a79b356-istio-data\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t\" (UID: \"ad417aab-5686-4ae2-9163-91af4a79b356\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t" Oct 30 16:36:27.914387 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:27.914236 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad417aab-5686-4ae2-9163-91af4a79b356-workload-certs\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t\" (UID: \"ad417aab-5686-4ae2-9163-91af4a79b356\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t" Oct 30 16:36:27.914387 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:27.914311 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad417aab-5686-4ae2-9163-91af4a79b356-istio-podinfo\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t\" (UID: \"ad417aab-5686-4ae2-9163-91af4a79b356\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t" Oct 30 16:36:27.914387 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:27.914361 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad417aab-5686-4ae2-9163-91af4a79b356-istio-token\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t\" (UID: \"ad417aab-5686-4ae2-9163-91af4a79b356\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t" Oct 30 16:36:27.914507 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:27.914414 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad417aab-5686-4ae2-9163-91af4a79b356-istio-envoy\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t\" (UID: \"ad417aab-5686-4ae2-9163-91af4a79b356\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t" Oct 30 16:36:27.914507 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:27.914461 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad417aab-5686-4ae2-9163-91af4a79b356-credential-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t\" (UID: \"ad417aab-5686-4ae2-9163-91af4a79b356\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t" Oct 30 16:36:27.914603 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:27.914579 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad417aab-5686-4ae2-9163-91af4a79b356-istiod-ca-cert\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t\" (UID: \"ad417aab-5686-4ae2-9163-91af4a79b356\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t" Oct 30 16:36:27.914661 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:27.914644 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad417aab-5686-4ae2-9163-91af4a79b356-workload-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t\" (UID: \"ad417aab-5686-4ae2-9163-91af4a79b356\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t" Oct 30 16:36:27.914711 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:27.914697 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkh2h\" (UniqueName: \"kubernetes.io/projected/ad417aab-5686-4ae2-9163-91af4a79b356-kube-api-access-lkh2h\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t\" (UID: \"ad417aab-5686-4ae2-9163-91af4a79b356\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t" Oct 30 16:36:28.016016 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:28.015967 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad417aab-5686-4ae2-9163-91af4a79b356-istio-podinfo\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t\" (UID: \"ad417aab-5686-4ae2-9163-91af4a79b356\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t" Oct 30 16:36:28.016230 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:28.016043 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad417aab-5686-4ae2-9163-91af4a79b356-istio-token\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t\" (UID: \"ad417aab-5686-4ae2-9163-91af4a79b356\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t" Oct 30 16:36:28.016230 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:28.016082 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad417aab-5686-4ae2-9163-91af4a79b356-istio-envoy\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t\" (UID: \"ad417aab-5686-4ae2-9163-91af4a79b356\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t" Oct 30 16:36:28.016230 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:28.016107 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad417aab-5686-4ae2-9163-91af4a79b356-credential-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t\" (UID: \"ad417aab-5686-4ae2-9163-91af4a79b356\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t" Oct 30 16:36:28.016230 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:28.016166 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad417aab-5686-4ae2-9163-91af4a79b356-istiod-ca-cert\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t\" (UID: \"ad417aab-5686-4ae2-9163-91af4a79b356\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t" Oct 30 16:36:28.016230 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:28.016211 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad417aab-5686-4ae2-9163-91af4a79b356-workload-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t\" (UID: \"ad417aab-5686-4ae2-9163-91af4a79b356\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t" Oct 30 16:36:28.016491 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:28.016255 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkh2h\" (UniqueName: \"kubernetes.io/projected/ad417aab-5686-4ae2-9163-91af4a79b356-kube-api-access-lkh2h\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t\" (UID: \"ad417aab-5686-4ae2-9163-91af4a79b356\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t" Oct 30 16:36:28.016491 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:28.016312 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad417aab-5686-4ae2-9163-91af4a79b356-istio-data\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t\" (UID: \"ad417aab-5686-4ae2-9163-91af4a79b356\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t" Oct 30 16:36:28.016491 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:28.016356 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad417aab-5686-4ae2-9163-91af4a79b356-workload-certs\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t\" (UID: \"ad417aab-5686-4ae2-9163-91af4a79b356\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t" Oct 30 16:36:28.016822 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:28.016733 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad417aab-5686-4ae2-9163-91af4a79b356-workload-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t\" (UID: \"ad417aab-5686-4ae2-9163-91af4a79b356\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t" Oct 30 16:36:28.016946 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:28.016889 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad417aab-5686-4ae2-9163-91af4a79b356-credential-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t\" (UID: \"ad417aab-5686-4ae2-9163-91af4a79b356\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t" Oct 30 16:36:28.017030 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:28.017009 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad417aab-5686-4ae2-9163-91af4a79b356-workload-certs\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t\" (UID: \"ad417aab-5686-4ae2-9163-91af4a79b356\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t" Oct 30 16:36:28.017173 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:28.017154 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad417aab-5686-4ae2-9163-91af4a79b356-istiod-ca-cert\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t\" (UID: \"ad417aab-5686-4ae2-9163-91af4a79b356\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t" Oct 30 16:36:28.017573 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:28.017548 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad417aab-5686-4ae2-9163-91af4a79b356-istio-data\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t\" (UID: \"ad417aab-5686-4ae2-9163-91af4a79b356\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t" Oct 30 16:36:28.018756 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:28.018732 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad417aab-5686-4ae2-9163-91af4a79b356-istio-envoy\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t\" (UID: \"ad417aab-5686-4ae2-9163-91af4a79b356\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t" Oct 30 16:36:28.018901 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:28.018883 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad417aab-5686-4ae2-9163-91af4a79b356-istio-podinfo\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t\" (UID: \"ad417aab-5686-4ae2-9163-91af4a79b356\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t" Oct 30 16:36:28.042530 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:28.042504 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad417aab-5686-4ae2-9163-91af4a79b356-istio-token\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t\" (UID: \"ad417aab-5686-4ae2-9163-91af4a79b356\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t" Oct 30 16:36:28.043171 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:28.043149 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkh2h\" (UniqueName: \"kubernetes.io/projected/ad417aab-5686-4ae2-9163-91af4a79b356-kube-api-access-lkh2h\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t\" (UID: \"ad417aab-5686-4ae2-9163-91af4a79b356\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t" Oct 30 16:36:28.127511 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:28.127431 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t" Oct 30 16:36:28.457889 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:28.457826 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t"] Oct 30 16:36:28.464547 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:36:28.464517 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad417aab_5686_4ae2_9163_91af4a79b356.slice/crio-a8620de0a9e9313eb120bb19036cfdb35cad98bd9cd63f2764eb33e27aa559d1 WatchSource:0}: Error finding container a8620de0a9e9313eb120bb19036cfdb35cad98bd9cd63f2764eb33e27aa559d1: Status 404 returned error can't find the container with id a8620de0a9e9313eb120bb19036cfdb35cad98bd9cd63f2764eb33e27aa559d1 Oct 30 16:36:28.466908 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:28.466873 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:36:28.467017 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:28.466937 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:36:28.467017 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:28.466965 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:36:29.371430 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:29.371394 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t" event={"ID":"ad417aab-5686-4ae2-9163-91af4a79b356","Type":"ContainerStarted","Data":"d266cdc2c7712fcb40ea4d7125601d5ddf3af201f1a5c5b8fd9217fd245504ef"} Oct 30 16:36:29.371850 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:29.371445 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t" event={"ID":"ad417aab-5686-4ae2-9163-91af4a79b356","Type":"ContainerStarted","Data":"a8620de0a9e9313eb120bb19036cfdb35cad98bd9cd63f2764eb33e27aa559d1"} Oct 30 16:36:29.432363 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:29.432315 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t" podStartSLOduration=2.432300656 podStartE2EDuration="2.432300656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:36:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:36:29.418252672 +0000 UTC m=+816.645427773" watchObservedRunningTime="2026-10-30 16:36:29.432300656 +0000 UTC m=+816.659475755" Oct 30 16:36:30.134811 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:30.133119 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t" Oct 30 16:36:30.137302 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:30.137269 4493 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 16:36:30.137454 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:30.137341 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t" podUID="ad417aab-5686-4ae2-9163-91af4a79b356" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 16:36:30.369247 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:36:30.369212 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 16:36:31.134811 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:31.134283 4493 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 16:36:31.158290 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:31.155025 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t" podUID="ad417aab-5686-4ae2-9163-91af4a79b356" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 16:36:32.131193 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:32.131159 4493 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 16:36:32.131449 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:32.131422 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t" podUID="ad417aab-5686-4ae2-9163-91af4a79b356" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 16:36:33.134810 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:33.129134 4493 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 16:36:33.134810 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:33.129192 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t" podUID="ad417aab-5686-4ae2-9163-91af4a79b356" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 16:36:34.133818 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:34.133199 4493 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 16:36:34.133818 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:34.133266 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t" podUID="ad417aab-5686-4ae2-9163-91af4a79b356" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 16:36:35.129814 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:35.128204 4493 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 16:36:35.129814 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:35.128266 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t" podUID="ad417aab-5686-4ae2-9163-91af4a79b356" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 16:36:36.132709 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:36.132675 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t" Oct 30 16:36:36.133324 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:36.133095 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t" Oct 30 16:36:36.133876 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:36.133853 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t" Oct 30 16:36:43.163328 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:43.163281 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw"] Oct 30 16:36:43.168569 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:43.168540 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" Oct 30 16:36:43.196414 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:43.196387 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-2d365c\"/\"distinct-2-w3-2d365c-kserve-self-signed-certs\"" Oct 30 16:36:43.201921 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:43.201876 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw"] Oct 30 16:36:43.265040 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:43.264973 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5abe703-fee0-4185-aa22-88d1db06392b-dshm\") pod \"distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw\" (UID: \"b5abe703-fee0-4185-aa22-88d1db06392b\") " pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" Oct 30 16:36:43.265207 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:43.265118 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqcbx\" (UniqueName: \"kubernetes.io/projected/b5abe703-fee0-4185-aa22-88d1db06392b-kube-api-access-hqcbx\") pod \"distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw\" (UID: \"b5abe703-fee0-4185-aa22-88d1db06392b\") " pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" Oct 30 16:36:43.265267 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:43.265228 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5abe703-fee0-4185-aa22-88d1db06392b-home\") pod \"distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw\" (UID: \"b5abe703-fee0-4185-aa22-88d1db06392b\") " pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" Oct 30 16:36:43.265371 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:43.265350 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5abe703-fee0-4185-aa22-88d1db06392b-tls-certs\") pod \"distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw\" (UID: \"b5abe703-fee0-4185-aa22-88d1db06392b\") " pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" Oct 30 16:36:43.265428 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:43.265397 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5abe703-fee0-4185-aa22-88d1db06392b-tmp-dir\") pod \"distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw\" (UID: \"b5abe703-fee0-4185-aa22-88d1db06392b\") " pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" Oct 30 16:36:43.265428 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:43.265414 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5abe703-fee0-4185-aa22-88d1db06392b-model-cache\") pod \"distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw\" (UID: \"b5abe703-fee0-4185-aa22-88d1db06392b\") " pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" Oct 30 16:36:43.372475 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:43.372430 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqcbx\" (UniqueName: \"kubernetes.io/projected/b5abe703-fee0-4185-aa22-88d1db06392b-kube-api-access-hqcbx\") pod \"distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw\" (UID: \"b5abe703-fee0-4185-aa22-88d1db06392b\") " pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" Oct 30 16:36:43.372688 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:43.372544 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5abe703-fee0-4185-aa22-88d1db06392b-home\") pod \"distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw\" (UID: \"b5abe703-fee0-4185-aa22-88d1db06392b\") " pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" Oct 30 16:36:43.372688 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:43.372642 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5abe703-fee0-4185-aa22-88d1db06392b-tls-certs\") pod \"distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw\" (UID: \"b5abe703-fee0-4185-aa22-88d1db06392b\") " pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" Oct 30 16:36:43.372850 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:43.372687 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5abe703-fee0-4185-aa22-88d1db06392b-tmp-dir\") pod \"distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw\" (UID: \"b5abe703-fee0-4185-aa22-88d1db06392b\") " pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" Oct 30 16:36:43.372850 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:43.372712 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5abe703-fee0-4185-aa22-88d1db06392b-model-cache\") pod \"distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw\" (UID: \"b5abe703-fee0-4185-aa22-88d1db06392b\") " pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" Oct 30 16:36:43.372850 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:43.372749 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5abe703-fee0-4185-aa22-88d1db06392b-dshm\") pod \"distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw\" (UID: \"b5abe703-fee0-4185-aa22-88d1db06392b\") " pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" Oct 30 16:36:43.374814 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:43.374542 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5abe703-fee0-4185-aa22-88d1db06392b-tmp-dir\") pod \"distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw\" (UID: \"b5abe703-fee0-4185-aa22-88d1db06392b\") " pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" Oct 30 16:36:43.375238 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:43.375212 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5abe703-fee0-4185-aa22-88d1db06392b-model-cache\") pod \"distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw\" (UID: \"b5abe703-fee0-4185-aa22-88d1db06392b\") " pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" Oct 30 16:36:43.375692 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:43.375621 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5abe703-fee0-4185-aa22-88d1db06392b-home\") pod \"distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw\" (UID: \"b5abe703-fee0-4185-aa22-88d1db06392b\") " pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" Oct 30 16:36:43.379960 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:43.379936 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5abe703-fee0-4185-aa22-88d1db06392b-dshm\") pod \"distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw\" (UID: \"b5abe703-fee0-4185-aa22-88d1db06392b\") " pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" Oct 30 16:36:43.380892 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:43.380870 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5abe703-fee0-4185-aa22-88d1db06392b-tls-certs\") pod \"distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw\" (UID: \"b5abe703-fee0-4185-aa22-88d1db06392b\") " pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" Oct 30 16:36:43.433171 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:43.433084 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb"] Oct 30 16:36:43.452004 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:43.451975 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb" Oct 30 16:36:43.464381 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:43.464350 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb"] Oct 30 16:36:43.468842 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:43.465482 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqcbx\" (UniqueName: \"kubernetes.io/projected/b5abe703-fee0-4185-aa22-88d1db06392b-kube-api-access-hqcbx\") pod \"distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw\" (UID: \"b5abe703-fee0-4185-aa22-88d1db06392b\") " pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" Oct 30 16:36:43.475206 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:43.474138 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e2c8233-f7d9-4624-99b3-e022639cbc45-istio-token\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb\" (UID: \"8e2c8233-f7d9-4624-99b3-e022639cbc45\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb" Oct 30 16:36:43.475206 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:43.474179 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2c8233-f7d9-4624-99b3-e022639cbc45-workload-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb\" (UID: \"8e2c8233-f7d9-4624-99b3-e022639cbc45\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb" Oct 30 16:36:43.475206 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:43.474227 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2c8233-f7d9-4624-99b3-e022639cbc45-credential-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb\" (UID: \"8e2c8233-f7d9-4624-99b3-e022639cbc45\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb" Oct 30 16:36:43.475206 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:43.474288 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e2c8233-f7d9-4624-99b3-e022639cbc45-istio-podinfo\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb\" (UID: \"8e2c8233-f7d9-4624-99b3-e022639cbc45\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb" Oct 30 16:36:43.475206 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:43.474354 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msftw\" (UniqueName: \"kubernetes.io/projected/8e2c8233-f7d9-4624-99b3-e022639cbc45-kube-api-access-msftw\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb\" (UID: \"8e2c8233-f7d9-4624-99b3-e022639cbc45\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb" Oct 30 16:36:43.475206 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:43.474426 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e2c8233-f7d9-4624-99b3-e022639cbc45-istio-data\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb\" (UID: \"8e2c8233-f7d9-4624-99b3-e022639cbc45\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb" Oct 30 16:36:43.475206 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:43.474459 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e2c8233-f7d9-4624-99b3-e022639cbc45-workload-certs\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb\" (UID: \"8e2c8233-f7d9-4624-99b3-e022639cbc45\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb" Oct 30 16:36:43.475206 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:43.474483 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e2c8233-f7d9-4624-99b3-e022639cbc45-istio-envoy\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb\" (UID: \"8e2c8233-f7d9-4624-99b3-e022639cbc45\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb" Oct 30 16:36:43.475206 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:43.474529 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e2c8233-f7d9-4624-99b3-e022639cbc45-istiod-ca-cert\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb\" (UID: \"8e2c8233-f7d9-4624-99b3-e022639cbc45\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb" Oct 30 16:36:43.485214 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:43.485191 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" Oct 30 16:36:43.575115 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:43.575085 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e2c8233-f7d9-4624-99b3-e022639cbc45-istiod-ca-cert\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb\" (UID: \"8e2c8233-f7d9-4624-99b3-e022639cbc45\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb" Oct 30 16:36:43.575237 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:43.575152 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e2c8233-f7d9-4624-99b3-e022639cbc45-istio-token\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb\" (UID: \"8e2c8233-f7d9-4624-99b3-e022639cbc45\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb" Oct 30 16:36:43.575237 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:43.575169 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2c8233-f7d9-4624-99b3-e022639cbc45-workload-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb\" (UID: \"8e2c8233-f7d9-4624-99b3-e022639cbc45\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb" Oct 30 16:36:43.575237 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:43.575197 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2c8233-f7d9-4624-99b3-e022639cbc45-credential-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb\" (UID: \"8e2c8233-f7d9-4624-99b3-e022639cbc45\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb" Oct 30 16:36:43.575237 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:43.575234 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e2c8233-f7d9-4624-99b3-e022639cbc45-istio-podinfo\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb\" (UID: \"8e2c8233-f7d9-4624-99b3-e022639cbc45\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb" Oct 30 16:36:43.575403 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:43.575270 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msftw\" (UniqueName: \"kubernetes.io/projected/8e2c8233-f7d9-4624-99b3-e022639cbc45-kube-api-access-msftw\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb\" (UID: \"8e2c8233-f7d9-4624-99b3-e022639cbc45\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb" Oct 30 16:36:43.575403 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:43.575313 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e2c8233-f7d9-4624-99b3-e022639cbc45-istio-data\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb\" (UID: \"8e2c8233-f7d9-4624-99b3-e022639cbc45\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb" Oct 30 16:36:43.575403 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:43.575345 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e2c8233-f7d9-4624-99b3-e022639cbc45-workload-certs\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb\" (UID: \"8e2c8233-f7d9-4624-99b3-e022639cbc45\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb" Oct 30 16:36:43.575403 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:43.575370 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e2c8233-f7d9-4624-99b3-e022639cbc45-istio-envoy\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb\" (UID: \"8e2c8233-f7d9-4624-99b3-e022639cbc45\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb" Oct 30 16:36:43.576418 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:43.575866 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2c8233-f7d9-4624-99b3-e022639cbc45-credential-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb\" (UID: \"8e2c8233-f7d9-4624-99b3-e022639cbc45\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb" Oct 30 16:36:43.576418 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:43.576329 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e2c8233-f7d9-4624-99b3-e022639cbc45-istio-data\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb\" (UID: \"8e2c8233-f7d9-4624-99b3-e022639cbc45\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb" Oct 30 16:36:43.576418 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:43.576383 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e2c8233-f7d9-4624-99b3-e022639cbc45-istiod-ca-cert\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb\" (UID: \"8e2c8233-f7d9-4624-99b3-e022639cbc45\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb" Oct 30 16:36:43.577123 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:43.577088 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2c8233-f7d9-4624-99b3-e022639cbc45-workload-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb\" (UID: \"8e2c8233-f7d9-4624-99b3-e022639cbc45\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb" Oct 30 16:36:43.577208 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:43.577157 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e2c8233-f7d9-4624-99b3-e022639cbc45-workload-certs\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb\" (UID: \"8e2c8233-f7d9-4624-99b3-e022639cbc45\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb" Oct 30 16:36:43.579762 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:43.579729 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e2c8233-f7d9-4624-99b3-e022639cbc45-istio-podinfo\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb\" (UID: \"8e2c8233-f7d9-4624-99b3-e022639cbc45\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb" Oct 30 16:36:43.580146 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:43.580108 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e2c8233-f7d9-4624-99b3-e022639cbc45-istio-envoy\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb\" (UID: \"8e2c8233-f7d9-4624-99b3-e022639cbc45\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb" Oct 30 16:36:43.601872 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:43.601835 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e2c8233-f7d9-4624-99b3-e022639cbc45-istio-token\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb\" (UID: \"8e2c8233-f7d9-4624-99b3-e022639cbc45\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb" Oct 30 16:36:43.602144 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:43.602117 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msftw\" (UniqueName: \"kubernetes.io/projected/8e2c8233-f7d9-4624-99b3-e022639cbc45-kube-api-access-msftw\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb\" (UID: \"8e2c8233-f7d9-4624-99b3-e022639cbc45\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb" Oct 30 16:36:43.786026 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:43.785938 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb" Oct 30 16:36:43.805720 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:36:43.805690 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb5abe703_fee0_4185_aa22_88d1db06392b.slice/crio-3b536a4979e61f5b3b04ab6e8894c76d14b5bc2e3b690004c5346b5fd09a40bb WatchSource:0}: Error finding container 3b536a4979e61f5b3b04ab6e8894c76d14b5bc2e3b690004c5346b5fd09a40bb: Status 404 returned error can't find the container with id 3b536a4979e61f5b3b04ab6e8894c76d14b5bc2e3b690004c5346b5fd09a40bb Oct 30 16:36:43.806734 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:43.806707 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw"] Oct 30 16:36:44.128452 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:44.128419 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb"] Oct 30 16:36:44.128718 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:36:44.128698 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e2c8233_f7d9_4624_99b3_e022639cbc45.slice/crio-d4568bba043d6effa567f04f6b5b025b6a2cdb32bbe4cd899b0b9086c2b96063 WatchSource:0}: Error finding container d4568bba043d6effa567f04f6b5b025b6a2cdb32bbe4cd899b0b9086c2b96063: Status 404 returned error can't find the container with id d4568bba043d6effa567f04f6b5b025b6a2cdb32bbe4cd899b0b9086c2b96063 Oct 30 16:36:44.131921 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:44.131886 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:36:44.131996 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:44.131967 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:36:44.132034 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:44.132009 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:36:44.493731 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:44.493694 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb" event={"ID":"8e2c8233-f7d9-4624-99b3-e022639cbc45","Type":"ContainerStarted","Data":"872c385de502349f4b39cedb60ad37b3882de5fc255df10988cfb03051c7130c"} Oct 30 16:36:44.494250 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:44.493750 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb" event={"ID":"8e2c8233-f7d9-4624-99b3-e022639cbc45","Type":"ContainerStarted","Data":"d4568bba043d6effa567f04f6b5b025b6a2cdb32bbe4cd899b0b9086c2b96063"} Oct 30 16:36:44.495487 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:44.495459 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" event={"ID":"b5abe703-fee0-4185-aa22-88d1db06392b","Type":"ContainerStarted","Data":"bc6fea780408c43d6c2c4a80863af8d452c26a9fcf578f9110d9d64c4fb7a422"} Oct 30 16:36:44.495612 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:44.495501 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" event={"ID":"b5abe703-fee0-4185-aa22-88d1db06392b","Type":"ContainerStarted","Data":"3b536a4979e61f5b3b04ab6e8894c76d14b5bc2e3b690004c5346b5fd09a40bb"} Oct 30 16:36:44.542474 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:44.542417 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb" podStartSLOduration=1.5423972670000001 podStartE2EDuration="1.542397267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:36:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:36:44.538675903 +0000 UTC m=+831.765851006" watchObservedRunningTime="2026-10-30 16:36:44.542397267 +0000 UTC m=+831.769572369" Oct 30 16:36:44.582741 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:44.582690 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" podStartSLOduration=1.5826774019999998 podStartE2EDuration="1.582677402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:36:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:36:44.579076919 +0000 UTC m=+831.806252020" watchObservedRunningTime="2026-10-30 16:36:44.582677402 +0000 UTC m=+831.809852502" Oct 30 16:36:44.786257 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:44.786174 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb" Oct 30 16:36:45.485957 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:45.485924 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" Oct 30 16:36:45.787541 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:45.787508 4493 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 16:36:45.787923 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:45.787560 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb" podUID="8e2c8233-f7d9-4624-99b3-e022639cbc45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 16:36:46.787260 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:46.787226 4493 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 16:36:46.787414 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:46.787280 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb" podUID="8e2c8233-f7d9-4624-99b3-e022639cbc45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 16:36:47.301385 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:47.301345 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 16:36:47.786590 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:47.786555 4493 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 16:36:47.786807 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:47.786606 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb" podUID="8e2c8233-f7d9-4624-99b3-e022639cbc45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 16:36:48.787255 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:48.787215 4493 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 16:36:48.787816 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:48.787263 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb" podUID="8e2c8233-f7d9-4624-99b3-e022639cbc45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 16:36:49.788297 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:49.788262 4493 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 16:36:49.788971 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:49.788921 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb" podUID="8e2c8233-f7d9-4624-99b3-e022639cbc45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 16:36:50.789810 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:50.789140 4493 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 16:36:50.789810 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:50.789250 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb" podUID="8e2c8233-f7d9-4624-99b3-e022639cbc45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 16:36:52.140488 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:52.140434 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb" Oct 30 16:36:52.141050 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:52.141026 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb" Oct 30 16:36:52.142189 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:52.142171 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb" Oct 30 16:36:53.486260 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:53.486223 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" Oct 30 16:36:53.499349 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:53.499319 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" Oct 30 16:36:53.661666 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:53.661625 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" Oct 30 16:36:56.875421 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:56.875385 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c0de514-tcr8z"] Oct 30 16:36:56.880439 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:56.880418 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c0de514-tcr8z" Oct 30 16:36:56.895045 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:56.895013 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:36:56.909833 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:56.906745 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:36:56.909833 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:56.907001 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 16:36:56.912083 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:56.912054 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c0de514-tcr8z"] Oct 30 16:36:56.912196 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:56.912189 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:36:56.932917 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:56.932772 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/811cd79d-91a9-4d1f-8cdc-5864ac4ca98f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c0de514-tcr8z\" (UID: \"811cd79d-91a9-4d1f-8cdc-5864ac4ca98f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c0de514-tcr8z" Oct 30 16:36:56.933099 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:56.932899 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xwpt\" (UniqueName: \"kubernetes.io/projected/811cd79d-91a9-4d1f-8cdc-5864ac4ca98f-kube-api-access-7xwpt\") pod \"maas-api-revoke-sub-2c0de514-tcr8z\" (UID: \"811cd79d-91a9-4d1f-8cdc-5864ac4ca98f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c0de514-tcr8z" Oct 30 16:36:56.933099 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:56.932974 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/811cd79d-91a9-4d1f-8cdc-5864ac4ca98f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c0de514-tcr8z\" (UID: \"811cd79d-91a9-4d1f-8cdc-5864ac4ca98f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c0de514-tcr8z" Oct 30 16:36:57.034861 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:57.034818 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/811cd79d-91a9-4d1f-8cdc-5864ac4ca98f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c0de514-tcr8z\" (UID: \"811cd79d-91a9-4d1f-8cdc-5864ac4ca98f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c0de514-tcr8z" Oct 30 16:36:57.035040 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:57.034919 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xwpt\" (UniqueName: \"kubernetes.io/projected/811cd79d-91a9-4d1f-8cdc-5864ac4ca98f-kube-api-access-7xwpt\") pod \"maas-api-revoke-sub-2c0de514-tcr8z\" (UID: \"811cd79d-91a9-4d1f-8cdc-5864ac4ca98f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c0de514-tcr8z" Oct 30 16:36:57.035040 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:57.034961 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/811cd79d-91a9-4d1f-8cdc-5864ac4ca98f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c0de514-tcr8z\" (UID: \"811cd79d-91a9-4d1f-8cdc-5864ac4ca98f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c0de514-tcr8z" Oct 30 16:36:57.035665 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:57.035631 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/811cd79d-91a9-4d1f-8cdc-5864ac4ca98f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c0de514-tcr8z\" (UID: \"811cd79d-91a9-4d1f-8cdc-5864ac4ca98f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c0de514-tcr8z" Oct 30 16:36:57.037943 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:57.037916 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/811cd79d-91a9-4d1f-8cdc-5864ac4ca98f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c0de514-tcr8z\" (UID: \"811cd79d-91a9-4d1f-8cdc-5864ac4ca98f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c0de514-tcr8z" Oct 30 16:36:57.058418 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:57.058394 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xwpt\" (UniqueName: \"kubernetes.io/projected/811cd79d-91a9-4d1f-8cdc-5864ac4ca98f-kube-api-access-7xwpt\") pod \"maas-api-revoke-sub-2c0de514-tcr8z\" (UID: \"811cd79d-91a9-4d1f-8cdc-5864ac4ca98f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c0de514-tcr8z" Oct 30 16:36:57.194407 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:57.194374 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c0de514-tcr8z" Oct 30 16:36:57.515329 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:57.515030 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c0de514-tcr8z"] Oct 30 16:36:57.517503 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:36:57.517472 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod811cd79d_91a9_4d1f_8cdc_5864ac4ca98f.slice/crio-fe3c550a8ecb8fe78b00361eb8f28bb46cd9a6a78ae87ad377a0d310bada4af9 WatchSource:0}: Error finding container fe3c550a8ecb8fe78b00361eb8f28bb46cd9a6a78ae87ad377a0d310bada4af9: Status 404 returned error can't find the container with id fe3c550a8ecb8fe78b00361eb8f28bb46cd9a6a78ae87ad377a0d310bada4af9 Oct 30 16:36:57.694092 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:57.694046 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c0de514-tcr8z" event={"ID":"811cd79d-91a9-4d1f-8cdc-5864ac4ca98f","Type":"ContainerStarted","Data":"fe3c550a8ecb8fe78b00361eb8f28bb46cd9a6a78ae87ad377a0d310bada4af9"} Oct 30 16:36:58.702333 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:58.702240 4493 generic.go:356] "Generic (PLEG): container finished" podID="811cd79d-91a9-4d1f-8cdc-5864ac4ca98f" containerID="a2da5326a49b6564d4f278c2ad0720516043455fc6e39710d482284d53f07248" exitCode=0 Oct 30 16:36:58.702333 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:58.702299 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c0de514-tcr8z" event={"ID":"811cd79d-91a9-4d1f-8cdc-5864ac4ca98f","Type":"ContainerDied","Data":"a2da5326a49b6564d4f278c2ad0720516043455fc6e39710d482284d53f07248"} Oct 30 16:36:59.276897 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:59.274370 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr"] Oct 30 16:36:59.297809 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:59.292902 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr" Oct 30 16:36:59.338820 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:59.321441 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr"] Oct 30 16:36:59.371805 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:59.371275 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b860be8-5304-4364-a0b6-e06906cf5f4f-credential-socket\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr\" (UID: \"6b860be8-5304-4364-a0b6-e06906cf5f4f\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr" Oct 30 16:36:59.371805 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:59.371317 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b860be8-5304-4364-a0b6-e06906cf5f4f-workload-socket\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr\" (UID: \"6b860be8-5304-4364-a0b6-e06906cf5f4f\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr" Oct 30 16:36:59.371805 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:59.371425 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djkb7\" (UniqueName: \"kubernetes.io/projected/6b860be8-5304-4364-a0b6-e06906cf5f4f-kube-api-access-djkb7\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr\" (UID: \"6b860be8-5304-4364-a0b6-e06906cf5f4f\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr" Oct 30 16:36:59.371805 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:59.371490 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b860be8-5304-4364-a0b6-e06906cf5f4f-workload-certs\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr\" (UID: \"6b860be8-5304-4364-a0b6-e06906cf5f4f\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr" Oct 30 16:36:59.371805 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:59.371567 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b860be8-5304-4364-a0b6-e06906cf5f4f-istio-envoy\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr\" (UID: \"6b860be8-5304-4364-a0b6-e06906cf5f4f\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr" Oct 30 16:36:59.371805 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:59.371607 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b860be8-5304-4364-a0b6-e06906cf5f4f-istiod-ca-cert\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr\" (UID: \"6b860be8-5304-4364-a0b6-e06906cf5f4f\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr" Oct 30 16:36:59.371805 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:59.371632 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b860be8-5304-4364-a0b6-e06906cf5f4f-istio-podinfo\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr\" (UID: \"6b860be8-5304-4364-a0b6-e06906cf5f4f\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr" Oct 30 16:36:59.371805 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:59.371722 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b860be8-5304-4364-a0b6-e06906cf5f4f-istio-token\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr\" (UID: \"6b860be8-5304-4364-a0b6-e06906cf5f4f\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr" Oct 30 16:36:59.371805 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:59.371759 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b860be8-5304-4364-a0b6-e06906cf5f4f-istio-data\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr\" (UID: \"6b860be8-5304-4364-a0b6-e06906cf5f4f\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr" Oct 30 16:36:59.472688 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:59.472643 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b860be8-5304-4364-a0b6-e06906cf5f4f-istiod-ca-cert\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr\" (UID: \"6b860be8-5304-4364-a0b6-e06906cf5f4f\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr" Oct 30 16:36:59.473012 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:59.472986 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b860be8-5304-4364-a0b6-e06906cf5f4f-istio-podinfo\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr\" (UID: \"6b860be8-5304-4364-a0b6-e06906cf5f4f\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr" Oct 30 16:36:59.473164 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:59.473149 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b860be8-5304-4364-a0b6-e06906cf5f4f-istio-token\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr\" (UID: \"6b860be8-5304-4364-a0b6-e06906cf5f4f\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr" Oct 30 16:36:59.473272 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:59.473258 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b860be8-5304-4364-a0b6-e06906cf5f4f-istio-data\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr\" (UID: \"6b860be8-5304-4364-a0b6-e06906cf5f4f\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr" Oct 30 16:36:59.473460 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:59.473446 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b860be8-5304-4364-a0b6-e06906cf5f4f-credential-socket\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr\" (UID: \"6b860be8-5304-4364-a0b6-e06906cf5f4f\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr" Oct 30 16:36:59.473567 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:59.473554 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b860be8-5304-4364-a0b6-e06906cf5f4f-workload-socket\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr\" (UID: \"6b860be8-5304-4364-a0b6-e06906cf5f4f\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr" Oct 30 16:36:59.473676 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:59.473662 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djkb7\" (UniqueName: \"kubernetes.io/projected/6b860be8-5304-4364-a0b6-e06906cf5f4f-kube-api-access-djkb7\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr\" (UID: \"6b860be8-5304-4364-a0b6-e06906cf5f4f\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr" Oct 30 16:36:59.473827 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:59.473771 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b860be8-5304-4364-a0b6-e06906cf5f4f-workload-certs\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr\" (UID: \"6b860be8-5304-4364-a0b6-e06906cf5f4f\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr" Oct 30 16:36:59.473992 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:59.473977 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b860be8-5304-4364-a0b6-e06906cf5f4f-istio-envoy\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr\" (UID: \"6b860be8-5304-4364-a0b6-e06906cf5f4f\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr" Oct 30 16:36:59.476750 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:59.475875 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b860be8-5304-4364-a0b6-e06906cf5f4f-credential-socket\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr\" (UID: \"6b860be8-5304-4364-a0b6-e06906cf5f4f\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr" Oct 30 16:36:59.476750 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:59.476486 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b860be8-5304-4364-a0b6-e06906cf5f4f-istiod-ca-cert\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr\" (UID: \"6b860be8-5304-4364-a0b6-e06906cf5f4f\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr" Oct 30 16:36:59.477159 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:59.477137 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b860be8-5304-4364-a0b6-e06906cf5f4f-workload-certs\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr\" (UID: \"6b860be8-5304-4364-a0b6-e06906cf5f4f\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr" Oct 30 16:36:59.478034 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:59.477964 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b860be8-5304-4364-a0b6-e06906cf5f4f-istio-data\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr\" (UID: \"6b860be8-5304-4364-a0b6-e06906cf5f4f\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr" Oct 30 16:36:59.478206 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:59.478189 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b860be8-5304-4364-a0b6-e06906cf5f4f-workload-socket\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr\" (UID: \"6b860be8-5304-4364-a0b6-e06906cf5f4f\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr" Oct 30 16:36:59.484037 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:59.483999 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b860be8-5304-4364-a0b6-e06906cf5f4f-istio-podinfo\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr\" (UID: \"6b860be8-5304-4364-a0b6-e06906cf5f4f\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr" Oct 30 16:36:59.500544 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:59.500443 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djkb7\" (UniqueName: \"kubernetes.io/projected/6b860be8-5304-4364-a0b6-e06906cf5f4f-kube-api-access-djkb7\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr\" (UID: \"6b860be8-5304-4364-a0b6-e06906cf5f4f\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr" Oct 30 16:36:59.505807 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:59.504826 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b860be8-5304-4364-a0b6-e06906cf5f4f-istio-token\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr\" (UID: \"6b860be8-5304-4364-a0b6-e06906cf5f4f\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr" Oct 30 16:36:59.520817 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:59.517832 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b860be8-5304-4364-a0b6-e06906cf5f4f-istio-envoy\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr\" (UID: \"6b860be8-5304-4364-a0b6-e06906cf5f4f\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr" Oct 30 16:36:59.662413 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:59.662383 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr" Oct 30 16:36:59.951101 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:59.951073 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c0de514-tcr8z" Oct 30 16:36:59.982809 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:59.981978 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/811cd79d-91a9-4d1f-8cdc-5864ac4ca98f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/811cd79d-91a9-4d1f-8cdc-5864ac4ca98f-maas-api-xks-ca\") pod \"811cd79d-91a9-4d1f-8cdc-5864ac4ca98f\" (UID: \"811cd79d-91a9-4d1f-8cdc-5864ac4ca98f\") " Oct 30 16:36:59.982809 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:59.982078 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/811cd79d-91a9-4d1f-8cdc-5864ac4ca98f-kube-api-access-7xwpt\" (UniqueName: \"kubernetes.io/projected/811cd79d-91a9-4d1f-8cdc-5864ac4ca98f-kube-api-access-7xwpt\") pod \"811cd79d-91a9-4d1f-8cdc-5864ac4ca98f\" (UID: \"811cd79d-91a9-4d1f-8cdc-5864ac4ca98f\") " Oct 30 16:36:59.983022 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:59.982860 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/811cd79d-91a9-4d1f-8cdc-5864ac4ca98f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/811cd79d-91a9-4d1f-8cdc-5864ac4ca98f-maas-api-openshift-ca\") pod \"811cd79d-91a9-4d1f-8cdc-5864ac4ca98f\" (UID: \"811cd79d-91a9-4d1f-8cdc-5864ac4ca98f\") " Oct 30 16:36:59.984569 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:59.983367 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/811cd79d-91a9-4d1f-8cdc-5864ac4ca98f-maas-api-openshift-ca" pod "811cd79d-91a9-4d1f-8cdc-5864ac4ca98f" (UID: "811cd79d-91a9-4d1f-8cdc-5864ac4ca98f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:36:59.984569 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:59.983992 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/811cd79d-91a9-4d1f-8cdc-5864ac4ca98f-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:37:00.000932 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:59.989912 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/811cd79d-91a9-4d1f-8cdc-5864ac4ca98f-kube-api-access-7xwpt" pod "811cd79d-91a9-4d1f-8cdc-5864ac4ca98f" (UID: "811cd79d-91a9-4d1f-8cdc-5864ac4ca98f"). InnerVolumeSpecName "kube-api-access-7xwpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:37:00.000932 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:36:59.990297 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/811cd79d-91a9-4d1f-8cdc-5864ac4ca98f-maas-api-xks-ca" pod "811cd79d-91a9-4d1f-8cdc-5864ac4ca98f" (UID: "811cd79d-91a9-4d1f-8cdc-5864ac4ca98f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:37:00.013303 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:00.011921 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr"] Oct 30 16:37:00.088072 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:00.082406 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:37:00.088072 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:00.082499 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:37:00.088072 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:00.082545 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:37:00.088072 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:00.085734 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/811cd79d-91a9-4d1f-8cdc-5864ac4ca98f-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:37:00.088072 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:00.085760 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xwpt\" (UniqueName: \"kubernetes.io/projected/811cd79d-91a9-4d1f-8cdc-5864ac4ca98f-kube-api-access-7xwpt\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:37:00.486862 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:37:00.485650 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 16:37:00.774863 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:00.773991 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr" event={"ID":"6b860be8-5304-4364-a0b6-e06906cf5f4f","Type":"ContainerStarted","Data":"8f5c470a5e709f6e2880deb6300e233b992e725dcc8a499b5ef07b7ced37b460"} Oct 30 16:37:00.774863 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:00.774056 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr" event={"ID":"6b860be8-5304-4364-a0b6-e06906cf5f4f","Type":"ContainerStarted","Data":"246d6c2c3f96a72e3322bddb3e4e241065231f4d9e5744700c0265e6f9d18a78"} Oct 30 16:37:00.783938 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:00.783613 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c0de514-tcr8z" event={"ID":"811cd79d-91a9-4d1f-8cdc-5864ac4ca98f","Type":"ContainerDied","Data":"fe3c550a8ecb8fe78b00361eb8f28bb46cd9a6a78ae87ad377a0d310bada4af9"} Oct 30 16:37:00.783938 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:00.783673 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe3c550a8ecb8fe78b00361eb8f28bb46cd9a6a78ae87ad377a0d310bada4af9" Oct 30 16:37:00.783938 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:00.783738 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c0de514-tcr8z" Oct 30 16:37:00.879408 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:00.879322 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr" podStartSLOduration=1.879299827 podStartE2EDuration="1.879299827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:36:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:37:00.845651068 +0000 UTC m=+848.072826168" watchObservedRunningTime="2026-10-30 16:37:00.879299827 +0000 UTC m=+848.106474928" Oct 30 16:37:01.662897 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:01.662845 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr" Oct 30 16:37:01.664412 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:01.664388 4493 patch_prober.go:29] interesting pod/e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 16:37:01.664536 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:01.664434 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr" podUID="6b860be8-5304-4364-a0b6-e06906cf5f4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 16:37:02.663105 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:02.663071 4493 patch_prober.go:29] interesting pod/e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 16:37:02.663513 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:02.663147 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr" podUID="6b860be8-5304-4364-a0b6-e06906cf5f4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 16:37:03.662741 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:03.662711 4493 patch_prober.go:29] interesting pod/e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 16:37:03.662899 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:03.662772 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr" podUID="6b860be8-5304-4364-a0b6-e06906cf5f4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 16:37:04.663212 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:04.663171 4493 patch_prober.go:29] interesting pod/e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 16:37:04.663705 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:04.663232 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr" podUID="6b860be8-5304-4364-a0b6-e06906cf5f4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 16:37:05.663483 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:05.663447 4493 patch_prober.go:29] interesting pod/e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 16:37:05.664011 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:05.663522 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr" podUID="6b860be8-5304-4364-a0b6-e06906cf5f4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 16:37:06.663200 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:06.663160 4493 patch_prober.go:29] interesting pod/e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 16:37:06.663383 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:06.663235 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr" podUID="6b860be8-5304-4364-a0b6-e06906cf5f4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 16:37:08.089867 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:08.089828 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr" Oct 30 16:37:08.090628 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:08.090601 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr" Oct 30 16:37:08.091734 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:08.091706 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr" Oct 30 16:37:11.760731 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:11.760695 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe256302-h86vt"] Oct 30 16:37:11.761235 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:11.761222 4493 cpu_manager.go:414] "containerMap: removing container" podUID="811cd79d-91a9-4d1f-8cdc-5864ac4ca98f" containerName="revoke-keys" Oct 30 16:37:11.761294 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:11.761237 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="811cd79d-91a9-4d1f-8cdc-5864ac4ca98f" containerName="revoke-keys" Oct 30 16:37:11.761330 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:11.761314 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="811cd79d-91a9-4d1f-8cdc-5864ac4ca98f" containerName="revoke-keys" Oct 30 16:37:11.764704 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:11.764684 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe256302-h86vt" Oct 30 16:37:11.779244 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:11.779215 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:37:11.791115 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:11.791070 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 16:37:11.791274 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:11.791131 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:37:11.791490 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:11.791466 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:37:11.796060 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:11.796037 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe256302-h86vt"] Oct 30 16:37:11.860197 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:11.860167 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z777j\" (UniqueName: \"kubernetes.io/projected/303b0588-e73c-4d56-abef-328644e8ebe2-kube-api-access-z777j\") pod \"maas-api-revoke-sub-fe256302-h86vt\" (UID: \"303b0588-e73c-4d56-abef-328644e8ebe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe256302-h86vt" Oct 30 16:37:11.860368 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:11.860212 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/303b0588-e73c-4d56-abef-328644e8ebe2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe256302-h86vt\" (UID: \"303b0588-e73c-4d56-abef-328644e8ebe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe256302-h86vt" Oct 30 16:37:11.860423 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:11.860402 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/303b0588-e73c-4d56-abef-328644e8ebe2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe256302-h86vt\" (UID: \"303b0588-e73c-4d56-abef-328644e8ebe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe256302-h86vt" Oct 30 16:37:11.961656 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:11.961626 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/303b0588-e73c-4d56-abef-328644e8ebe2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe256302-h86vt\" (UID: \"303b0588-e73c-4d56-abef-328644e8ebe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe256302-h86vt" Oct 30 16:37:11.961848 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:11.961695 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z777j\" (UniqueName: \"kubernetes.io/projected/303b0588-e73c-4d56-abef-328644e8ebe2-kube-api-access-z777j\") pod \"maas-api-revoke-sub-fe256302-h86vt\" (UID: \"303b0588-e73c-4d56-abef-328644e8ebe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe256302-h86vt" Oct 30 16:37:11.961848 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:11.961718 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/303b0588-e73c-4d56-abef-328644e8ebe2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe256302-h86vt\" (UID: \"303b0588-e73c-4d56-abef-328644e8ebe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe256302-h86vt" Oct 30 16:37:11.962368 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:11.962332 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/303b0588-e73c-4d56-abef-328644e8ebe2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe256302-h86vt\" (UID: \"303b0588-e73c-4d56-abef-328644e8ebe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe256302-h86vt" Oct 30 16:37:11.964497 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:11.964477 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/303b0588-e73c-4d56-abef-328644e8ebe2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe256302-h86vt\" (UID: \"303b0588-e73c-4d56-abef-328644e8ebe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe256302-h86vt" Oct 30 16:37:11.985428 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:11.985400 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z777j\" (UniqueName: \"kubernetes.io/projected/303b0588-e73c-4d56-abef-328644e8ebe2-kube-api-access-z777j\") pod \"maas-api-revoke-sub-fe256302-h86vt\" (UID: \"303b0588-e73c-4d56-abef-328644e8ebe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe256302-h86vt" Oct 30 16:37:12.078636 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:12.078575 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe256302-h86vt" Oct 30 16:37:12.382814 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:12.382761 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe256302-h86vt"] Oct 30 16:37:12.384284 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:37:12.384261 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod303b0588_e73c_4d56_abef_328644e8ebe2.slice/crio-e901080c8ccad36eec8602fd945a10c43d679b74d16b656a9f1255e7b1cea3ea WatchSource:0}: Error finding container e901080c8ccad36eec8602fd945a10c43d679b74d16b656a9f1255e7b1cea3ea: Status 404 returned error can't find the container with id e901080c8ccad36eec8602fd945a10c43d679b74d16b656a9f1255e7b1cea3ea Oct 30 16:37:12.903283 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:12.903250 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe256302-h86vt" event={"ID":"303b0588-e73c-4d56-abef-328644e8ebe2","Type":"ContainerStarted","Data":"457a1a65ce23b0fb987df44f19c512eebcacefb605eac7d90a44f55ae922dfad"} Oct 30 16:37:12.903664 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:12.903306 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe256302-h86vt" event={"ID":"303b0588-e73c-4d56-abef-328644e8ebe2","Type":"ContainerStarted","Data":"e901080c8ccad36eec8602fd945a10c43d679b74d16b656a9f1255e7b1cea3ea"} Oct 30 16:37:12.947752 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:12.947704 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe256302-h86vt" podStartSLOduration=1.9476917089999999 podStartE2EDuration="1.947691709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:37:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:37:12.946219087 +0000 UTC m=+860.173394187" watchObservedRunningTime="2026-10-30 16:37:12.947691709 +0000 UTC m=+860.174866847" Oct 30 16:37:12.976238 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:12.976171 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9"] Oct 30 16:37:12.980384 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:12.980363 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9" Oct 30 16:37:13.028761 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:13.028731 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9"] Oct 30 16:37:13.075955 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:13.075924 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b08f59e7-615a-4d20-ba53-ad4bcf0d5728-istio-podinfo\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9\" (UID: \"b08f59e7-615a-4d20-ba53-ad4bcf0d5728\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9" Oct 30 16:37:13.076116 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:13.076045 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b08f59e7-615a-4d20-ba53-ad4bcf0d5728-istio-envoy\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9\" (UID: \"b08f59e7-615a-4d20-ba53-ad4bcf0d5728\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9" Oct 30 16:37:13.076116 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:13.076076 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhsn2\" (UniqueName: \"kubernetes.io/projected/b08f59e7-615a-4d20-ba53-ad4bcf0d5728-kube-api-access-hhsn2\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9\" (UID: \"b08f59e7-615a-4d20-ba53-ad4bcf0d5728\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9" Oct 30 16:37:13.076116 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:13.076096 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b08f59e7-615a-4d20-ba53-ad4bcf0d5728-workload-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9\" (UID: \"b08f59e7-615a-4d20-ba53-ad4bcf0d5728\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9" Oct 30 16:37:13.076222 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:13.076120 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b08f59e7-615a-4d20-ba53-ad4bcf0d5728-istio-data\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9\" (UID: \"b08f59e7-615a-4d20-ba53-ad4bcf0d5728\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9" Oct 30 16:37:13.076222 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:13.076146 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b08f59e7-615a-4d20-ba53-ad4bcf0d5728-istiod-ca-cert\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9\" (UID: \"b08f59e7-615a-4d20-ba53-ad4bcf0d5728\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9" Oct 30 16:37:13.076222 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:13.076201 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b08f59e7-615a-4d20-ba53-ad4bcf0d5728-workload-certs\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9\" (UID: \"b08f59e7-615a-4d20-ba53-ad4bcf0d5728\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9" Oct 30 16:37:13.076324 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:13.076249 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b08f59e7-615a-4d20-ba53-ad4bcf0d5728-credential-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9\" (UID: \"b08f59e7-615a-4d20-ba53-ad4bcf0d5728\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9" Oct 30 16:37:13.076563 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:13.076545 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b08f59e7-615a-4d20-ba53-ad4bcf0d5728-istio-token\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9\" (UID: \"b08f59e7-615a-4d20-ba53-ad4bcf0d5728\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9" Oct 30 16:37:13.177631 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:13.177590 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b08f59e7-615a-4d20-ba53-ad4bcf0d5728-istiod-ca-cert\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9\" (UID: \"b08f59e7-615a-4d20-ba53-ad4bcf0d5728\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9" Oct 30 16:37:13.177857 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:13.177659 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b08f59e7-615a-4d20-ba53-ad4bcf0d5728-workload-certs\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9\" (UID: \"b08f59e7-615a-4d20-ba53-ad4bcf0d5728\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9" Oct 30 16:37:13.177857 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:13.177705 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b08f59e7-615a-4d20-ba53-ad4bcf0d5728-credential-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9\" (UID: \"b08f59e7-615a-4d20-ba53-ad4bcf0d5728\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9" Oct 30 16:37:13.177857 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:13.177773 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b08f59e7-615a-4d20-ba53-ad4bcf0d5728-istio-token\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9\" (UID: \"b08f59e7-615a-4d20-ba53-ad4bcf0d5728\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9" Oct 30 16:37:13.177857 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:13.177852 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b08f59e7-615a-4d20-ba53-ad4bcf0d5728-istio-podinfo\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9\" (UID: \"b08f59e7-615a-4d20-ba53-ad4bcf0d5728\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9" Oct 30 16:37:13.178088 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:13.177921 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b08f59e7-615a-4d20-ba53-ad4bcf0d5728-istio-envoy\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9\" (UID: \"b08f59e7-615a-4d20-ba53-ad4bcf0d5728\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9" Oct 30 16:37:13.178088 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:13.177963 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhsn2\" (UniqueName: \"kubernetes.io/projected/b08f59e7-615a-4d20-ba53-ad4bcf0d5728-kube-api-access-hhsn2\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9\" (UID: \"b08f59e7-615a-4d20-ba53-ad4bcf0d5728\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9" Oct 30 16:37:13.178088 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:13.178002 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b08f59e7-615a-4d20-ba53-ad4bcf0d5728-workload-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9\" (UID: \"b08f59e7-615a-4d20-ba53-ad4bcf0d5728\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9" Oct 30 16:37:13.178088 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:13.178032 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b08f59e7-615a-4d20-ba53-ad4bcf0d5728-istio-data\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9\" (UID: \"b08f59e7-615a-4d20-ba53-ad4bcf0d5728\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9" Oct 30 16:37:13.178297 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:13.178104 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b08f59e7-615a-4d20-ba53-ad4bcf0d5728-workload-certs\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9\" (UID: \"b08f59e7-615a-4d20-ba53-ad4bcf0d5728\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9" Oct 30 16:37:13.178297 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:13.178174 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b08f59e7-615a-4d20-ba53-ad4bcf0d5728-credential-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9\" (UID: \"b08f59e7-615a-4d20-ba53-ad4bcf0d5728\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9" Oct 30 16:37:13.178400 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:13.178375 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b08f59e7-615a-4d20-ba53-ad4bcf0d5728-istio-data\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9\" (UID: \"b08f59e7-615a-4d20-ba53-ad4bcf0d5728\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9" Oct 30 16:37:13.178455 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:13.178419 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b08f59e7-615a-4d20-ba53-ad4bcf0d5728-workload-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9\" (UID: \"b08f59e7-615a-4d20-ba53-ad4bcf0d5728\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9" Oct 30 16:37:13.178455 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:13.178428 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b08f59e7-615a-4d20-ba53-ad4bcf0d5728-istiod-ca-cert\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9\" (UID: \"b08f59e7-615a-4d20-ba53-ad4bcf0d5728\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9" Oct 30 16:37:13.181233 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:13.181042 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b08f59e7-615a-4d20-ba53-ad4bcf0d5728-istio-podinfo\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9\" (UID: \"b08f59e7-615a-4d20-ba53-ad4bcf0d5728\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9" Oct 30 16:37:13.181233 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:13.181203 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b08f59e7-615a-4d20-ba53-ad4bcf0d5728-istio-envoy\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9\" (UID: \"b08f59e7-615a-4d20-ba53-ad4bcf0d5728\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9" Oct 30 16:37:13.201058 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:13.201028 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b08f59e7-615a-4d20-ba53-ad4bcf0d5728-istio-token\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9\" (UID: \"b08f59e7-615a-4d20-ba53-ad4bcf0d5728\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9" Oct 30 16:37:13.201234 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:13.201156 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhsn2\" (UniqueName: \"kubernetes.io/projected/b08f59e7-615a-4d20-ba53-ad4bcf0d5728-kube-api-access-hhsn2\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9\" (UID: \"b08f59e7-615a-4d20-ba53-ad4bcf0d5728\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9" Oct 30 16:37:13.296312 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:13.296229 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9" Oct 30 16:37:13.621835 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:13.621700 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9"] Oct 30 16:37:13.624503 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:37:13.624474 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb08f59e7_615a_4d20_ba53_ad4bcf0d5728.slice/crio-0ba42f6b293d42ad617f1f10e623ddaa2d56fe672d99999bae17b576a305ad62 WatchSource:0}: Error finding container 0ba42f6b293d42ad617f1f10e623ddaa2d56fe672d99999bae17b576a305ad62: Status 404 returned error can't find the container with id 0ba42f6b293d42ad617f1f10e623ddaa2d56fe672d99999bae17b576a305ad62 Oct 30 16:37:13.627635 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:13.627593 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:37:13.627753 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:13.627690 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:37:13.627753 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:13.627735 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:37:13.913480 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:13.913442 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9" event={"ID":"b08f59e7-615a-4d20-ba53-ad4bcf0d5728","Type":"ContainerStarted","Data":"e4801a8260760009ece4694c2a4c932f5949126a2ef461d9a34735bd6a2367cd"} Oct 30 16:37:13.914002 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:13.913507 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9" event={"ID":"b08f59e7-615a-4d20-ba53-ad4bcf0d5728","Type":"ContainerStarted","Data":"0ba42f6b293d42ad617f1f10e623ddaa2d56fe672d99999bae17b576a305ad62"} Oct 30 16:37:13.915745 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:13.915718 4493 generic.go:356] "Generic (PLEG): container finished" podID="303b0588-e73c-4d56-abef-328644e8ebe2" containerID="457a1a65ce23b0fb987df44f19c512eebcacefb605eac7d90a44f55ae922dfad" exitCode=0 Oct 30 16:37:13.915958 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:13.915848 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe256302-h86vt" event={"ID":"303b0588-e73c-4d56-abef-328644e8ebe2","Type":"ContainerDied","Data":"457a1a65ce23b0fb987df44f19c512eebcacefb605eac7d90a44f55ae922dfad"} Oct 30 16:37:13.960752 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:13.960708 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9" podStartSLOduration=1.960692144 podStartE2EDuration="1.960692144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:37:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:37:13.958613342 +0000 UTC m=+861.185788442" watchObservedRunningTime="2026-10-30 16:37:13.960692144 +0000 UTC m=+861.187867247" Oct 30 16:37:14.297175 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:14.297094 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9" Oct 30 16:37:14.298829 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:14.298797 4493 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 16:37:14.298973 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:14.298848 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9" podUID="b08f59e7-615a-4d20-ba53-ad4bcf0d5728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 16:37:15.181123 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:15.181098 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe256302-h86vt" Oct 30 16:37:15.296775 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:15.296699 4493 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 16:37:15.296775 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:15.296746 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9" podUID="b08f59e7-615a-4d20-ba53-ad4bcf0d5728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 16:37:15.313340 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:15.313304 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/303b0588-e73c-4d56-abef-328644e8ebe2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/303b0588-e73c-4d56-abef-328644e8ebe2-maas-api-xks-ca\") pod \"303b0588-e73c-4d56-abef-328644e8ebe2\" (UID: \"303b0588-e73c-4d56-abef-328644e8ebe2\") " Oct 30 16:37:15.314012 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:15.313989 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/303b0588-e73c-4d56-abef-328644e8ebe2-kube-api-access-z777j\" (UniqueName: \"kubernetes.io/projected/303b0588-e73c-4d56-abef-328644e8ebe2-kube-api-access-z777j\") pod \"303b0588-e73c-4d56-abef-328644e8ebe2\" (UID: \"303b0588-e73c-4d56-abef-328644e8ebe2\") " Oct 30 16:37:15.314193 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:15.314177 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/303b0588-e73c-4d56-abef-328644e8ebe2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/303b0588-e73c-4d56-abef-328644e8ebe2-maas-api-openshift-ca\") pod \"303b0588-e73c-4d56-abef-328644e8ebe2\" (UID: \"303b0588-e73c-4d56-abef-328644e8ebe2\") " Oct 30 16:37:15.320146 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:15.320119 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/303b0588-e73c-4d56-abef-328644e8ebe2-maas-api-openshift-ca" pod "303b0588-e73c-4d56-abef-328644e8ebe2" (UID: "303b0588-e73c-4d56-abef-328644e8ebe2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:37:15.329813 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:15.328914 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/303b0588-e73c-4d56-abef-328644e8ebe2-maas-api-xks-ca" pod "303b0588-e73c-4d56-abef-328644e8ebe2" (UID: "303b0588-e73c-4d56-abef-328644e8ebe2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:37:15.340695 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:15.340571 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/303b0588-e73c-4d56-abef-328644e8ebe2-kube-api-access-z777j" pod "303b0588-e73c-4d56-abef-328644e8ebe2" (UID: "303b0588-e73c-4d56-abef-328644e8ebe2"). InnerVolumeSpecName "kube-api-access-z777j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:37:15.427775 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:15.427730 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/303b0588-e73c-4d56-abef-328644e8ebe2-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:37:15.427775 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:15.427774 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z777j\" (UniqueName: \"kubernetes.io/projected/303b0588-e73c-4d56-abef-328644e8ebe2-kube-api-access-z777j\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:37:15.428045 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:15.427812 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/303b0588-e73c-4d56-abef-328644e8ebe2-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:37:15.950941 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:15.950907 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe256302-h86vt" Oct 30 16:37:15.950941 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:15.950926 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe256302-h86vt" event={"ID":"303b0588-e73c-4d56-abef-328644e8ebe2","Type":"ContainerDied","Data":"e901080c8ccad36eec8602fd945a10c43d679b74d16b656a9f1255e7b1cea3ea"} Oct 30 16:37:15.951182 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:15.950983 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e901080c8ccad36eec8602fd945a10c43d679b74d16b656a9f1255e7b1cea3ea" Oct 30 16:37:16.090188 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:16.090157 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 16:37:16.297350 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:16.297262 4493 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 16:37:16.297350 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:16.297328 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9" podUID="b08f59e7-615a-4d20-ba53-ad4bcf0d5728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 16:37:17.352811 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:17.351037 4493 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 16:37:17.352811 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:17.351099 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9" podUID="b08f59e7-615a-4d20-ba53-ad4bcf0d5728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 16:37:17.858820 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:17.852753 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7"] Oct 30 16:37:17.858820 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:17.853646 4493 cpu_manager.go:414] "containerMap: removing container" podUID="303b0588-e73c-4d56-abef-328644e8ebe2" containerName="revoke-keys" Oct 30 16:37:17.858820 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:17.853668 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="303b0588-e73c-4d56-abef-328644e8ebe2" containerName="revoke-keys" Oct 30 16:37:17.879814 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:17.864882 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="303b0588-e73c-4d56-abef-328644e8ebe2" containerName="revoke-keys" Oct 30 16:37:17.901377 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:17.900417 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7" Oct 30 16:37:17.901572 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:17.901553 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7"] Oct 30 16:37:17.920813 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:17.917622 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-420ff3-openshift-default-dockercfg-mlfh4\"" Oct 30 16:37:18.057810 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:18.036866 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efbed66f-95b0-4fd7-af06-388840a2a364-istio-podinfo\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7\" (UID: \"efbed66f-95b0-4fd7-af06-388840a2a364\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7" Oct 30 16:37:18.067809 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:18.058088 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efbed66f-95b0-4fd7-af06-388840a2a364-istio-data\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7\" (UID: \"efbed66f-95b0-4fd7-af06-388840a2a364\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7" Oct 30 16:37:18.067809 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:18.058165 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efbed66f-95b0-4fd7-af06-388840a2a364-istiod-ca-cert\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7\" (UID: \"efbed66f-95b0-4fd7-af06-388840a2a364\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7" Oct 30 16:37:18.067809 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:18.058294 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efbed66f-95b0-4fd7-af06-388840a2a364-istio-token\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7\" (UID: \"efbed66f-95b0-4fd7-af06-388840a2a364\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7" Oct 30 16:37:18.067809 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:18.058398 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efbed66f-95b0-4fd7-af06-388840a2a364-workload-socket\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7\" (UID: \"efbed66f-95b0-4fd7-af06-388840a2a364\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7" Oct 30 16:37:18.067809 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:18.058474 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efbed66f-95b0-4fd7-af06-388840a2a364-istio-envoy\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7\" (UID: \"efbed66f-95b0-4fd7-af06-388840a2a364\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7" Oct 30 16:37:18.067809 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:18.058535 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efbed66f-95b0-4fd7-af06-388840a2a364-workload-certs\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7\" (UID: \"efbed66f-95b0-4fd7-af06-388840a2a364\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7" Oct 30 16:37:18.067809 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:18.058567 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efbed66f-95b0-4fd7-af06-388840a2a364-credential-socket\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7\" (UID: \"efbed66f-95b0-4fd7-af06-388840a2a364\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7" Oct 30 16:37:18.067809 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:18.058693 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwshr\" (UniqueName: \"kubernetes.io/projected/efbed66f-95b0-4fd7-af06-388840a2a364-kube-api-access-gwshr\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7\" (UID: \"efbed66f-95b0-4fd7-af06-388840a2a364\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7" Oct 30 16:37:18.170816 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:18.161462 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efbed66f-95b0-4fd7-af06-388840a2a364-istiod-ca-cert\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7\" (UID: \"efbed66f-95b0-4fd7-af06-388840a2a364\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7" Oct 30 16:37:18.171146 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:18.171119 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efbed66f-95b0-4fd7-af06-388840a2a364-istio-token\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7\" (UID: \"efbed66f-95b0-4fd7-af06-388840a2a364\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7" Oct 30 16:37:18.171327 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:18.171309 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efbed66f-95b0-4fd7-af06-388840a2a364-workload-socket\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7\" (UID: \"efbed66f-95b0-4fd7-af06-388840a2a364\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7" Oct 30 16:37:18.171465 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:18.171449 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efbed66f-95b0-4fd7-af06-388840a2a364-istio-envoy\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7\" (UID: \"efbed66f-95b0-4fd7-af06-388840a2a364\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7" Oct 30 16:37:18.171617 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:18.171602 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efbed66f-95b0-4fd7-af06-388840a2a364-workload-certs\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7\" (UID: \"efbed66f-95b0-4fd7-af06-388840a2a364\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7" Oct 30 16:37:18.171728 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:18.171712 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efbed66f-95b0-4fd7-af06-388840a2a364-credential-socket\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7\" (UID: \"efbed66f-95b0-4fd7-af06-388840a2a364\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7" Oct 30 16:37:18.176009 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:18.175983 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwshr\" (UniqueName: \"kubernetes.io/projected/efbed66f-95b0-4fd7-af06-388840a2a364-kube-api-access-gwshr\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7\" (UID: \"efbed66f-95b0-4fd7-af06-388840a2a364\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7" Oct 30 16:37:18.176190 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:18.176172 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efbed66f-95b0-4fd7-af06-388840a2a364-istio-podinfo\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7\" (UID: \"efbed66f-95b0-4fd7-af06-388840a2a364\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7" Oct 30 16:37:18.176314 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:18.176299 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efbed66f-95b0-4fd7-af06-388840a2a364-istio-data\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7\" (UID: \"efbed66f-95b0-4fd7-af06-388840a2a364\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7" Oct 30 16:37:18.200805 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:18.176884 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efbed66f-95b0-4fd7-af06-388840a2a364-istio-data\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7\" (UID: \"efbed66f-95b0-4fd7-af06-388840a2a364\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7" Oct 30 16:37:18.202110 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:18.202082 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efbed66f-95b0-4fd7-af06-388840a2a364-workload-certs\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7\" (UID: \"efbed66f-95b0-4fd7-af06-388840a2a364\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7" Oct 30 16:37:18.211860 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:18.202737 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efbed66f-95b0-4fd7-af06-388840a2a364-istiod-ca-cert\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7\" (UID: \"efbed66f-95b0-4fd7-af06-388840a2a364\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7" Oct 30 16:37:18.212021 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:18.203190 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efbed66f-95b0-4fd7-af06-388840a2a364-credential-socket\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7\" (UID: \"efbed66f-95b0-4fd7-af06-388840a2a364\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7" Oct 30 16:37:18.212144 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:18.203378 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efbed66f-95b0-4fd7-af06-388840a2a364-workload-socket\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7\" (UID: \"efbed66f-95b0-4fd7-af06-388840a2a364\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7" Oct 30 16:37:18.248804 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:18.248700 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efbed66f-95b0-4fd7-af06-388840a2a364-istio-podinfo\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7\" (UID: \"efbed66f-95b0-4fd7-af06-388840a2a364\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7" Oct 30 16:37:18.249139 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:18.249113 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efbed66f-95b0-4fd7-af06-388840a2a364-istio-envoy\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7\" (UID: \"efbed66f-95b0-4fd7-af06-388840a2a364\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7" Oct 30 16:37:18.268629 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:18.268597 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efbed66f-95b0-4fd7-af06-388840a2a364-istio-token\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7\" (UID: \"efbed66f-95b0-4fd7-af06-388840a2a364\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7" Oct 30 16:37:18.279808 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:18.279461 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwshr\" (UniqueName: \"kubernetes.io/projected/efbed66f-95b0-4fd7-af06-388840a2a364-kube-api-access-gwshr\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7\" (UID: \"efbed66f-95b0-4fd7-af06-388840a2a364\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7" Oct 30 16:37:18.298215 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:18.298183 4493 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 16:37:18.298474 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:18.298440 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9" podUID="b08f59e7-615a-4d20-ba53-ad4bcf0d5728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 16:37:18.537180 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:18.537073 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7" Oct 30 16:37:18.733810 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:18.733088 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" podUID="08baad08-cd72-4305-b92d-c11a84ef276f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:18.814069 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:18.812875 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" podUID="e76172b1-659d-45bd-91c6-1a0ce4062335" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:18.969811 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:18.966878 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" podUID="18b72648-f523-405e-b5cd-1266ff13f8f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:19.396837 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:19.385123 4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" podUID="80b9bd9a-30d6-45ff-b318-26fcdc608e5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:19.396837 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:19.387413 4493 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 16:37:19.396837 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:19.387453 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9" podUID="b08f59e7-615a-4d20-ba53-ad4bcf0d5728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 16:37:19.499569 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:19.499532 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7"] Oct 30 16:37:19.615816 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:19.603599 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" podUID="80b9bd9a-30d6-45ff-b318-26fcdc608e5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:19.738886 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:19.734102 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" podUID="08baad08-cd72-4305-b92d-c11a84ef276f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:19.738886 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:19.734872 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" Oct 30 16:37:19.738886 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:19.736305 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" podUID="14805045-a727-41ba-8bd0-100d41f9f27d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:19.774057 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:37:19.770863 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefbed66f_95b0_4fd7_af06_388840a2a364.slice/crio-4e3ae3d764c3ca01821bfda6d3aba9854fce4456ff97b0e246fad34816bd1188 WatchSource:0}: Error finding container 4e3ae3d764c3ca01821bfda6d3aba9854fce4456ff97b0e246fad34816bd1188: Status 404 returned error can't find the container with id 4e3ae3d764c3ca01821bfda6d3aba9854fce4456ff97b0e246fad34816bd1188 Oct 30 16:37:19.879848 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:19.875729 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" podUID="e76172b1-659d-45bd-91c6-1a0ce4062335" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 16:37:19.879848 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:19.876119 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" Oct 30 16:37:19.882560 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:19.882526 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:37:19.882671 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:19.882609 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:37:19.882671 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:19.882652 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:37:19.885564 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:19.885544 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" Oct 30 16:37:19.897946 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:19.897918 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" Oct 30 16:37:20.071315 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:20.071285 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7" event={"ID":"efbed66f-95b0-4fd7-af06-388840a2a364","Type":"ContainerStarted","Data":"4e3ae3d764c3ca01821bfda6d3aba9854fce4456ff97b0e246fad34816bd1188"} Oct 30 16:37:20.297705 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:20.297618 4493 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 16:37:20.297705 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:20.297684 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9" podUID="b08f59e7-615a-4d20-ba53-ad4bcf0d5728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 16:37:20.972330 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:20.972293 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ffa75a6-4l9c8"] Oct 30 16:37:20.976501 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:20.976480 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ffa75a6-4l9c8" Oct 30 16:37:20.989238 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:20.988324 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ffa75a6-4l9c8"] Oct 30 16:37:20.991084 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:20.991060 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:37:20.991167 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:20.991109 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:37:20.991167 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:20.991109 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:37:20.991301 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:20.991279 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 16:37:21.083738 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:21.083695 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7" event={"ID":"efbed66f-95b0-4fd7-af06-388840a2a364","Type":"ContainerStarted","Data":"321c60a2853e7c03d368707a76747e28e8b5576ae64eb10fab27174df047b587"} Oct 30 16:37:21.089013 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:21.088990 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bc3dce4-cd68-4d7c-b0a9-d19859ddfe10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ffa75a6-4l9c8\" (UID: \"9bc3dce4-cd68-4d7c-b0a9-d19859ddfe10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ffa75a6-4l9c8" Oct 30 16:37:21.089177 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:21.089044 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7kh6\" (UniqueName: \"kubernetes.io/projected/9bc3dce4-cd68-4d7c-b0a9-d19859ddfe10-kube-api-access-k7kh6\") pod \"maas-api-revoke-sub-9ffa75a6-4l9c8\" (UID: \"9bc3dce4-cd68-4d7c-b0a9-d19859ddfe10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ffa75a6-4l9c8" Oct 30 16:37:21.089342 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:21.089313 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bc3dce4-cd68-4d7c-b0a9-d19859ddfe10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ffa75a6-4l9c8\" (UID: \"9bc3dce4-cd68-4d7c-b0a9-d19859ddfe10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ffa75a6-4l9c8" Oct 30 16:37:21.155028 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:21.154957 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7" podStartSLOduration=4.154937002 podStartE2EDuration="4.154937002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:37:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:37:21.153038582 +0000 UTC m=+868.380213686" watchObservedRunningTime="2026-10-30 16:37:21.154937002 +0000 UTC m=+868.382112103" Oct 30 16:37:21.190317 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:21.190287 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bc3dce4-cd68-4d7c-b0a9-d19859ddfe10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ffa75a6-4l9c8\" (UID: \"9bc3dce4-cd68-4d7c-b0a9-d19859ddfe10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ffa75a6-4l9c8" Oct 30 16:37:21.190464 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:21.190392 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bc3dce4-cd68-4d7c-b0a9-d19859ddfe10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ffa75a6-4l9c8\" (UID: \"9bc3dce4-cd68-4d7c-b0a9-d19859ddfe10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ffa75a6-4l9c8" Oct 30 16:37:21.190464 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:21.190424 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7kh6\" (UniqueName: \"kubernetes.io/projected/9bc3dce4-cd68-4d7c-b0a9-d19859ddfe10-kube-api-access-k7kh6\") pod \"maas-api-revoke-sub-9ffa75a6-4l9c8\" (UID: \"9bc3dce4-cd68-4d7c-b0a9-d19859ddfe10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ffa75a6-4l9c8" Oct 30 16:37:21.191195 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:21.191163 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bc3dce4-cd68-4d7c-b0a9-d19859ddfe10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ffa75a6-4l9c8\" (UID: \"9bc3dce4-cd68-4d7c-b0a9-d19859ddfe10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ffa75a6-4l9c8" Oct 30 16:37:21.193070 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:21.193040 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bc3dce4-cd68-4d7c-b0a9-d19859ddfe10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ffa75a6-4l9c8\" (UID: \"9bc3dce4-cd68-4d7c-b0a9-d19859ddfe10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ffa75a6-4l9c8" Oct 30 16:37:21.237928 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:21.237848 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7kh6\" (UniqueName: \"kubernetes.io/projected/9bc3dce4-cd68-4d7c-b0a9-d19859ddfe10-kube-api-access-k7kh6\") pod \"maas-api-revoke-sub-9ffa75a6-4l9c8\" (UID: \"9bc3dce4-cd68-4d7c-b0a9-d19859ddfe10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ffa75a6-4l9c8" Oct 30 16:37:21.292656 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:21.292625 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ffa75a6-4l9c8" Oct 30 16:37:21.297097 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:21.297058 4493 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 16:37:21.297464 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:21.297128 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9" podUID="b08f59e7-615a-4d20-ba53-ad4bcf0d5728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 16:37:21.545281 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:21.545154 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7" Oct 30 16:37:21.546855 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:21.546832 4493 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 16:37:21.546970 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:21.546891 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7" podUID="efbed66f-95b0-4fd7-af06-388840a2a364" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 16:37:21.652465 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:21.652435 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ffa75a6-4l9c8"] Oct 30 16:37:21.655121 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:37:21.655083 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bc3dce4_cd68_4d7c_b0a9_d19859ddfe10.slice/crio-576c67a44dc0b25325b392a4ccd61d930ddbb44aa6ff3b0b91ded19de38fc51c WatchSource:0}: Error finding container 576c67a44dc0b25325b392a4ccd61d930ddbb44aa6ff3b0b91ded19de38fc51c: Status 404 returned error can't find the container with id 576c67a44dc0b25325b392a4ccd61d930ddbb44aa6ff3b0b91ded19de38fc51c Oct 30 16:37:22.096505 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:22.096413 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ffa75a6-4l9c8" event={"ID":"9bc3dce4-cd68-4d7c-b0a9-d19859ddfe10","Type":"ContainerStarted","Data":"1276b7c3e01101bdbec125da6f37173034d17fd0a5c490a2ae1f398bf81ffdcc"} Oct 30 16:37:22.096505 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:22.096483 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ffa75a6-4l9c8" event={"ID":"9bc3dce4-cd68-4d7c-b0a9-d19859ddfe10","Type":"ContainerStarted","Data":"576c67a44dc0b25325b392a4ccd61d930ddbb44aa6ff3b0b91ded19de38fc51c"} Oct 30 16:37:22.541808 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:22.541174 4493 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 16:37:22.541808 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:22.541235 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7" podUID="efbed66f-95b0-4fd7-af06-388840a2a364" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 16:37:23.124866 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:23.121181 4493 generic.go:356] "Generic (PLEG): container finished" podID="9bc3dce4-cd68-4d7c-b0a9-d19859ddfe10" containerID="1276b7c3e01101bdbec125da6f37173034d17fd0a5c490a2ae1f398bf81ffdcc" exitCode=0 Oct 30 16:37:23.124866 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:23.122471 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ffa75a6-4l9c8" event={"ID":"9bc3dce4-cd68-4d7c-b0a9-d19859ddfe10","Type":"ContainerDied","Data":"1276b7c3e01101bdbec125da6f37173034d17fd0a5c490a2ae1f398bf81ffdcc"} Oct 30 16:37:23.341812 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:23.338882 4493 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:37:23.341812 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:23.338951 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9" podUID="b08f59e7-615a-4d20-ba53-ad4bcf0d5728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:23.538806 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:23.538158 4493 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 16:37:23.538806 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:23.538220 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7" podUID="efbed66f-95b0-4fd7-af06-388840a2a364" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 16:37:23.767348 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:23.767321 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9" Oct 30 16:37:23.767708 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:23.767686 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9" Oct 30 16:37:23.768484 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:23.768467 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9" Oct 30 16:37:24.378081 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:24.378047 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ffa75a6-4l9c8" Oct 30 16:37:24.381413 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:24.381382 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ef18671-splkp"] Oct 30 16:37:24.382270 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:24.382250 4493 cpu_manager.go:414] "containerMap: removing container" podUID="9bc3dce4-cd68-4d7c-b0a9-d19859ddfe10" containerName="revoke-keys" Oct 30 16:37:24.382331 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:24.382275 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bc3dce4-cd68-4d7c-b0a9-d19859ddfe10" containerName="revoke-keys" Oct 30 16:37:24.382434 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:24.382422 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bc3dce4-cd68-4d7c-b0a9-d19859ddfe10" containerName="revoke-keys" Oct 30 16:37:24.386294 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:24.386269 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef18671-splkp" Oct 30 16:37:24.404852 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:24.403721 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ef18671-splkp"] Oct 30 16:37:24.479833 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:24.479773 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bc3dce4-cd68-4d7c-b0a9-d19859ddfe10-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bc3dce4-cd68-4d7c-b0a9-d19859ddfe10-maas-api-openshift-ca\") pod \"9bc3dce4-cd68-4d7c-b0a9-d19859ddfe10\" (UID: \"9bc3dce4-cd68-4d7c-b0a9-d19859ddfe10\") " Oct 30 16:37:24.480041 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:24.479856 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bc3dce4-cd68-4d7c-b0a9-d19859ddfe10-kube-api-access-k7kh6\" (UniqueName: \"kubernetes.io/projected/9bc3dce4-cd68-4d7c-b0a9-d19859ddfe10-kube-api-access-k7kh6\") pod \"9bc3dce4-cd68-4d7c-b0a9-d19859ddfe10\" (UID: \"9bc3dce4-cd68-4d7c-b0a9-d19859ddfe10\") " Oct 30 16:37:24.480041 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:24.479988 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9bc3dce4-cd68-4d7c-b0a9-d19859ddfe10-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bc3dce4-cd68-4d7c-b0a9-d19859ddfe10-maas-api-xks-ca\") pod \"9bc3dce4-cd68-4d7c-b0a9-d19859ddfe10\" (UID: \"9bc3dce4-cd68-4d7c-b0a9-d19859ddfe10\") " Oct 30 16:37:24.480236 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:24.480202 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bc3dce4-cd68-4d7c-b0a9-d19859ddfe10-maas-api-openshift-ca" pod "9bc3dce4-cd68-4d7c-b0a9-d19859ddfe10" (UID: "9bc3dce4-cd68-4d7c-b0a9-d19859ddfe10"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:37:24.480384 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:24.480360 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f49l\" (UniqueName: \"kubernetes.io/projected/15735aee-4782-46de-94ea-fb3db181e6cd-kube-api-access-4f49l\") pod \"maas-api-revoke-sub-5ef18671-splkp\" (UID: \"15735aee-4782-46de-94ea-fb3db181e6cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef18671-splkp" Oct 30 16:37:24.480608 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:24.480583 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15735aee-4782-46de-94ea-fb3db181e6cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ef18671-splkp\" (UID: \"15735aee-4782-46de-94ea-fb3db181e6cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef18671-splkp" Oct 30 16:37:24.480743 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:24.480627 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15735aee-4782-46de-94ea-fb3db181e6cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ef18671-splkp\" (UID: \"15735aee-4782-46de-94ea-fb3db181e6cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef18671-splkp" Oct 30 16:37:24.480900 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:24.480853 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bc3dce4-cd68-4d7c-b0a9-d19859ddfe10-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:37:24.483208 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:24.483183 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bc3dce4-cd68-4d7c-b0a9-d19859ddfe10-maas-api-xks-ca" pod "9bc3dce4-cd68-4d7c-b0a9-d19859ddfe10" (UID: "9bc3dce4-cd68-4d7c-b0a9-d19859ddfe10"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:37:24.483386 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:24.483358 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bc3dce4-cd68-4d7c-b0a9-d19859ddfe10-kube-api-access-k7kh6" pod "9bc3dce4-cd68-4d7c-b0a9-d19859ddfe10" (UID: "9bc3dce4-cd68-4d7c-b0a9-d19859ddfe10"). InnerVolumeSpecName "kube-api-access-k7kh6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:37:24.538370 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:24.538330 4493 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 16:37:24.538561 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:24.538404 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7" podUID="efbed66f-95b0-4fd7-af06-388840a2a364" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 16:37:24.582540 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:24.582493 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15735aee-4782-46de-94ea-fb3db181e6cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ef18671-splkp\" (UID: \"15735aee-4782-46de-94ea-fb3db181e6cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef18671-splkp" Oct 30 16:37:24.582734 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:24.582654 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f49l\" (UniqueName: \"kubernetes.io/projected/15735aee-4782-46de-94ea-fb3db181e6cd-kube-api-access-4f49l\") pod \"maas-api-revoke-sub-5ef18671-splkp\" (UID: \"15735aee-4782-46de-94ea-fb3db181e6cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef18671-splkp" Oct 30 16:37:24.582837 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:24.582762 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15735aee-4782-46de-94ea-fb3db181e6cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ef18671-splkp\" (UID: \"15735aee-4782-46de-94ea-fb3db181e6cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef18671-splkp" Oct 30 16:37:24.582911 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:24.582860 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7kh6\" (UniqueName: \"kubernetes.io/projected/9bc3dce4-cd68-4d7c-b0a9-d19859ddfe10-kube-api-access-k7kh6\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:37:24.582911 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:24.582879 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bc3dce4-cd68-4d7c-b0a9-d19859ddfe10-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:37:24.583637 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:24.583609 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15735aee-4782-46de-94ea-fb3db181e6cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ef18671-splkp\" (UID: \"15735aee-4782-46de-94ea-fb3db181e6cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef18671-splkp" Oct 30 16:37:24.585603 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:24.585580 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15735aee-4782-46de-94ea-fb3db181e6cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ef18671-splkp\" (UID: \"15735aee-4782-46de-94ea-fb3db181e6cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef18671-splkp" Oct 30 16:37:24.632940 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:24.632515 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f49l\" (UniqueName: \"kubernetes.io/projected/15735aee-4782-46de-94ea-fb3db181e6cd-kube-api-access-4f49l\") pod \"maas-api-revoke-sub-5ef18671-splkp\" (UID: \"15735aee-4782-46de-94ea-fb3db181e6cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef18671-splkp" Oct 30 16:37:24.702887 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:24.702812 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef18671-splkp" Oct 30 16:37:24.741469 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:24.741440 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj"] Oct 30 16:37:24.751387 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:24.751350 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj"] Oct 30 16:37:24.751709 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:24.751681 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" Oct 30 16:37:24.766955 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:24.766914 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-a77b48\"/\"distinct-w1-a77b48-kserve-self-signed-certs\"" Oct 30 16:37:24.886763 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:24.886727 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e90f43d3-f5eb-45ab-af1e-340d0ce18966-tmp-dir\") pod \"distinct-w1-a77b48-kserve-946867657-kpkbj\" (UID: \"e90f43d3-f5eb-45ab-af1e-340d0ce18966\") " pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" Oct 30 16:37:24.886932 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:24.886873 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pffkv\" (UniqueName: \"kubernetes.io/projected/e90f43d3-f5eb-45ab-af1e-340d0ce18966-kube-api-access-pffkv\") pod \"distinct-w1-a77b48-kserve-946867657-kpkbj\" (UID: \"e90f43d3-f5eb-45ab-af1e-340d0ce18966\") " pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" Oct 30 16:37:24.887009 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:24.886929 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e90f43d3-f5eb-45ab-af1e-340d0ce18966-model-cache\") pod \"distinct-w1-a77b48-kserve-946867657-kpkbj\" (UID: \"e90f43d3-f5eb-45ab-af1e-340d0ce18966\") " pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" Oct 30 16:37:24.887009 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:24.886960 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e90f43d3-f5eb-45ab-af1e-340d0ce18966-dshm\") pod \"distinct-w1-a77b48-kserve-946867657-kpkbj\" (UID: \"e90f43d3-f5eb-45ab-af1e-340d0ce18966\") " pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" Oct 30 16:37:24.887115 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:24.887068 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e90f43d3-f5eb-45ab-af1e-340d0ce18966-tls-certs\") pod \"distinct-w1-a77b48-kserve-946867657-kpkbj\" (UID: \"e90f43d3-f5eb-45ab-af1e-340d0ce18966\") " pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" Oct 30 16:37:24.887115 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:24.887100 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e90f43d3-f5eb-45ab-af1e-340d0ce18966-home\") pod \"distinct-w1-a77b48-kserve-946867657-kpkbj\" (UID: \"e90f43d3-f5eb-45ab-af1e-340d0ce18966\") " pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" Oct 30 16:37:24.992995 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:24.988362 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e90f43d3-f5eb-45ab-af1e-340d0ce18966-tls-certs\") pod \"distinct-w1-a77b48-kserve-946867657-kpkbj\" (UID: \"e90f43d3-f5eb-45ab-af1e-340d0ce18966\") " pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" Oct 30 16:37:24.992995 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:24.988412 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e90f43d3-f5eb-45ab-af1e-340d0ce18966-home\") pod \"distinct-w1-a77b48-kserve-946867657-kpkbj\" (UID: \"e90f43d3-f5eb-45ab-af1e-340d0ce18966\") " pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" Oct 30 16:37:24.992995 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:24.988501 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e90f43d3-f5eb-45ab-af1e-340d0ce18966-tmp-dir\") pod \"distinct-w1-a77b48-kserve-946867657-kpkbj\" (UID: \"e90f43d3-f5eb-45ab-af1e-340d0ce18966\") " pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" Oct 30 16:37:24.992995 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:24.988579 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pffkv\" (UniqueName: \"kubernetes.io/projected/e90f43d3-f5eb-45ab-af1e-340d0ce18966-kube-api-access-pffkv\") pod \"distinct-w1-a77b48-kserve-946867657-kpkbj\" (UID: \"e90f43d3-f5eb-45ab-af1e-340d0ce18966\") " pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" Oct 30 16:37:24.992995 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:24.988631 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e90f43d3-f5eb-45ab-af1e-340d0ce18966-model-cache\") pod \"distinct-w1-a77b48-kserve-946867657-kpkbj\" (UID: \"e90f43d3-f5eb-45ab-af1e-340d0ce18966\") " pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" Oct 30 16:37:24.992995 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:24.988660 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e90f43d3-f5eb-45ab-af1e-340d0ce18966-dshm\") pod \"distinct-w1-a77b48-kserve-946867657-kpkbj\" (UID: \"e90f43d3-f5eb-45ab-af1e-340d0ce18966\") " pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" Oct 30 16:37:24.992995 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:24.992028 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e90f43d3-f5eb-45ab-af1e-340d0ce18966-dshm\") pod \"distinct-w1-a77b48-kserve-946867657-kpkbj\" (UID: \"e90f43d3-f5eb-45ab-af1e-340d0ce18966\") " pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" Oct 30 16:37:24.992995 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:24.992591 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e90f43d3-f5eb-45ab-af1e-340d0ce18966-model-cache\") pod \"distinct-w1-a77b48-kserve-946867657-kpkbj\" (UID: \"e90f43d3-f5eb-45ab-af1e-340d0ce18966\") " pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" Oct 30 16:37:24.992995 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:24.992671 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e90f43d3-f5eb-45ab-af1e-340d0ce18966-tmp-dir\") pod \"distinct-w1-a77b48-kserve-946867657-kpkbj\" (UID: \"e90f43d3-f5eb-45ab-af1e-340d0ce18966\") " pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" Oct 30 16:37:24.992995 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:24.992860 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e90f43d3-f5eb-45ab-af1e-340d0ce18966-home\") pod \"distinct-w1-a77b48-kserve-946867657-kpkbj\" (UID: \"e90f43d3-f5eb-45ab-af1e-340d0ce18966\") " pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" Oct 30 16:37:25.059020 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:25.004178 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e90f43d3-f5eb-45ab-af1e-340d0ce18966-tls-certs\") pod \"distinct-w1-a77b48-kserve-946867657-kpkbj\" (UID: \"e90f43d3-f5eb-45ab-af1e-340d0ce18966\") " pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" Oct 30 16:37:25.170270 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:25.170007 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pffkv\" (UniqueName: \"kubernetes.io/projected/e90f43d3-f5eb-45ab-af1e-340d0ce18966-kube-api-access-pffkv\") pod \"distinct-w1-a77b48-kserve-946867657-kpkbj\" (UID: \"e90f43d3-f5eb-45ab-af1e-340d0ce18966\") " pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" Oct 30 16:37:25.170417 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:25.170392 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ef18671-splkp"] Oct 30 16:37:25.175923 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:25.175899 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ffa75a6-4l9c8" Oct 30 16:37:25.175923 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:25.175911 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ffa75a6-4l9c8" event={"ID":"9bc3dce4-cd68-4d7c-b0a9-d19859ddfe10","Type":"ContainerDied","Data":"576c67a44dc0b25325b392a4ccd61d930ddbb44aa6ff3b0b91ded19de38fc51c"} Oct 30 16:37:25.176100 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:25.175959 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="576c67a44dc0b25325b392a4ccd61d930ddbb44aa6ff3b0b91ded19de38fc51c" Oct 30 16:37:25.370339 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:25.370263 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" Oct 30 16:37:25.542817 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:25.542365 4493 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 16:37:25.542817 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:25.542424 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7" podUID="efbed66f-95b0-4fd7-af06-388840a2a364" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 16:37:25.755338 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:25.755308 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj"] Oct 30 16:37:26.184290 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:26.184250 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef18671-splkp" event={"ID":"15735aee-4782-46de-94ea-fb3db181e6cd","Type":"ContainerStarted","Data":"6a5e91035c45373e435a70623b35501767d83fdd007d727f7cd57f05f5732af9"} Oct 30 16:37:26.184472 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:26.184340 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef18671-splkp" event={"ID":"15735aee-4782-46de-94ea-fb3db181e6cd","Type":"ContainerStarted","Data":"a3da77e699e57867bc33cede4fccf7eae4996340cba97517f0160a06793e3640"} Oct 30 16:37:26.186202 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:26.186175 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" event={"ID":"e90f43d3-f5eb-45ab-af1e-340d0ce18966","Type":"ContainerStarted","Data":"2a0643d5e6dd8f438957ba320dd567863f76906a436a3d9bc75da172a37d9a88"} Oct 30 16:37:26.186320 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:26.186214 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" event={"ID":"e90f43d3-f5eb-45ab-af1e-340d0ce18966","Type":"ContainerStarted","Data":"d0ed18014cc41a718118239104d1090c21977177c4293720197a164dd67edd09"} Oct 30 16:37:26.225146 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:26.225046 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef18671-splkp" podStartSLOduration=2.225027823 podStartE2EDuration="2.225027823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:37:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:37:26.224578924 +0000 UTC m=+873.451754024" watchObservedRunningTime="2026-10-30 16:37:26.225027823 +0000 UTC m=+873.452202924" Oct 30 16:37:26.268870 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:26.268807 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" podStartSLOduration=2.268778433 podStartE2EDuration="2.268778433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:37:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:37:26.267078605 +0000 UTC m=+873.494253707" watchObservedRunningTime="2026-10-30 16:37:26.268778433 +0000 UTC m=+873.495953533" Oct 30 16:37:26.370891 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:26.370847 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" Oct 30 16:37:26.539059 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:26.538977 4493 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 16:37:26.539059 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:26.539034 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7" podUID="efbed66f-95b0-4fd7-af06-388840a2a364" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 16:37:27.195250 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:27.195212 4493 generic.go:356] "Generic (PLEG): container finished" podID="15735aee-4782-46de-94ea-fb3db181e6cd" containerID="6a5e91035c45373e435a70623b35501767d83fdd007d727f7cd57f05f5732af9" exitCode=0 Oct 30 16:37:27.195770 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:27.195303 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef18671-splkp" event={"ID":"15735aee-4782-46de-94ea-fb3db181e6cd","Type":"ContainerDied","Data":"6a5e91035c45373e435a70623b35501767d83fdd007d727f7cd57f05f5732af9"} Oct 30 16:37:27.543351 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:27.543270 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7" Oct 30 16:37:27.543775 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:27.543756 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7" Oct 30 16:37:27.544631 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:27.544601 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7" Oct 30 16:37:28.049174 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:28.049147 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br"] Oct 30 16:37:28.053760 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:28.053672 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br" Oct 30 16:37:28.081801 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:28.081757 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br"] Oct 30 16:37:28.132605 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:28.132577 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b30c2fa0-a3d3-4942-9596-bb6b245a238f-istio-podinfo\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br\" (UID: \"b30c2fa0-a3d3-4942-9596-bb6b245a238f\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br" Oct 30 16:37:28.132777 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:28.132611 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b30c2fa0-a3d3-4942-9596-bb6b245a238f-workload-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br\" (UID: \"b30c2fa0-a3d3-4942-9596-bb6b245a238f\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br" Oct 30 16:37:28.132777 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:28.132631 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b30c2fa0-a3d3-4942-9596-bb6b245a238f-istiod-ca-cert\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br\" (UID: \"b30c2fa0-a3d3-4942-9596-bb6b245a238f\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br" Oct 30 16:37:28.132881 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:28.132869 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b30c2fa0-a3d3-4942-9596-bb6b245a238f-workload-certs\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br\" (UID: \"b30c2fa0-a3d3-4942-9596-bb6b245a238f\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br" Oct 30 16:37:28.132920 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:28.132908 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b30c2fa0-a3d3-4942-9596-bb6b245a238f-istio-envoy\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br\" (UID: \"b30c2fa0-a3d3-4942-9596-bb6b245a238f\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br" Oct 30 16:37:28.132974 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:28.132941 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b30c2fa0-a3d3-4942-9596-bb6b245a238f-credential-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br\" (UID: \"b30c2fa0-a3d3-4942-9596-bb6b245a238f\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br" Oct 30 16:37:28.133009 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:28.132989 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b30c2fa0-a3d3-4942-9596-bb6b245a238f-istio-data\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br\" (UID: \"b30c2fa0-a3d3-4942-9596-bb6b245a238f\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br" Oct 30 16:37:28.133087 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:28.133075 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b30c2fa0-a3d3-4942-9596-bb6b245a238f-istio-token\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br\" (UID: \"b30c2fa0-a3d3-4942-9596-bb6b245a238f\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br" Oct 30 16:37:28.133126 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:28.133093 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t5n5\" (UniqueName: \"kubernetes.io/projected/b30c2fa0-a3d3-4942-9596-bb6b245a238f-kube-api-access-8t5n5\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br\" (UID: \"b30c2fa0-a3d3-4942-9596-bb6b245a238f\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br" Oct 30 16:37:28.235354 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:28.235312 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b30c2fa0-a3d3-4942-9596-bb6b245a238f-istio-envoy\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br\" (UID: \"b30c2fa0-a3d3-4942-9596-bb6b245a238f\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br" Oct 30 16:37:28.235850 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:28.235370 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b30c2fa0-a3d3-4942-9596-bb6b245a238f-credential-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br\" (UID: \"b30c2fa0-a3d3-4942-9596-bb6b245a238f\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br" Oct 30 16:37:28.235850 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:28.235414 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b30c2fa0-a3d3-4942-9596-bb6b245a238f-istio-data\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br\" (UID: \"b30c2fa0-a3d3-4942-9596-bb6b245a238f\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br" Oct 30 16:37:28.235850 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:28.235551 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b30c2fa0-a3d3-4942-9596-bb6b245a238f-istio-token\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br\" (UID: \"b30c2fa0-a3d3-4942-9596-bb6b245a238f\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br" Oct 30 16:37:28.235850 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:28.235577 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t5n5\" (UniqueName: \"kubernetes.io/projected/b30c2fa0-a3d3-4942-9596-bb6b245a238f-kube-api-access-8t5n5\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br\" (UID: \"b30c2fa0-a3d3-4942-9596-bb6b245a238f\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br" Oct 30 16:37:28.235850 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:28.235641 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b30c2fa0-a3d3-4942-9596-bb6b245a238f-istio-podinfo\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br\" (UID: \"b30c2fa0-a3d3-4942-9596-bb6b245a238f\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br" Oct 30 16:37:28.235850 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:28.235699 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b30c2fa0-a3d3-4942-9596-bb6b245a238f-workload-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br\" (UID: \"b30c2fa0-a3d3-4942-9596-bb6b245a238f\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br" Oct 30 16:37:28.235850 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:28.235732 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b30c2fa0-a3d3-4942-9596-bb6b245a238f-istiod-ca-cert\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br\" (UID: \"b30c2fa0-a3d3-4942-9596-bb6b245a238f\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br" Oct 30 16:37:28.236241 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:28.235947 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b30c2fa0-a3d3-4942-9596-bb6b245a238f-workload-certs\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br\" (UID: \"b30c2fa0-a3d3-4942-9596-bb6b245a238f\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br" Oct 30 16:37:28.236241 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:28.236021 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b30c2fa0-a3d3-4942-9596-bb6b245a238f-istio-data\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br\" (UID: \"b30c2fa0-a3d3-4942-9596-bb6b245a238f\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br" Oct 30 16:37:28.237390 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:28.236535 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b30c2fa0-a3d3-4942-9596-bb6b245a238f-credential-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br\" (UID: \"b30c2fa0-a3d3-4942-9596-bb6b245a238f\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br" Oct 30 16:37:28.237390 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:28.236603 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b30c2fa0-a3d3-4942-9596-bb6b245a238f-workload-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br\" (UID: \"b30c2fa0-a3d3-4942-9596-bb6b245a238f\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br" Oct 30 16:37:28.237390 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:28.236877 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b30c2fa0-a3d3-4942-9596-bb6b245a238f-workload-certs\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br\" (UID: \"b30c2fa0-a3d3-4942-9596-bb6b245a238f\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br" Oct 30 16:37:28.237390 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:28.237336 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b30c2fa0-a3d3-4942-9596-bb6b245a238f-istiod-ca-cert\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br\" (UID: \"b30c2fa0-a3d3-4942-9596-bb6b245a238f\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br" Oct 30 16:37:28.238755 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:28.238712 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b30c2fa0-a3d3-4942-9596-bb6b245a238f-istio-envoy\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br\" (UID: \"b30c2fa0-a3d3-4942-9596-bb6b245a238f\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br" Oct 30 16:37:28.238901 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:28.238760 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b30c2fa0-a3d3-4942-9596-bb6b245a238f-istio-podinfo\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br\" (UID: \"b30c2fa0-a3d3-4942-9596-bb6b245a238f\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br" Oct 30 16:37:28.260900 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:28.260855 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t5n5\" (UniqueName: \"kubernetes.io/projected/b30c2fa0-a3d3-4942-9596-bb6b245a238f-kube-api-access-8t5n5\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br\" (UID: \"b30c2fa0-a3d3-4942-9596-bb6b245a238f\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br" Oct 30 16:37:28.265682 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:28.265635 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b30c2fa0-a3d3-4942-9596-bb6b245a238f-istio-token\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br\" (UID: \"b30c2fa0-a3d3-4942-9596-bb6b245a238f\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br" Oct 30 16:37:28.348672 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:28.348641 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef18671-splkp" Oct 30 16:37:28.374383 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:28.374357 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br" Oct 30 16:37:28.444809 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:28.443849 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15735aee-4782-46de-94ea-fb3db181e6cd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15735aee-4782-46de-94ea-fb3db181e6cd-maas-api-openshift-ca\") pod \"15735aee-4782-46de-94ea-fb3db181e6cd\" (UID: \"15735aee-4782-46de-94ea-fb3db181e6cd\") " Oct 30 16:37:28.444809 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:28.443980 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15735aee-4782-46de-94ea-fb3db181e6cd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15735aee-4782-46de-94ea-fb3db181e6cd-maas-api-xks-ca\") pod \"15735aee-4782-46de-94ea-fb3db181e6cd\" (UID: \"15735aee-4782-46de-94ea-fb3db181e6cd\") " Oct 30 16:37:28.444809 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:28.444174 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15735aee-4782-46de-94ea-fb3db181e6cd-kube-api-access-4f49l\" (UniqueName: \"kubernetes.io/projected/15735aee-4782-46de-94ea-fb3db181e6cd-kube-api-access-4f49l\") pod \"15735aee-4782-46de-94ea-fb3db181e6cd\" (UID: \"15735aee-4782-46de-94ea-fb3db181e6cd\") " Oct 30 16:37:28.444809 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:28.444482 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15735aee-4782-46de-94ea-fb3db181e6cd-maas-api-openshift-ca" pod "15735aee-4782-46de-94ea-fb3db181e6cd" (UID: "15735aee-4782-46de-94ea-fb3db181e6cd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:37:28.445152 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:28.444846 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15735aee-4782-46de-94ea-fb3db181e6cd-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:37:28.451434 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:28.451401 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15735aee-4782-46de-94ea-fb3db181e6cd-maas-api-xks-ca" pod "15735aee-4782-46de-94ea-fb3db181e6cd" (UID: "15735aee-4782-46de-94ea-fb3db181e6cd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:37:28.457147 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:28.457120 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15735aee-4782-46de-94ea-fb3db181e6cd-kube-api-access-4f49l" pod "15735aee-4782-46de-94ea-fb3db181e6cd" (UID: "15735aee-4782-46de-94ea-fb3db181e6cd"). InnerVolumeSpecName "kube-api-access-4f49l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:37:28.546684 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:28.546645 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4f49l\" (UniqueName: \"kubernetes.io/projected/15735aee-4782-46de-94ea-fb3db181e6cd-kube-api-access-4f49l\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:37:28.546684 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:28.546689 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15735aee-4782-46de-94ea-fb3db181e6cd-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:37:28.713896 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:28.713867 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br"] Oct 30 16:37:28.714889 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:37:28.714860 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb30c2fa0_a3d3_4942_9596_bb6b245a238f.slice/crio-c74b4dcbb76dc9a177937c013c4f606af39c6fde1e6db5b9ced2987a107b141e WatchSource:0}: Error finding container c74b4dcbb76dc9a177937c013c4f606af39c6fde1e6db5b9ced2987a107b141e: Status 404 returned error can't find the container with id c74b4dcbb76dc9a177937c013c4f606af39c6fde1e6db5b9ced2987a107b141e Oct 30 16:37:28.718007 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:28.717965 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:37:28.718079 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:28.718053 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:37:28.718117 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:28.718086 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:37:29.213636 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:29.213593 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br" event={"ID":"b30c2fa0-a3d3-4942-9596-bb6b245a238f","Type":"ContainerStarted","Data":"75c694a4b9d0335d33b9544ceba1690e4f05a135e5cb26ce021dd775702cdaa9"} Oct 30 16:37:29.213858 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:29.213662 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br" event={"ID":"b30c2fa0-a3d3-4942-9596-bb6b245a238f","Type":"ContainerStarted","Data":"c74b4dcbb76dc9a177937c013c4f606af39c6fde1e6db5b9ced2987a107b141e"} Oct 30 16:37:29.215242 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:29.215212 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef18671-splkp" event={"ID":"15735aee-4782-46de-94ea-fb3db181e6cd","Type":"ContainerDied","Data":"a3da77e699e57867bc33cede4fccf7eae4996340cba97517f0160a06793e3640"} Oct 30 16:37:29.215374 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:29.215264 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3da77e699e57867bc33cede4fccf7eae4996340cba97517f0160a06793e3640" Oct 30 16:37:29.215374 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:29.215230 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef18671-splkp" Oct 30 16:37:29.263249 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:29.263028 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br" podStartSLOduration=1.263009413 podStartE2EDuration="1.263009413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:37:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:37:29.259731239 +0000 UTC m=+876.486906339" watchObservedRunningTime="2026-10-30 16:37:29.263009413 +0000 UTC m=+876.490184517" Oct 30 16:37:29.375272 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:29.375238 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br" Oct 30 16:37:29.375448 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:29.375303 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm"] Oct 30 16:37:29.375951 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:29.375931 4493 cpu_manager.go:414] "containerMap: removing container" podUID="15735aee-4782-46de-94ea-fb3db181e6cd" containerName="revoke-keys" Oct 30 16:37:29.375951 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:29.375951 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15735aee-4782-46de-94ea-fb3db181e6cd" containerName="revoke-keys" Oct 30 16:37:29.376108 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:29.376057 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="15735aee-4782-46de-94ea-fb3db181e6cd" containerName="revoke-keys" Oct 30 16:37:29.376855 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:29.376829 4493 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 16:37:29.376936 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:29.376878 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br" podUID="b30c2fa0-a3d3-4942-9596-bb6b245a238f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 16:37:29.379837 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:29.379820 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm" Oct 30 16:37:29.447813 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:29.447633 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm"] Oct 30 16:37:29.465448 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:29.465345 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a035fea-221e-4b14-aa1f-68bd02b90794-istio-podinfo\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm\" (UID: \"0a035fea-221e-4b14-aa1f-68bd02b90794\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm" Oct 30 16:37:29.465448 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:29.465394 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a035fea-221e-4b14-aa1f-68bd02b90794-istiod-ca-cert\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm\" (UID: \"0a035fea-221e-4b14-aa1f-68bd02b90794\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm" Oct 30 16:37:29.465448 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:29.465411 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a035fea-221e-4b14-aa1f-68bd02b90794-workload-certs\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm\" (UID: \"0a035fea-221e-4b14-aa1f-68bd02b90794\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm" Oct 30 16:37:29.465716 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:29.465543 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a035fea-221e-4b14-aa1f-68bd02b90794-istio-token\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm\" (UID: \"0a035fea-221e-4b14-aa1f-68bd02b90794\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm" Oct 30 16:37:29.465716 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:29.465577 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a035fea-221e-4b14-aa1f-68bd02b90794-workload-socket\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm\" (UID: \"0a035fea-221e-4b14-aa1f-68bd02b90794\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm" Oct 30 16:37:29.465716 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:29.465596 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a035fea-221e-4b14-aa1f-68bd02b90794-credential-socket\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm\" (UID: \"0a035fea-221e-4b14-aa1f-68bd02b90794\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm" Oct 30 16:37:29.465716 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:29.465639 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwmjs\" (UniqueName: \"kubernetes.io/projected/0a035fea-221e-4b14-aa1f-68bd02b90794-kube-api-access-jwmjs\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm\" (UID: \"0a035fea-221e-4b14-aa1f-68bd02b90794\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm" Oct 30 16:37:29.465963 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:29.465753 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a035fea-221e-4b14-aa1f-68bd02b90794-istio-envoy\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm\" (UID: \"0a035fea-221e-4b14-aa1f-68bd02b90794\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm" Oct 30 16:37:29.465963 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:29.465817 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a035fea-221e-4b14-aa1f-68bd02b90794-istio-data\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm\" (UID: \"0a035fea-221e-4b14-aa1f-68bd02b90794\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm" Oct 30 16:37:29.566680 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:29.566643 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a035fea-221e-4b14-aa1f-68bd02b90794-istio-podinfo\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm\" (UID: \"0a035fea-221e-4b14-aa1f-68bd02b90794\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm" Oct 30 16:37:29.566878 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:29.566696 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a035fea-221e-4b14-aa1f-68bd02b90794-istiod-ca-cert\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm\" (UID: \"0a035fea-221e-4b14-aa1f-68bd02b90794\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm" Oct 30 16:37:29.566878 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:29.566722 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a035fea-221e-4b14-aa1f-68bd02b90794-workload-certs\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm\" (UID: \"0a035fea-221e-4b14-aa1f-68bd02b90794\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm" Oct 30 16:37:29.566878 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:29.566770 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a035fea-221e-4b14-aa1f-68bd02b90794-istio-token\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm\" (UID: \"0a035fea-221e-4b14-aa1f-68bd02b90794\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm" Oct 30 16:37:29.566878 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:29.566821 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a035fea-221e-4b14-aa1f-68bd02b90794-workload-socket\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm\" (UID: \"0a035fea-221e-4b14-aa1f-68bd02b90794\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm" Oct 30 16:37:29.566878 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:29.566851 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a035fea-221e-4b14-aa1f-68bd02b90794-credential-socket\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm\" (UID: \"0a035fea-221e-4b14-aa1f-68bd02b90794\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm" Oct 30 16:37:29.567138 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:29.566883 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwmjs\" (UniqueName: \"kubernetes.io/projected/0a035fea-221e-4b14-aa1f-68bd02b90794-kube-api-access-jwmjs\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm\" (UID: \"0a035fea-221e-4b14-aa1f-68bd02b90794\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm" Oct 30 16:37:29.567138 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:29.566995 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a035fea-221e-4b14-aa1f-68bd02b90794-istio-envoy\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm\" (UID: \"0a035fea-221e-4b14-aa1f-68bd02b90794\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm" Oct 30 16:37:29.567138 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:29.567063 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a035fea-221e-4b14-aa1f-68bd02b90794-istio-data\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm\" (UID: \"0a035fea-221e-4b14-aa1f-68bd02b90794\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm" Oct 30 16:37:29.567300 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:29.567194 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a035fea-221e-4b14-aa1f-68bd02b90794-workload-certs\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm\" (UID: \"0a035fea-221e-4b14-aa1f-68bd02b90794\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm" Oct 30 16:37:29.567355 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:29.567303 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a035fea-221e-4b14-aa1f-68bd02b90794-workload-socket\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm\" (UID: \"0a035fea-221e-4b14-aa1f-68bd02b90794\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm" Oct 30 16:37:29.567415 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:29.567378 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a035fea-221e-4b14-aa1f-68bd02b90794-credential-socket\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm\" (UID: \"0a035fea-221e-4b14-aa1f-68bd02b90794\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm" Oct 30 16:37:29.567519 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:29.567498 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a035fea-221e-4b14-aa1f-68bd02b90794-istio-data\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm\" (UID: \"0a035fea-221e-4b14-aa1f-68bd02b90794\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm" Oct 30 16:37:29.567710 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:29.567692 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a035fea-221e-4b14-aa1f-68bd02b90794-istiod-ca-cert\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm\" (UID: \"0a035fea-221e-4b14-aa1f-68bd02b90794\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm" Oct 30 16:37:29.569300 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:29.569280 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a035fea-221e-4b14-aa1f-68bd02b90794-istio-envoy\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm\" (UID: \"0a035fea-221e-4b14-aa1f-68bd02b90794\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm" Oct 30 16:37:29.569578 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:29.569559 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a035fea-221e-4b14-aa1f-68bd02b90794-istio-podinfo\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm\" (UID: \"0a035fea-221e-4b14-aa1f-68bd02b90794\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm" Oct 30 16:37:29.590355 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:29.590326 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a035fea-221e-4b14-aa1f-68bd02b90794-istio-token\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm\" (UID: \"0a035fea-221e-4b14-aa1f-68bd02b90794\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm" Oct 30 16:37:29.590690 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:29.590668 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwmjs\" (UniqueName: \"kubernetes.io/projected/0a035fea-221e-4b14-aa1f-68bd02b90794-kube-api-access-jwmjs\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm\" (UID: \"0a035fea-221e-4b14-aa1f-68bd02b90794\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm" Oct 30 16:37:29.703812 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:29.703750 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm" Oct 30 16:37:30.060909 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:30.060882 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm"] Oct 30 16:37:30.062053 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:37:30.062024 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a035fea_221e_4b14_aa1f_68bd02b90794.slice/crio-d2bf16d8615b56b93df0af0a6a725ac4aae8eb58a3baf7cab37b4be30fe2c80d WatchSource:0}: Error finding container d2bf16d8615b56b93df0af0a6a725ac4aae8eb58a3baf7cab37b4be30fe2c80d: Status 404 returned error can't find the container with id d2bf16d8615b56b93df0af0a6a725ac4aae8eb58a3baf7cab37b4be30fe2c80d Oct 30 16:37:30.064476 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:30.064445 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:37:30.064543 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:30.064508 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:37:30.064587 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:30.064540 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:37:30.253925 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:30.253876 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm" event={"ID":"0a035fea-221e-4b14-aa1f-68bd02b90794","Type":"ContainerStarted","Data":"cccb47c9e56127d3f2ca388ef3894f129f5e89c739da639105f252f4fc631778"} Oct 30 16:37:30.254141 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:30.253940 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm" event={"ID":"0a035fea-221e-4b14-aa1f-68bd02b90794","Type":"ContainerStarted","Data":"d2bf16d8615b56b93df0af0a6a725ac4aae8eb58a3baf7cab37b4be30fe2c80d"} Oct 30 16:37:30.321024 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:30.319569 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm" podStartSLOduration=1.319549856 podStartE2EDuration="1.319549856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:37:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:37:30.300834305 +0000 UTC m=+877.528009409" watchObservedRunningTime="2026-10-30 16:37:30.319549856 +0000 UTC m=+877.546724957" Oct 30 16:37:30.375100 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:30.375071 4493 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 16:37:30.375243 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:30.375119 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br" podUID="b30c2fa0-a3d3-4942-9596-bb6b245a238f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 16:37:30.376904 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:37:30.376877 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 16:37:30.707833 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:30.705384 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm" Oct 30 16:37:31.376838 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:31.376179 4493 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 16:37:31.376838 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:31.376238 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br" podUID="b30c2fa0-a3d3-4942-9596-bb6b245a238f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 16:37:31.432753 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:31.432700 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a7428e5-9zqd4"] Oct 30 16:37:31.477370 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:31.477166 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a7428e5-9zqd4"] Oct 30 16:37:31.477703 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:31.477323 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a7428e5-9zqd4" Oct 30 16:37:31.496187 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:31.495771 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/052926f9-c4c0-4540-8927-64a40bb4ed40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a7428e5-9zqd4\" (UID: \"052926f9-c4c0-4540-8927-64a40bb4ed40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a7428e5-9zqd4" Oct 30 16:37:31.496462 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:31.496439 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/052926f9-c4c0-4540-8927-64a40bb4ed40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a7428e5-9zqd4\" (UID: \"052926f9-c4c0-4540-8927-64a40bb4ed40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a7428e5-9zqd4" Oct 30 16:37:31.496597 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:31.496579 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 16:37:31.496773 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:31.496754 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:37:31.496900 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:31.496851 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:37:31.496967 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:31.496947 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:37:31.497057 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:31.497040 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq6dv\" (UniqueName: \"kubernetes.io/projected/052926f9-c4c0-4540-8927-64a40bb4ed40-kube-api-access-kq6dv\") pod \"maas-api-revoke-sub-8a7428e5-9zqd4\" (UID: \"052926f9-c4c0-4540-8927-64a40bb4ed40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a7428e5-9zqd4" Oct 30 16:37:31.640860 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:31.620932 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq6dv\" (UniqueName: \"kubernetes.io/projected/052926f9-c4c0-4540-8927-64a40bb4ed40-kube-api-access-kq6dv\") pod \"maas-api-revoke-sub-8a7428e5-9zqd4\" (UID: \"052926f9-c4c0-4540-8927-64a40bb4ed40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a7428e5-9zqd4" Oct 30 16:37:31.640860 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:31.621066 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/052926f9-c4c0-4540-8927-64a40bb4ed40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a7428e5-9zqd4\" (UID: \"052926f9-c4c0-4540-8927-64a40bb4ed40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a7428e5-9zqd4" Oct 30 16:37:31.640860 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:31.621197 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/052926f9-c4c0-4540-8927-64a40bb4ed40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a7428e5-9zqd4\" (UID: \"052926f9-c4c0-4540-8927-64a40bb4ed40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a7428e5-9zqd4" Oct 30 16:37:31.640860 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:31.631562 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/052926f9-c4c0-4540-8927-64a40bb4ed40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a7428e5-9zqd4\" (UID: \"052926f9-c4c0-4540-8927-64a40bb4ed40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a7428e5-9zqd4" Oct 30 16:37:31.691153 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:31.683298 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/052926f9-c4c0-4540-8927-64a40bb4ed40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a7428e5-9zqd4\" (UID: \"052926f9-c4c0-4540-8927-64a40bb4ed40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a7428e5-9zqd4" Oct 30 16:37:31.692053 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:31.692021 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq6dv\" (UniqueName: \"kubernetes.io/projected/052926f9-c4c0-4540-8927-64a40bb4ed40-kube-api-access-kq6dv\") pod \"maas-api-revoke-sub-8a7428e5-9zqd4\" (UID: \"052926f9-c4c0-4540-8927-64a40bb4ed40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a7428e5-9zqd4" Oct 30 16:37:31.757810 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:31.727478 4493 patch_prober.go:29] interesting pod/e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 16:37:31.757810 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:31.727560 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm" podUID="0a035fea-221e-4b14-aa1f-68bd02b90794" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 16:37:31.864809 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:31.863021 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a7428e5-9zqd4" Oct 30 16:37:32.463469 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:32.463238 4493 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 16:37:32.463469 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:32.463293 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br" podUID="b30c2fa0-a3d3-4942-9596-bb6b245a238f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 16:37:32.630866 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:32.626855 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" podUID="80b9bd9a-30d6-45ff-b318-26fcdc608e5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 16:37:32.785812 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:32.783323 4493 patch_prober.go:29] interesting pod/e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 16:37:32.785812 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:32.783383 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm" podUID="0a035fea-221e-4b14-aa1f-68bd02b90794" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 16:37:32.859829 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:32.854411 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a7428e5-9zqd4"] Oct 30 16:37:32.922506 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:32.919873 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" podUID="08baad08-cd72-4305-b92d-c11a84ef276f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 16:37:32.922506 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:32.920516 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" podUID="e76172b1-659d-45bd-91c6-1a0ce4062335" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 16:37:33.060260 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:33.059880 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" podUID="0eadc5d4-98c3-4bf2-b3c2-f73fe249182f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:33.388890 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:33.388161 4493 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 16:37:33.394819 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:33.388243 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br" podUID="b30c2fa0-a3d3-4942-9596-bb6b245a238f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 16:37:33.565869 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:33.562948 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" podUID="e43738f3-d55f-4c0a-a364-bf2ac76605c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:33.639828 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:33.638105 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" podUID="80b9bd9a-30d6-45ff-b318-26fcdc608e5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 16:37:33.639828 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:33.638853 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" Oct 30 16:37:33.658819 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:33.658154 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" podUID="b5abe703-fee0-4185-aa22-88d1db06392b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:33.675806 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:33.674624 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a7428e5-9zqd4" event={"ID":"052926f9-c4c0-4540-8927-64a40bb4ed40","Type":"ContainerStarted","Data":"ccd3c80420e82492596790378c2d3d2ff86cbcddb35579c58b826841fa0b19bf"} Oct 30 16:37:33.675806 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:33.674692 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a7428e5-9zqd4" event={"ID":"052926f9-c4c0-4540-8927-64a40bb4ed40","Type":"ContainerStarted","Data":"25365b53102083a2b5a6f60b0d75e615dada4a8e4933ee01801f21f17b651ccf"} Oct 30 16:37:33.709820 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:33.706269 4493 patch_prober.go:29] interesting pod/e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 16:37:33.709820 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:33.706337 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm" podUID="0a035fea-221e-4b14-aa1f-68bd02b90794" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 16:37:33.770068 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:33.770014 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a7428e5-9zqd4" podStartSLOduration=2.769991459 podStartE2EDuration="2.769991459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:37:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:37:33.733690497 +0000 UTC m=+880.960865599" watchObservedRunningTime="2026-10-30 16:37:33.769991459 +0000 UTC m=+880.997166560" Oct 30 16:37:33.839955 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:33.801133 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" podUID="14805045-a727-41ba-8bd0-100d41f9f27d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:33.943817 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:33.935240 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" podUID="e76172b1-659d-45bd-91c6-1a0ce4062335" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 16:37:33.943817 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:33.935601 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" podUID="08baad08-cd72-4305-b92d-c11a84ef276f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 16:37:33.943817 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:33.936201 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" Oct 30 16:37:33.943817 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:33.936253 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" Oct 30 16:37:34.061063 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:34.061020 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" podUID="0eadc5d4-98c3-4bf2-b3c2-f73fe249182f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:34.069067 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:34.069038 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" Oct 30 16:37:34.393810 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:34.383978 4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" podUID="14805045-a727-41ba-8bd0-100d41f9f27d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:34.492880 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:34.486510 4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" podUID="b5abe703-fee0-4185-aa22-88d1db06392b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:34.493391 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:34.493265 4493 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 16:37:34.493391 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:34.493319 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br" podUID="b30c2fa0-a3d3-4942-9596-bb6b245a238f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 16:37:34.627838 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:34.572412 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" podUID="e43738f3-d55f-4c0a-a364-bf2ac76605c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:34.627838 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:34.589134 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" Oct 30 16:37:34.650198 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:34.648900 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" podUID="80b9bd9a-30d6-45ff-b318-26fcdc608e5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:34.730292 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:34.730246 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" podUID="b5abe703-fee0-4185-aa22-88d1db06392b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Oct 30 16:37:34.738261 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:34.738220 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" Oct 30 16:37:34.761953 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:34.761480 4493 patch_prober.go:29] interesting pod/e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 16:37:34.761953 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:34.761529 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm" podUID="0a035fea-221e-4b14-aa1f-68bd02b90794" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 16:37:34.805831 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:34.804266 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" podUID="14805045-a727-41ba-8bd0-100d41f9f27d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:34.805831 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:34.805102 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" Oct 30 16:37:34.910997 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:34.910950 4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" podUID="e43738f3-d55f-4c0a-a364-bf2ac76605c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:34.983621 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:34.947581 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" podUID="e76172b1-659d-45bd-91c6-1a0ce4062335" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:34.983621 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:34.947978 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" podUID="08baad08-cd72-4305-b92d-c11a84ef276f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 16:37:34.983621 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:34.948202 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" podUID="18b72648-f523-405e-b5cd-1266ff13f8f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:35.127193 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:35.071586 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" podUID="0eadc5d4-98c3-4bf2-b3c2-f73fe249182f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:35.390914 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:35.381069 4493 generic.go:356] "Generic (PLEG): container finished" podID="052926f9-c4c0-4540-8927-64a40bb4ed40" containerID="ccd3c80420e82492596790378c2d3d2ff86cbcddb35579c58b826841fa0b19bf" exitCode=0 Oct 30 16:37:35.458052 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:35.458017 4493 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 16:37:35.458354 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:35.458321 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br" podUID="b30c2fa0-a3d3-4942-9596-bb6b245a238f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 16:37:35.478808 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:35.460064 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a7428e5-9zqd4" event={"ID":"052926f9-c4c0-4540-8927-64a40bb4ed40","Type":"ContainerDied","Data":"ccd3c80420e82492596790378c2d3d2ff86cbcddb35579c58b826841fa0b19bf"} Oct 30 16:37:35.478808 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:35.460224 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" Oct 30 16:37:35.567810 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:35.566110 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" Oct 30 16:37:35.583971 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:35.583845 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" podUID="e43738f3-d55f-4c0a-a364-bf2ac76605c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:35.613243 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:35.613217 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" Oct 30 16:37:35.614871 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:35.614851 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" Oct 30 16:37:35.616818 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:35.616024 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" Oct 30 16:37:35.658059 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:35.657936 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" podUID="80b9bd9a-30d6-45ff-b318-26fcdc608e5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 16:37:35.702203 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:35.702178 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" Oct 30 16:37:35.708902 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:35.708870 4493 patch_prober.go:29] interesting pod/e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 16:37:35.709380 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:35.709355 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm" podUID="0a035fea-221e-4b14-aa1f-68bd02b90794" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 16:37:35.726929 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:35.709297 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" Oct 30 16:37:35.794655 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:35.794625 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" Oct 30 16:37:35.818810 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:35.812961 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" Oct 30 16:37:36.392227 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:36.383884 4493 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 16:37:36.392227 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:36.383943 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br" podUID="b30c2fa0-a3d3-4942-9596-bb6b245a238f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 16:37:36.493467 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:36.493439 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" Oct 30 16:37:36.707821 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:36.707728 4493 patch_prober.go:29] interesting pod/e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 16:37:36.767029 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:36.766994 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm" podUID="0a035fea-221e-4b14-aa1f-68bd02b90794" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 16:37:36.881824 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:36.880492 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a7428e5-9zqd4" Oct 30 16:37:36.971964 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:36.971877 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/052926f9-c4c0-4540-8927-64a40bb4ed40-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/052926f9-c4c0-4540-8927-64a40bb4ed40-maas-api-xks-ca\") pod \"052926f9-c4c0-4540-8927-64a40bb4ed40\" (UID: \"052926f9-c4c0-4540-8927-64a40bb4ed40\") " Oct 30 16:37:36.972137 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:36.971992 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/052926f9-c4c0-4540-8927-64a40bb4ed40-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/052926f9-c4c0-4540-8927-64a40bb4ed40-maas-api-openshift-ca\") pod \"052926f9-c4c0-4540-8927-64a40bb4ed40\" (UID: \"052926f9-c4c0-4540-8927-64a40bb4ed40\") " Oct 30 16:37:36.972137 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:36.972092 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/052926f9-c4c0-4540-8927-64a40bb4ed40-kube-api-access-kq6dv\" (UniqueName: \"kubernetes.io/projected/052926f9-c4c0-4540-8927-64a40bb4ed40-kube-api-access-kq6dv\") pod \"052926f9-c4c0-4540-8927-64a40bb4ed40\" (UID: \"052926f9-c4c0-4540-8927-64a40bb4ed40\") " Oct 30 16:37:36.972760 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:36.972690 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/052926f9-c4c0-4540-8927-64a40bb4ed40-maas-api-openshift-ca" pod "052926f9-c4c0-4540-8927-64a40bb4ed40" (UID: "052926f9-c4c0-4540-8927-64a40bb4ed40"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:37:36.974573 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:36.974533 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/052926f9-c4c0-4540-8927-64a40bb4ed40-maas-api-xks-ca" pod "052926f9-c4c0-4540-8927-64a40bb4ed40" (UID: "052926f9-c4c0-4540-8927-64a40bb4ed40"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:37:36.975777 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:36.975756 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/052926f9-c4c0-4540-8927-64a40bb4ed40-kube-api-access-kq6dv" pod "052926f9-c4c0-4540-8927-64a40bb4ed40" (UID: "052926f9-c4c0-4540-8927-64a40bb4ed40"). InnerVolumeSpecName "kube-api-access-kq6dv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:37:37.074405 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:37.074346 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kq6dv\" (UniqueName: \"kubernetes.io/projected/052926f9-c4c0-4540-8927-64a40bb4ed40-kube-api-access-kq6dv\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:37:37.074405 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:37.074395 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/052926f9-c4c0-4540-8927-64a40bb4ed40-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:37:37.074627 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:37.074412 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/052926f9-c4c0-4540-8927-64a40bb4ed40-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:37:37.375224 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:37.375143 4493 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 16:37:37.375457 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:37.375428 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br" podUID="b30c2fa0-a3d3-4942-9596-bb6b245a238f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 16:37:37.445846 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:37.445755 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a7428e5-9zqd4" Oct 30 16:37:37.448277 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:37.448234 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a7428e5-9zqd4" event={"ID":"052926f9-c4c0-4540-8927-64a40bb4ed40","Type":"ContainerDied","Data":"25365b53102083a2b5a6f60b0d75e615dada4a8e4933ee01801f21f17b651ccf"} Oct 30 16:37:37.448464 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:37.448449 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25365b53102083a2b5a6f60b0d75e615dada4a8e4933ee01801f21f17b651ccf" Oct 30 16:37:37.719810 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:37.709246 4493 patch_prober.go:29] interesting pod/e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 16:37:37.719810 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:37.709304 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm" podUID="0a035fea-221e-4b14-aa1f-68bd02b90794" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 16:37:38.334856 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:38.334826 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-600903dc-6wp5m"] Oct 30 16:37:38.336101 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:38.336077 4493 cpu_manager.go:414] "containerMap: removing container" podUID="052926f9-c4c0-4540-8927-64a40bb4ed40" containerName="revoke-keys" Oct 30 16:37:38.336284 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:38.336266 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="052926f9-c4c0-4540-8927-64a40bb4ed40" containerName="revoke-keys" Oct 30 16:37:38.336544 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:38.336528 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="052926f9-c4c0-4540-8927-64a40bb4ed40" containerName="revoke-keys" Oct 30 16:37:38.340642 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:38.340624 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-600903dc-6wp5m" Oct 30 16:37:38.365067 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:38.365014 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-600903dc-6wp5m"] Oct 30 16:37:38.388742 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:38.386616 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:37:38.388742 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:38.386727 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:37:38.388742 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:38.386898 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:37:38.388742 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:38.387003 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 16:37:38.413822 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:38.405719 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l92jv\" (UniqueName: \"kubernetes.io/projected/51f92005-2411-4550-828b-667d303b557f-kube-api-access-l92jv\") pod \"maas-api-revoke-sub-600903dc-6wp5m\" (UID: \"51f92005-2411-4550-828b-667d303b557f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-600903dc-6wp5m" Oct 30 16:37:38.413822 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:38.405983 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51f92005-2411-4550-828b-667d303b557f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-600903dc-6wp5m\" (UID: \"51f92005-2411-4550-828b-667d303b557f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-600903dc-6wp5m" Oct 30 16:37:38.413822 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:38.406032 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51f92005-2411-4550-828b-667d303b557f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-600903dc-6wp5m\" (UID: \"51f92005-2411-4550-828b-667d303b557f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-600903dc-6wp5m" Oct 30 16:37:38.500815 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:38.498080 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br" Oct 30 16:37:38.501507 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:38.501484 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br" Oct 30 16:37:38.503176 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:38.503143 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br" Oct 30 16:37:38.516065 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:38.516035 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51f92005-2411-4550-828b-667d303b557f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-600903dc-6wp5m\" (UID: \"51f92005-2411-4550-828b-667d303b557f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-600903dc-6wp5m" Oct 30 16:37:38.516306 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:38.516272 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51f92005-2411-4550-828b-667d303b557f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-600903dc-6wp5m\" (UID: \"51f92005-2411-4550-828b-667d303b557f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-600903dc-6wp5m" Oct 30 16:37:38.516539 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:38.516521 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l92jv\" (UniqueName: \"kubernetes.io/projected/51f92005-2411-4550-828b-667d303b557f-kube-api-access-l92jv\") pod \"maas-api-revoke-sub-600903dc-6wp5m\" (UID: \"51f92005-2411-4550-828b-667d303b557f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-600903dc-6wp5m" Oct 30 16:37:38.523825 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:38.520900 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51f92005-2411-4550-828b-667d303b557f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-600903dc-6wp5m\" (UID: \"51f92005-2411-4550-828b-667d303b557f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-600903dc-6wp5m" Oct 30 16:37:38.532040 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:38.532016 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51f92005-2411-4550-828b-667d303b557f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-600903dc-6wp5m\" (UID: \"51f92005-2411-4550-828b-667d303b557f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-600903dc-6wp5m" Oct 30 16:37:38.541161 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:38.541136 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l92jv\" (UniqueName: \"kubernetes.io/projected/51f92005-2411-4550-828b-667d303b557f-kube-api-access-l92jv\") pod \"maas-api-revoke-sub-600903dc-6wp5m\" (UID: \"51f92005-2411-4550-828b-667d303b557f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-600903dc-6wp5m" Oct 30 16:37:38.700110 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:38.700075 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-600903dc-6wp5m" Oct 30 16:37:38.704577 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:38.704550 4493 patch_prober.go:29] interesting pod/e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 16:37:38.704722 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:38.704604 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm" podUID="0a035fea-221e-4b14-aa1f-68bd02b90794" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 16:37:39.024834 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:39.024626 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-600903dc-6wp5m"] Oct 30 16:37:39.488915 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:39.488881 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-600903dc-6wp5m" event={"ID":"51f92005-2411-4550-828b-667d303b557f","Type":"ContainerStarted","Data":"ead2ff9af8f361dc0b90a380dab78fa126dd350c733d7f422ea41e180364b49d"} Oct 30 16:37:39.489128 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:39.488965 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-600903dc-6wp5m" event={"ID":"51f92005-2411-4550-828b-667d303b557f","Type":"ContainerStarted","Data":"59035ab80bb33af14240c5299d81190331c609f818c29838290ecb7eab6bc87a"} Oct 30 16:37:39.534495 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:39.534437 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-600903dc-6wp5m" podStartSLOduration=1.534418485 podStartE2EDuration="1.534418485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:37:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:37:39.53247268 +0000 UTC m=+886.759647780" watchObservedRunningTime="2026-10-30 16:37:39.534418485 +0000 UTC m=+886.761593587" Oct 30 16:37:40.491221 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:40.491191 4493 generic.go:356] "Generic (PLEG): container finished" podID="51f92005-2411-4550-828b-667d303b557f" containerID="ead2ff9af8f361dc0b90a380dab78fa126dd350c733d7f422ea41e180364b49d" exitCode=0 Oct 30 16:37:40.491627 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:40.491274 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-600903dc-6wp5m" event={"ID":"51f92005-2411-4550-828b-667d303b557f","Type":"ContainerDied","Data":"ead2ff9af8f361dc0b90a380dab78fa126dd350c733d7f422ea41e180364b49d"} Oct 30 16:37:40.745020 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:40.744917 4493 patch_prober.go:29] interesting pod/e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:37:40.745020 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:40.744983 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm" podUID="0a035fea-221e-4b14-aa1f-68bd02b90794" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:40.769162 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:40.768885 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 16:37:40.964303 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:40.964264 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm" Oct 30 16:37:40.964622 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:40.964602 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm" Oct 30 16:37:40.965536 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:40.965517 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm" Oct 30 16:37:41.873364 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:41.873330 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-600903dc-6wp5m" Oct 30 16:37:42.010824 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:42.009182 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51f92005-2411-4550-828b-667d303b557f-kube-api-access-l92jv\" (UniqueName: \"kubernetes.io/projected/51f92005-2411-4550-828b-667d303b557f-kube-api-access-l92jv\") pod \"51f92005-2411-4550-828b-667d303b557f\" (UID: \"51f92005-2411-4550-828b-667d303b557f\") " Oct 30 16:37:42.085532 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:42.084331 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51f92005-2411-4550-828b-667d303b557f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51f92005-2411-4550-828b-667d303b557f-maas-api-xks-ca\") pod \"51f92005-2411-4550-828b-667d303b557f\" (UID: \"51f92005-2411-4550-828b-667d303b557f\") " Oct 30 16:37:42.085532 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:42.084420 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51f92005-2411-4550-828b-667d303b557f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51f92005-2411-4550-828b-667d303b557f-maas-api-openshift-ca\") pod \"51f92005-2411-4550-828b-667d303b557f\" (UID: \"51f92005-2411-4550-828b-667d303b557f\") " Oct 30 16:37:42.085532 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:42.085469 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51f92005-2411-4550-828b-667d303b557f-maas-api-openshift-ca" pod "51f92005-2411-4550-828b-667d303b557f" (UID: "51f92005-2411-4550-828b-667d303b557f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:37:42.130504 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:42.130446 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51f92005-2411-4550-828b-667d303b557f-maas-api-xks-ca" pod "51f92005-2411-4550-828b-667d303b557f" (UID: "51f92005-2411-4550-828b-667d303b557f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:37:42.134891 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:42.134853 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51f92005-2411-4550-828b-667d303b557f-kube-api-access-l92jv" pod "51f92005-2411-4550-828b-667d303b557f" (UID: "51f92005-2411-4550-828b-667d303b557f"). InnerVolumeSpecName "kube-api-access-l92jv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:37:42.189814 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:42.189756 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51f92005-2411-4550-828b-667d303b557f-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:37:42.190089 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:42.190066 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51f92005-2411-4550-828b-667d303b557f-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:37:42.190203 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:42.190188 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l92jv\" (UniqueName: \"kubernetes.io/projected/51f92005-2411-4550-828b-667d303b557f-kube-api-access-l92jv\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:37:42.568041 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:42.568006 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-600903dc-6wp5m" event={"ID":"51f92005-2411-4550-828b-667d303b557f","Type":"ContainerDied","Data":"59035ab80bb33af14240c5299d81190331c609f818c29838290ecb7eab6bc87a"} Oct 30 16:37:42.568313 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:42.568291 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59035ab80bb33af14240c5299d81190331c609f818c29838290ecb7eab6bc87a" Oct 30 16:37:42.568570 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:42.568547 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-600903dc-6wp5m" Oct 30 16:37:43.568810 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:43.568610 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" podUID="b5abe703-fee0-4185-aa22-88d1db06392b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:43.598686 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:43.598649 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" podUID="e43738f3-d55f-4c0a-a364-bf2ac76605c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 16:37:44.722433 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:44.722405 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a3be62e-ktqk2"] Oct 30 16:37:44.722966 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:44.722952 4493 cpu_manager.go:414] "containerMap: removing container" podUID="51f92005-2411-4550-828b-667d303b557f" containerName="revoke-keys" Oct 30 16:37:44.722966 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:44.722967 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51f92005-2411-4550-828b-667d303b557f" containerName="revoke-keys" Oct 30 16:37:44.723079 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:44.723070 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="51f92005-2411-4550-828b-667d303b557f" containerName="revoke-keys" Oct 30 16:37:44.726832 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:44.726817 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a3be62e-ktqk2" Oct 30 16:37:44.756534 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:44.756502 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:37:44.756742 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:44.756614 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 16:37:44.756742 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:44.756642 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:37:44.757252 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:44.757230 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:37:44.767026 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:44.766985 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a3be62e-ktqk2"] Oct 30 16:37:44.786464 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:44.786433 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea6938c8-3f32-4aa9-8a26-660f2aa4e20f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a3be62e-ktqk2\" (UID: \"ea6938c8-3f32-4aa9-8a26-660f2aa4e20f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a3be62e-ktqk2" Oct 30 16:37:44.786607 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:44.786563 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxvk6\" (UniqueName: \"kubernetes.io/projected/ea6938c8-3f32-4aa9-8a26-660f2aa4e20f-kube-api-access-nxvk6\") pod \"maas-api-revoke-sub-7a3be62e-ktqk2\" (UID: \"ea6938c8-3f32-4aa9-8a26-660f2aa4e20f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a3be62e-ktqk2" Oct 30 16:37:44.786673 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:44.786617 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea6938c8-3f32-4aa9-8a26-660f2aa4e20f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a3be62e-ktqk2\" (UID: \"ea6938c8-3f32-4aa9-8a26-660f2aa4e20f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a3be62e-ktqk2" Oct 30 16:37:44.887391 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:44.887359 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxvk6\" (UniqueName: \"kubernetes.io/projected/ea6938c8-3f32-4aa9-8a26-660f2aa4e20f-kube-api-access-nxvk6\") pod \"maas-api-revoke-sub-7a3be62e-ktqk2\" (UID: \"ea6938c8-3f32-4aa9-8a26-660f2aa4e20f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a3be62e-ktqk2" Oct 30 16:37:44.887557 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:44.887430 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea6938c8-3f32-4aa9-8a26-660f2aa4e20f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a3be62e-ktqk2\" (UID: \"ea6938c8-3f32-4aa9-8a26-660f2aa4e20f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a3be62e-ktqk2" Oct 30 16:37:44.887557 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:44.887511 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea6938c8-3f32-4aa9-8a26-660f2aa4e20f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a3be62e-ktqk2\" (UID: \"ea6938c8-3f32-4aa9-8a26-660f2aa4e20f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a3be62e-ktqk2" Oct 30 16:37:44.888264 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:44.888240 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea6938c8-3f32-4aa9-8a26-660f2aa4e20f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a3be62e-ktqk2\" (UID: \"ea6938c8-3f32-4aa9-8a26-660f2aa4e20f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a3be62e-ktqk2" Oct 30 16:37:44.890020 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:44.890000 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea6938c8-3f32-4aa9-8a26-660f2aa4e20f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a3be62e-ktqk2\" (UID: \"ea6938c8-3f32-4aa9-8a26-660f2aa4e20f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a3be62e-ktqk2" Oct 30 16:37:44.911089 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:44.911064 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxvk6\" (UniqueName: \"kubernetes.io/projected/ea6938c8-3f32-4aa9-8a26-660f2aa4e20f-kube-api-access-nxvk6\") pod \"maas-api-revoke-sub-7a3be62e-ktqk2\" (UID: \"ea6938c8-3f32-4aa9-8a26-660f2aa4e20f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a3be62e-ktqk2" Oct 30 16:37:45.040613 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:45.040520 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a3be62e-ktqk2" Oct 30 16:37:45.353963 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:45.353925 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a3be62e-ktqk2"] Oct 30 16:37:45.354184 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:37:45.354164 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea6938c8_3f32_4aa9_8a26_660f2aa4e20f.slice/crio-7b0d2963d6b3b0f1d5d6d09ee761f5e54620a462b616eaa8e7fde6f24dd57933 WatchSource:0}: Error finding container 7b0d2963d6b3b0f1d5d6d09ee761f5e54620a462b616eaa8e7fde6f24dd57933: Status 404 returned error can't find the container with id 7b0d2963d6b3b0f1d5d6d09ee761f5e54620a462b616eaa8e7fde6f24dd57933 Oct 30 16:37:45.659505 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:45.659473 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a3be62e-ktqk2" event={"ID":"ea6938c8-3f32-4aa9-8a26-660f2aa4e20f","Type":"ContainerStarted","Data":"7b0d2963d6b3b0f1d5d6d09ee761f5e54620a462b616eaa8e7fde6f24dd57933"} Oct 30 16:37:46.668810 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:46.668748 4493 generic.go:356] "Generic (PLEG): container finished" podID="ea6938c8-3f32-4aa9-8a26-660f2aa4e20f" containerID="2b8e8d16c924705aacaaf346634f0763b46c19f06e000e09852709a14d294e79" exitCode=0 Oct 30 16:37:46.669205 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:46.668837 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a3be62e-ktqk2" event={"ID":"ea6938c8-3f32-4aa9-8a26-660f2aa4e20f","Type":"ContainerDied","Data":"2b8e8d16c924705aacaaf346634f0763b46c19f06e000e09852709a14d294e79"} Oct 30 16:37:47.821378 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:47.821349 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a3be62e-ktqk2" Oct 30 16:37:47.927574 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:47.927505 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea6938c8-3f32-4aa9-8a26-660f2aa4e20f-kube-api-access-nxvk6\" (UniqueName: \"kubernetes.io/projected/ea6938c8-3f32-4aa9-8a26-660f2aa4e20f-kube-api-access-nxvk6\") pod \"ea6938c8-3f32-4aa9-8a26-660f2aa4e20f\" (UID: \"ea6938c8-3f32-4aa9-8a26-660f2aa4e20f\") " Oct 30 16:37:47.927713 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:47.927692 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea6938c8-3f32-4aa9-8a26-660f2aa4e20f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea6938c8-3f32-4aa9-8a26-660f2aa4e20f-maas-api-openshift-ca\") pod \"ea6938c8-3f32-4aa9-8a26-660f2aa4e20f\" (UID: \"ea6938c8-3f32-4aa9-8a26-660f2aa4e20f\") " Oct 30 16:37:47.927763 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:47.927752 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea6938c8-3f32-4aa9-8a26-660f2aa4e20f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea6938c8-3f32-4aa9-8a26-660f2aa4e20f-maas-api-xks-ca\") pod \"ea6938c8-3f32-4aa9-8a26-660f2aa4e20f\" (UID: \"ea6938c8-3f32-4aa9-8a26-660f2aa4e20f\") " Oct 30 16:37:47.928106 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:47.928078 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea6938c8-3f32-4aa9-8a26-660f2aa4e20f-maas-api-openshift-ca" pod "ea6938c8-3f32-4aa9-8a26-660f2aa4e20f" (UID: "ea6938c8-3f32-4aa9-8a26-660f2aa4e20f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:37:47.930009 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:47.929964 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea6938c8-3f32-4aa9-8a26-660f2aa4e20f-kube-api-access-nxvk6" pod "ea6938c8-3f32-4aa9-8a26-660f2aa4e20f" (UID: "ea6938c8-3f32-4aa9-8a26-660f2aa4e20f"). InnerVolumeSpecName "kube-api-access-nxvk6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:37:47.930447 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:47.930425 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea6938c8-3f32-4aa9-8a26-660f2aa4e20f-maas-api-xks-ca" pod "ea6938c8-3f32-4aa9-8a26-660f2aa4e20f" (UID: "ea6938c8-3f32-4aa9-8a26-660f2aa4e20f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:37:48.029951 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:48.029817 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea6938c8-3f32-4aa9-8a26-660f2aa4e20f-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:37:48.029951 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:48.029892 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea6938c8-3f32-4aa9-8a26-660f2aa4e20f-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:37:48.029951 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:48.029907 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxvk6\" (UniqueName: \"kubernetes.io/projected/ea6938c8-3f32-4aa9-8a26-660f2aa4e20f-kube-api-access-nxvk6\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:37:48.685181 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:48.685151 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a3be62e-ktqk2" Oct 30 16:37:48.685351 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:48.685197 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a3be62e-ktqk2" event={"ID":"ea6938c8-3f32-4aa9-8a26-660f2aa4e20f","Type":"ContainerDied","Data":"7b0d2963d6b3b0f1d5d6d09ee761f5e54620a462b616eaa8e7fde6f24dd57933"} Oct 30 16:37:48.685351 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:48.685265 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b0d2963d6b3b0f1d5d6d09ee761f5e54620a462b616eaa8e7fde6f24dd57933" Oct 30 16:37:48.762191 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:48.762152 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b602696d-vqqhk"] Oct 30 16:37:48.762685 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:48.762670 4493 cpu_manager.go:414] "containerMap: removing container" podUID="ea6938c8-3f32-4aa9-8a26-660f2aa4e20f" containerName="revoke-keys" Oct 30 16:37:48.762685 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:48.762685 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea6938c8-3f32-4aa9-8a26-660f2aa4e20f" containerName="revoke-keys" Oct 30 16:37:48.762888 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:48.762809 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea6938c8-3f32-4aa9-8a26-660f2aa4e20f" containerName="revoke-keys" Oct 30 16:37:48.766399 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:48.766381 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b602696d-vqqhk" Oct 30 16:37:48.783646 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:48.783626 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:37:48.783758 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:48.783660 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:37:48.783758 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:48.783691 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:37:48.796021 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:48.795998 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 16:37:48.802115 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:48.802087 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b602696d-vqqhk"] Oct 30 16:37:48.845290 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:48.845253 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12a64168-7ea5-4014-a6ff-d7070ed00f84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b602696d-vqqhk\" (UID: \"12a64168-7ea5-4014-a6ff-d7070ed00f84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b602696d-vqqhk" Oct 30 16:37:48.845290 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:48.845288 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12a64168-7ea5-4014-a6ff-d7070ed00f84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b602696d-vqqhk\" (UID: \"12a64168-7ea5-4014-a6ff-d7070ed00f84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b602696d-vqqhk" Oct 30 16:37:48.845810 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:48.845549 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tndfj\" (UniqueName: \"kubernetes.io/projected/12a64168-7ea5-4014-a6ff-d7070ed00f84-kube-api-access-tndfj\") pod \"maas-api-revoke-sub-b602696d-vqqhk\" (UID: \"12a64168-7ea5-4014-a6ff-d7070ed00f84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b602696d-vqqhk" Oct 30 16:37:48.946853 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:48.946734 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tndfj\" (UniqueName: \"kubernetes.io/projected/12a64168-7ea5-4014-a6ff-d7070ed00f84-kube-api-access-tndfj\") pod \"maas-api-revoke-sub-b602696d-vqqhk\" (UID: \"12a64168-7ea5-4014-a6ff-d7070ed00f84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b602696d-vqqhk" Oct 30 16:37:48.946853 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:48.946832 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12a64168-7ea5-4014-a6ff-d7070ed00f84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b602696d-vqqhk\" (UID: \"12a64168-7ea5-4014-a6ff-d7070ed00f84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b602696d-vqqhk" Oct 30 16:37:48.947070 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:48.946855 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12a64168-7ea5-4014-a6ff-d7070ed00f84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b602696d-vqqhk\" (UID: \"12a64168-7ea5-4014-a6ff-d7070ed00f84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b602696d-vqqhk" Oct 30 16:37:48.947476 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:48.947455 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12a64168-7ea5-4014-a6ff-d7070ed00f84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b602696d-vqqhk\" (UID: \"12a64168-7ea5-4014-a6ff-d7070ed00f84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b602696d-vqqhk" Oct 30 16:37:48.949377 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:48.949355 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12a64168-7ea5-4014-a6ff-d7070ed00f84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b602696d-vqqhk\" (UID: \"12a64168-7ea5-4014-a6ff-d7070ed00f84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b602696d-vqqhk" Oct 30 16:37:48.970019 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:48.969995 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tndfj\" (UniqueName: \"kubernetes.io/projected/12a64168-7ea5-4014-a6ff-d7070ed00f84-kube-api-access-tndfj\") pod \"maas-api-revoke-sub-b602696d-vqqhk\" (UID: \"12a64168-7ea5-4014-a6ff-d7070ed00f84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b602696d-vqqhk" Oct 30 16:37:49.080401 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:49.080363 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b602696d-vqqhk" Oct 30 16:37:49.395025 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:49.394987 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b602696d-vqqhk"] Oct 30 16:37:49.693117 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:49.693048 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b602696d-vqqhk" event={"ID":"12a64168-7ea5-4014-a6ff-d7070ed00f84","Type":"ContainerStarted","Data":"c4a3f68ea0a028d3b6b5daf001c8a78fa53326c751b8bd5f5dbc003d2d92c8bc"} Oct 30 16:37:50.701763 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:50.701725 4493 generic.go:356] "Generic (PLEG): container finished" podID="12a64168-7ea5-4014-a6ff-d7070ed00f84" containerID="f4a6fcca5ed927bf9fa1a3cc71e14a6388a5c53ad77d2051bd58acec6332c1ee" exitCode=0 Oct 30 16:37:50.702201 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:50.701796 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b602696d-vqqhk" event={"ID":"12a64168-7ea5-4014-a6ff-d7070ed00f84","Type":"ContainerDied","Data":"f4a6fcca5ed927bf9fa1a3cc71e14a6388a5c53ad77d2051bd58acec6332c1ee"} Oct 30 16:37:52.339864 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:52.325006 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b602696d-vqqhk" Oct 30 16:37:52.408811 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:52.386280 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12a64168-7ea5-4014-a6ff-d7070ed00f84-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12a64168-7ea5-4014-a6ff-d7070ed00f84-maas-api-openshift-ca\") pod \"12a64168-7ea5-4014-a6ff-d7070ed00f84\" (UID: \"12a64168-7ea5-4014-a6ff-d7070ed00f84\") " Oct 30 16:37:52.408811 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:52.386471 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12a64168-7ea5-4014-a6ff-d7070ed00f84-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12a64168-7ea5-4014-a6ff-d7070ed00f84-maas-api-xks-ca\") pod \"12a64168-7ea5-4014-a6ff-d7070ed00f84\" (UID: \"12a64168-7ea5-4014-a6ff-d7070ed00f84\") " Oct 30 16:37:52.408811 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:52.386508 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12a64168-7ea5-4014-a6ff-d7070ed00f84-kube-api-access-tndfj\" (UniqueName: \"kubernetes.io/projected/12a64168-7ea5-4014-a6ff-d7070ed00f84-kube-api-access-tndfj\") pod \"12a64168-7ea5-4014-a6ff-d7070ed00f84\" (UID: \"12a64168-7ea5-4014-a6ff-d7070ed00f84\") " Oct 30 16:37:52.408811 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:52.387685 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12a64168-7ea5-4014-a6ff-d7070ed00f84-maas-api-openshift-ca" pod "12a64168-7ea5-4014-a6ff-d7070ed00f84" (UID: "12a64168-7ea5-4014-a6ff-d7070ed00f84"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:37:52.443179 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:52.420665 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12a64168-7ea5-4014-a6ff-d7070ed00f84-kube-api-access-tndfj" pod "12a64168-7ea5-4014-a6ff-d7070ed00f84" (UID: "12a64168-7ea5-4014-a6ff-d7070ed00f84"). InnerVolumeSpecName "kube-api-access-tndfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:37:52.443179 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:52.420814 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12a64168-7ea5-4014-a6ff-d7070ed00f84-maas-api-xks-ca" pod "12a64168-7ea5-4014-a6ff-d7070ed00f84" (UID: "12a64168-7ea5-4014-a6ff-d7070ed00f84"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:37:52.491822 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:52.491761 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12a64168-7ea5-4014-a6ff-d7070ed00f84-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:37:52.492020 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:52.491827 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tndfj\" (UniqueName: \"kubernetes.io/projected/12a64168-7ea5-4014-a6ff-d7070ed00f84-kube-api-access-tndfj\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:37:52.492020 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:52.491849 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12a64168-7ea5-4014-a6ff-d7070ed00f84-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:37:52.875810 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:52.875657 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b602696d-vqqhk" event={"ID":"12a64168-7ea5-4014-a6ff-d7070ed00f84","Type":"ContainerDied","Data":"c4a3f68ea0a028d3b6b5daf001c8a78fa53326c751b8bd5f5dbc003d2d92c8bc"} Oct 30 16:37:52.875810 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:52.875762 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4a3f68ea0a028d3b6b5daf001c8a78fa53326c751b8bd5f5dbc003d2d92c8bc" Oct 30 16:37:52.876078 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:52.875860 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b602696d-vqqhk" Oct 30 16:37:53.355809 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:53.354992 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-56v9s_d3a1c417-2698-4462-b7ff-1d2e66d7a10f/ovn-acl-logging/0.log" Oct 30 16:37:53.401808 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:53.401368 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-56v9s_d3a1c417-2698-4462-b7ff-1d2e66d7a10f/ovn-acl-logging/0.log" Oct 30 16:37:53.569817 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:53.569499 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" podUID="b5abe703-fee0-4185-aa22-88d1db06392b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:53.656873 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:53.638859 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" podUID="0eadc5d4-98c3-4bf2-b3c2-f73fe249182f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:53.656873 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:53.639183 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" podUID="14805045-a727-41ba-8bd0-100d41f9f27d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 16:37:53.824812 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:53.823923 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" podUID="80b9bd9a-30d6-45ff-b318-26fcdc608e5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:54.136946 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:54.128056 4493 patch_prober.go:29] interesting pod/e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:37:54.136946 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:54.128122 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr" podUID="6b860be8-5304-4364-a0b6-e06906cf5f4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:54.380655 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:54.380595 4493 patch_prober.go:29] interesting pod/e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:37:54.381174 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:54.380664 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl" podUID="c933c090-d160-449b-a3e3-75aec1c2ce36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:54.578931 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:54.578892 4493 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:37:54.579239 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:54.579207 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br" podUID="b30c2fa0-a3d3-4942-9596-bb6b245a238f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:54.584964 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:54.584862 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" podUID="e43738f3-d55f-4c0a-a364-bf2ac76605c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:54.916965 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:54.916917 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" podUID="18b72648-f523-405e-b5cd-1266ff13f8f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:54.966337 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:54.966296 4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" podUID="e43738f3-d55f-4c0a-a364-bf2ac76605c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:58.513678 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:58.513646 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt"] Oct 30 16:37:58.514314 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:58.514296 4493 cpu_manager.go:414] "containerMap: removing container" podUID="12a64168-7ea5-4014-a6ff-d7070ed00f84" containerName="revoke-keys" Oct 30 16:37:58.514314 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:58.514313 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12a64168-7ea5-4014-a6ff-d7070ed00f84" containerName="revoke-keys" Oct 30 16:37:58.514459 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:58.514413 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="12a64168-7ea5-4014-a6ff-d7070ed00f84" containerName="revoke-keys" Oct 30 16:37:58.519153 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:58.519131 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" Oct 30 16:37:58.546236 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:58.546183 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-18519c\"/\"distinct-2-w2-18519c-kserve-self-signed-certs\"" Oct 30 16:37:58.553062 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:58.553040 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt"] Oct 30 16:37:58.572524 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:58.572442 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hf9dq\" (UniqueName: \"kubernetes.io/projected/f3e1c8ff-fcc3-4a1f-9730-c617d57b5193-kube-api-access-hf9dq\") pod \"distinct-2-w2-18519c-kserve-85f6799df5-xjjxt\" (UID: \"f3e1c8ff-fcc3-4a1f-9730-c617d57b5193\") " pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" Oct 30 16:37:58.572871 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:58.572761 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3e1c8ff-fcc3-4a1f-9730-c617d57b5193-dshm\") pod \"distinct-2-w2-18519c-kserve-85f6799df5-xjjxt\" (UID: \"f3e1c8ff-fcc3-4a1f-9730-c617d57b5193\") " pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" Oct 30 16:37:58.573117 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:58.573101 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3e1c8ff-fcc3-4a1f-9730-c617d57b5193-tls-certs\") pod \"distinct-2-w2-18519c-kserve-85f6799df5-xjjxt\" (UID: \"f3e1c8ff-fcc3-4a1f-9730-c617d57b5193\") " pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" Oct 30 16:37:58.573296 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:58.573280 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3e1c8ff-fcc3-4a1f-9730-c617d57b5193-tmp-dir\") pod \"distinct-2-w2-18519c-kserve-85f6799df5-xjjxt\" (UID: \"f3e1c8ff-fcc3-4a1f-9730-c617d57b5193\") " pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" Oct 30 16:37:58.573468 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:58.573454 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3e1c8ff-fcc3-4a1f-9730-c617d57b5193-home\") pod \"distinct-2-w2-18519c-kserve-85f6799df5-xjjxt\" (UID: \"f3e1c8ff-fcc3-4a1f-9730-c617d57b5193\") " pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" Oct 30 16:37:58.573649 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:58.573636 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3e1c8ff-fcc3-4a1f-9730-c617d57b5193-model-cache\") pod \"distinct-2-w2-18519c-kserve-85f6799df5-xjjxt\" (UID: \"f3e1c8ff-fcc3-4a1f-9730-c617d57b5193\") " pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" Oct 30 16:37:58.675209 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:58.675171 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3e1c8ff-fcc3-4a1f-9730-c617d57b5193-model-cache\") pod \"distinct-2-w2-18519c-kserve-85f6799df5-xjjxt\" (UID: \"f3e1c8ff-fcc3-4a1f-9730-c617d57b5193\") " pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" Oct 30 16:37:58.675394 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:58.675298 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf9dq\" (UniqueName: \"kubernetes.io/projected/f3e1c8ff-fcc3-4a1f-9730-c617d57b5193-kube-api-access-hf9dq\") pod \"distinct-2-w2-18519c-kserve-85f6799df5-xjjxt\" (UID: \"f3e1c8ff-fcc3-4a1f-9730-c617d57b5193\") " pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" Oct 30 16:37:58.675394 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:58.675353 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3e1c8ff-fcc3-4a1f-9730-c617d57b5193-dshm\") pod \"distinct-2-w2-18519c-kserve-85f6799df5-xjjxt\" (UID: \"f3e1c8ff-fcc3-4a1f-9730-c617d57b5193\") " pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" Oct 30 16:37:58.675514 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:58.675447 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3e1c8ff-fcc3-4a1f-9730-c617d57b5193-tls-certs\") pod \"distinct-2-w2-18519c-kserve-85f6799df5-xjjxt\" (UID: \"f3e1c8ff-fcc3-4a1f-9730-c617d57b5193\") " pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" Oct 30 16:37:58.675514 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:58.675485 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3e1c8ff-fcc3-4a1f-9730-c617d57b5193-tmp-dir\") pod \"distinct-2-w2-18519c-kserve-85f6799df5-xjjxt\" (UID: \"f3e1c8ff-fcc3-4a1f-9730-c617d57b5193\") " pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" Oct 30 16:37:58.675628 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:58.675516 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3e1c8ff-fcc3-4a1f-9730-c617d57b5193-home\") pod \"distinct-2-w2-18519c-kserve-85f6799df5-xjjxt\" (UID: \"f3e1c8ff-fcc3-4a1f-9730-c617d57b5193\") " pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" Oct 30 16:37:58.675685 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:58.675645 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3e1c8ff-fcc3-4a1f-9730-c617d57b5193-model-cache\") pod \"distinct-2-w2-18519c-kserve-85f6799df5-xjjxt\" (UID: \"f3e1c8ff-fcc3-4a1f-9730-c617d57b5193\") " pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" Oct 30 16:37:58.675990 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:58.675876 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3e1c8ff-fcc3-4a1f-9730-c617d57b5193-home\") pod \"distinct-2-w2-18519c-kserve-85f6799df5-xjjxt\" (UID: \"f3e1c8ff-fcc3-4a1f-9730-c617d57b5193\") " pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" Oct 30 16:37:58.675990 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:58.675931 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3e1c8ff-fcc3-4a1f-9730-c617d57b5193-tmp-dir\") pod \"distinct-2-w2-18519c-kserve-85f6799df5-xjjxt\" (UID: \"f3e1c8ff-fcc3-4a1f-9730-c617d57b5193\") " pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" Oct 30 16:37:58.677964 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:58.677896 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3e1c8ff-fcc3-4a1f-9730-c617d57b5193-dshm\") pod \"distinct-2-w2-18519c-kserve-85f6799df5-xjjxt\" (UID: \"f3e1c8ff-fcc3-4a1f-9730-c617d57b5193\") " pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" Oct 30 16:37:58.678675 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:58.678651 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3e1c8ff-fcc3-4a1f-9730-c617d57b5193-tls-certs\") pod \"distinct-2-w2-18519c-kserve-85f6799df5-xjjxt\" (UID: \"f3e1c8ff-fcc3-4a1f-9730-c617d57b5193\") " pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" Oct 30 16:37:58.698011 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:58.697982 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1da9e70e-hnq6n"] Oct 30 16:37:58.701433 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:58.701410 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hf9dq\" (UniqueName: \"kubernetes.io/projected/f3e1c8ff-fcc3-4a1f-9730-c617d57b5193-kube-api-access-hf9dq\") pod \"distinct-2-w2-18519c-kserve-85f6799df5-xjjxt\" (UID: \"f3e1c8ff-fcc3-4a1f-9730-c617d57b5193\") " pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" Oct 30 16:37:58.702673 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:58.702653 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1da9e70e-hnq6n" Oct 30 16:37:58.722953 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:58.720672 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:37:58.722953 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:58.720802 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 16:37:58.722953 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:58.720886 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:37:58.722953 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:58.720942 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:37:58.747004 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:58.746941 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1da9e70e-hnq6n"] Oct 30 16:37:58.783320 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:58.781393 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18d34af8-18c4-453c-bfd3-7a4a8dffc7ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1da9e70e-hnq6n\" (UID: \"18d34af8-18c4-453c-bfd3-7a4a8dffc7ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1da9e70e-hnq6n" Oct 30 16:37:58.783320 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:58.781444 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18d34af8-18c4-453c-bfd3-7a4a8dffc7ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1da9e70e-hnq6n\" (UID: \"18d34af8-18c4-453c-bfd3-7a4a8dffc7ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1da9e70e-hnq6n" Oct 30 16:37:58.783320 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:58.781511 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87ppz\" (UniqueName: \"kubernetes.io/projected/18d34af8-18c4-453c-bfd3-7a4a8dffc7ff-kube-api-access-87ppz\") pod \"maas-api-revoke-sub-1da9e70e-hnq6n\" (UID: \"18d34af8-18c4-453c-bfd3-7a4a8dffc7ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1da9e70e-hnq6n" Oct 30 16:37:58.882042 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:58.882010 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" Oct 30 16:37:58.885370 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:58.883904 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18d34af8-18c4-453c-bfd3-7a4a8dffc7ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1da9e70e-hnq6n\" (UID: \"18d34af8-18c4-453c-bfd3-7a4a8dffc7ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1da9e70e-hnq6n" Oct 30 16:37:58.885370 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:58.883954 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18d34af8-18c4-453c-bfd3-7a4a8dffc7ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1da9e70e-hnq6n\" (UID: \"18d34af8-18c4-453c-bfd3-7a4a8dffc7ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1da9e70e-hnq6n" Oct 30 16:37:58.885370 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:58.884024 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87ppz\" (UniqueName: \"kubernetes.io/projected/18d34af8-18c4-453c-bfd3-7a4a8dffc7ff-kube-api-access-87ppz\") pod \"maas-api-revoke-sub-1da9e70e-hnq6n\" (UID: \"18d34af8-18c4-453c-bfd3-7a4a8dffc7ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1da9e70e-hnq6n" Oct 30 16:37:58.885370 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:58.885224 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18d34af8-18c4-453c-bfd3-7a4a8dffc7ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1da9e70e-hnq6n\" (UID: \"18d34af8-18c4-453c-bfd3-7a4a8dffc7ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1da9e70e-hnq6n" Oct 30 16:37:58.887740 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:58.887722 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18d34af8-18c4-453c-bfd3-7a4a8dffc7ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1da9e70e-hnq6n\" (UID: \"18d34af8-18c4-453c-bfd3-7a4a8dffc7ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1da9e70e-hnq6n" Oct 30 16:37:58.908319 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:58.908293 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87ppz\" (UniqueName: \"kubernetes.io/projected/18d34af8-18c4-453c-bfd3-7a4a8dffc7ff-kube-api-access-87ppz\") pod \"maas-api-revoke-sub-1da9e70e-hnq6n\" (UID: \"18d34af8-18c4-453c-bfd3-7a4a8dffc7ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1da9e70e-hnq6n" Oct 30 16:37:59.088163 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:59.088096 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1da9e70e-hnq6n" Oct 30 16:37:59.219541 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:59.219512 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt"] Oct 30 16:37:59.222586 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:37:59.222554 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf3e1c8ff_fcc3_4a1f_9730_c617d57b5193.slice/crio-9d2459c2b65702848197ff48d8f63bbb0b3ba48eea70de5fb5dc8a1eef208d5a WatchSource:0}: Error finding container 9d2459c2b65702848197ff48d8f63bbb0b3ba48eea70de5fb5dc8a1eef208d5a: Status 404 returned error can't find the container with id 9d2459c2b65702848197ff48d8f63bbb0b3ba48eea70de5fb5dc8a1eef208d5a Oct 30 16:37:59.403702 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:37:59.403677 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18d34af8_18c4_453c_bfd3_7a4a8dffc7ff.slice/crio-7e0bb7d4603522d6274c622e66a0b15c3f78f98073b7e49860f036bee4c64c70 WatchSource:0}: Error finding container 7e0bb7d4603522d6274c622e66a0b15c3f78f98073b7e49860f036bee4c64c70: Status 404 returned error can't find the container with id 7e0bb7d4603522d6274c622e66a0b15c3f78f98073b7e49860f036bee4c64c70 Oct 30 16:37:59.403932 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:37:59.403908 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1da9e70e-hnq6n"] Oct 30 16:38:00.061528 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:00.061435 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" event={"ID":"f3e1c8ff-fcc3-4a1f-9730-c617d57b5193","Type":"ContainerStarted","Data":"d22765a80c71546b77c331be945ade3d585939b7efc74b5b2adb47c9815bf2d4"} Oct 30 16:38:00.061528 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:00.061481 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" event={"ID":"f3e1c8ff-fcc3-4a1f-9730-c617d57b5193","Type":"ContainerStarted","Data":"9d2459c2b65702848197ff48d8f63bbb0b3ba48eea70de5fb5dc8a1eef208d5a"} Oct 30 16:38:00.063121 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:00.063091 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1da9e70e-hnq6n" event={"ID":"18d34af8-18c4-453c-bfd3-7a4a8dffc7ff","Type":"ContainerStarted","Data":"c380ce61c044d1c138352a4b070f6fc63caf9693031cc6380221910983280958"} Oct 30 16:38:00.063230 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:00.063147 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1da9e70e-hnq6n" event={"ID":"18d34af8-18c4-453c-bfd3-7a4a8dffc7ff","Type":"ContainerStarted","Data":"7e0bb7d4603522d6274c622e66a0b15c3f78f98073b7e49860f036bee4c64c70"} Oct 30 16:38:00.106123 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:00.106068 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" podStartSLOduration=2.106052878 podStartE2EDuration="2.106052878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:37:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:38:00.105901917 +0000 UTC m=+907.333077018" watchObservedRunningTime="2026-10-30 16:38:00.106052878 +0000 UTC m=+907.333227980" Oct 30 16:38:00.152156 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:00.152095 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1da9e70e-hnq6n" podStartSLOduration=2.152077028 podStartE2EDuration="2.152077028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:37:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:38:00.146896538 +0000 UTC m=+907.374071639" watchObservedRunningTime="2026-10-30 16:38:00.152077028 +0000 UTC m=+907.379252129" Oct 30 16:38:00.345145 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:38:00.345074 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 16:38:00.882948 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:00.882913 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" Oct 30 16:38:01.071770 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:01.071733 4493 generic.go:356] "Generic (PLEG): container finished" podID="18d34af8-18c4-453c-bfd3-7a4a8dffc7ff" containerID="c380ce61c044d1c138352a4b070f6fc63caf9693031cc6380221910983280958" exitCode=0 Oct 30 16:38:01.072219 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:01.071830 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1da9e70e-hnq6n" event={"ID":"18d34af8-18c4-453c-bfd3-7a4a8dffc7ff","Type":"ContainerDied","Data":"c380ce61c044d1c138352a4b070f6fc63caf9693031cc6380221910983280958"} Oct 30 16:38:02.268550 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:02.268527 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1da9e70e-hnq6n" Oct 30 16:38:02.334112 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:02.333264 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18d34af8-18c4-453c-bfd3-7a4a8dffc7ff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18d34af8-18c4-453c-bfd3-7a4a8dffc7ff-maas-api-openshift-ca\") pod \"18d34af8-18c4-453c-bfd3-7a4a8dffc7ff\" (UID: \"18d34af8-18c4-453c-bfd3-7a4a8dffc7ff\") " Oct 30 16:38:02.334112 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:02.333390 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18d34af8-18c4-453c-bfd3-7a4a8dffc7ff-kube-api-access-87ppz\" (UniqueName: \"kubernetes.io/projected/18d34af8-18c4-453c-bfd3-7a4a8dffc7ff-kube-api-access-87ppz\") pod \"18d34af8-18c4-453c-bfd3-7a4a8dffc7ff\" (UID: \"18d34af8-18c4-453c-bfd3-7a4a8dffc7ff\") " Oct 30 16:38:02.334112 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:02.333443 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18d34af8-18c4-453c-bfd3-7a4a8dffc7ff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18d34af8-18c4-453c-bfd3-7a4a8dffc7ff-maas-api-xks-ca\") pod \"18d34af8-18c4-453c-bfd3-7a4a8dffc7ff\" (UID: \"18d34af8-18c4-453c-bfd3-7a4a8dffc7ff\") " Oct 30 16:38:02.334112 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:02.333749 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18d34af8-18c4-453c-bfd3-7a4a8dffc7ff-maas-api-openshift-ca" pod "18d34af8-18c4-453c-bfd3-7a4a8dffc7ff" (UID: "18d34af8-18c4-453c-bfd3-7a4a8dffc7ff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:38:02.334112 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:02.334037 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18d34af8-18c4-453c-bfd3-7a4a8dffc7ff-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:38:02.336213 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:02.336188 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18d34af8-18c4-453c-bfd3-7a4a8dffc7ff-kube-api-access-87ppz" pod "18d34af8-18c4-453c-bfd3-7a4a8dffc7ff" (UID: "18d34af8-18c4-453c-bfd3-7a4a8dffc7ff"). InnerVolumeSpecName "kube-api-access-87ppz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:38:02.336408 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:02.336388 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18d34af8-18c4-453c-bfd3-7a4a8dffc7ff-maas-api-xks-ca" pod "18d34af8-18c4-453c-bfd3-7a4a8dffc7ff" (UID: "18d34af8-18c4-453c-bfd3-7a4a8dffc7ff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:38:02.435019 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:02.434990 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87ppz\" (UniqueName: \"kubernetes.io/projected/18d34af8-18c4-453c-bfd3-7a4a8dffc7ff-kube-api-access-87ppz\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:38:02.435019 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:02.435017 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18d34af8-18c4-453c-bfd3-7a4a8dffc7ff-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:38:03.010377 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:03.010340 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-660c8fee-trm9h"] Oct 30 16:38:03.010955 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:03.010939 4493 cpu_manager.go:414] "containerMap: removing container" podUID="18d34af8-18c4-453c-bfd3-7a4a8dffc7ff" containerName="revoke-keys" Oct 30 16:38:03.010955 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:03.010954 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18d34af8-18c4-453c-bfd3-7a4a8dffc7ff" containerName="revoke-keys" Oct 30 16:38:03.011098 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:03.011046 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="18d34af8-18c4-453c-bfd3-7a4a8dffc7ff" containerName="revoke-keys" Oct 30 16:38:03.014713 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:03.014690 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-660c8fee-trm9h" Oct 30 16:38:03.038550 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:03.038520 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-660c8fee-trm9h"] Oct 30 16:38:03.089669 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:03.089633 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1da9e70e-hnq6n" Oct 30 16:38:03.089842 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:03.089670 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1da9e70e-hnq6n" event={"ID":"18d34af8-18c4-453c-bfd3-7a4a8dffc7ff","Type":"ContainerDied","Data":"7e0bb7d4603522d6274c622e66a0b15c3f78f98073b7e49860f036bee4c64c70"} Oct 30 16:38:03.089842 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:03.089719 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e0bb7d4603522d6274c622e66a0b15c3f78f98073b7e49860f036bee4c64c70" Oct 30 16:38:03.151202 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:03.151170 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhqz7\" (UniqueName: \"kubernetes.io/projected/1bad8db2-5b23-43c5-a568-8e66d2dc59b5-kube-api-access-jhqz7\") pod \"maas-api-revoke-sub-660c8fee-trm9h\" (UID: \"1bad8db2-5b23-43c5-a568-8e66d2dc59b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-660c8fee-trm9h" Oct 30 16:38:03.151408 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:03.151375 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bad8db2-5b23-43c5-a568-8e66d2dc59b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-660c8fee-trm9h\" (UID: \"1bad8db2-5b23-43c5-a568-8e66d2dc59b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-660c8fee-trm9h" Oct 30 16:38:03.151476 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:03.151433 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bad8db2-5b23-43c5-a568-8e66d2dc59b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-660c8fee-trm9h\" (UID: \"1bad8db2-5b23-43c5-a568-8e66d2dc59b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-660c8fee-trm9h" Oct 30 16:38:03.252663 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:03.252570 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bad8db2-5b23-43c5-a568-8e66d2dc59b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-660c8fee-trm9h\" (UID: \"1bad8db2-5b23-43c5-a568-8e66d2dc59b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-660c8fee-trm9h" Oct 30 16:38:03.252663 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:03.252660 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bad8db2-5b23-43c5-a568-8e66d2dc59b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-660c8fee-trm9h\" (UID: \"1bad8db2-5b23-43c5-a568-8e66d2dc59b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-660c8fee-trm9h" Oct 30 16:38:03.252924 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:03.252682 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhqz7\" (UniqueName: \"kubernetes.io/projected/1bad8db2-5b23-43c5-a568-8e66d2dc59b5-kube-api-access-jhqz7\") pod \"maas-api-revoke-sub-660c8fee-trm9h\" (UID: \"1bad8db2-5b23-43c5-a568-8e66d2dc59b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-660c8fee-trm9h" Oct 30 16:38:03.253383 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:03.253360 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bad8db2-5b23-43c5-a568-8e66d2dc59b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-660c8fee-trm9h\" (UID: \"1bad8db2-5b23-43c5-a568-8e66d2dc59b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-660c8fee-trm9h" Oct 30 16:38:03.255269 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:03.255246 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bad8db2-5b23-43c5-a568-8e66d2dc59b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-660c8fee-trm9h\" (UID: \"1bad8db2-5b23-43c5-a568-8e66d2dc59b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-660c8fee-trm9h" Oct 30 16:38:03.276960 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:03.276926 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhqz7\" (UniqueName: \"kubernetes.io/projected/1bad8db2-5b23-43c5-a568-8e66d2dc59b5-kube-api-access-jhqz7\") pod \"maas-api-revoke-sub-660c8fee-trm9h\" (UID: \"1bad8db2-5b23-43c5-a568-8e66d2dc59b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-660c8fee-trm9h" Oct 30 16:38:03.330610 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:03.330581 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-660c8fee-trm9h" Oct 30 16:38:03.618230 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:03.618155 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 16:38:03.662919 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:03.662889 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-660c8fee-trm9h"] Oct 30 16:38:03.663443 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:38:03.663416 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bad8db2_5b23_43c5_a568_8e66d2dc59b5.slice/crio-35ed3262b9493fc75ab78ae9ed1bdab0176ba9ed7321229b09c027c921c4b091 WatchSource:0}: Error finding container 35ed3262b9493fc75ab78ae9ed1bdab0176ba9ed7321229b09c027c921c4b091: Status 404 returned error can't find the container with id 35ed3262b9493fc75ab78ae9ed1bdab0176ba9ed7321229b09c027c921c4b091 Oct 30 16:38:04.100089 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:04.100050 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-660c8fee-trm9h" event={"ID":"1bad8db2-5b23-43c5-a568-8e66d2dc59b5","Type":"ContainerStarted","Data":"b6eeeec2fd4bdf5fd568dd2b82208b2391e66387e60935c0109a0c3cad709f07"} Oct 30 16:38:04.100250 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:04.100111 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-660c8fee-trm9h" event={"ID":"1bad8db2-5b23-43c5-a568-8e66d2dc59b5","Type":"ContainerStarted","Data":"35ed3262b9493fc75ab78ae9ed1bdab0176ba9ed7321229b09c027c921c4b091"} Oct 30 16:38:04.152809 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:04.152545 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-660c8fee-trm9h" podStartSLOduration=2.152521248 podStartE2EDuration="2.152521248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:38:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:38:04.151018786 +0000 UTC m=+911.378193894" watchObservedRunningTime="2026-10-30 16:38:04.152521248 +0000 UTC m=+911.379696356" Oct 30 16:38:05.141854 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:05.141818 4493 generic.go:356] "Generic (PLEG): container finished" podID="1bad8db2-5b23-43c5-a568-8e66d2dc59b5" containerID="b6eeeec2fd4bdf5fd568dd2b82208b2391e66387e60935c0109a0c3cad709f07" exitCode=0 Oct 30 16:38:05.142805 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:05.142749 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-660c8fee-trm9h" event={"ID":"1bad8db2-5b23-43c5-a568-8e66d2dc59b5","Type":"ContainerDied","Data":"b6eeeec2fd4bdf5fd568dd2b82208b2391e66387e60935c0109a0c3cad709f07"} Oct 30 16:38:06.387709 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:06.387681 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-660c8fee-trm9h" Oct 30 16:38:06.567859 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:06.567738 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bad8db2-5b23-43c5-a568-8e66d2dc59b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bad8db2-5b23-43c5-a568-8e66d2dc59b5-maas-api-openshift-ca\") pod \"1bad8db2-5b23-43c5-a568-8e66d2dc59b5\" (UID: \"1bad8db2-5b23-43c5-a568-8e66d2dc59b5\") " Oct 30 16:38:06.567859 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:06.567858 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1bad8db2-5b23-43c5-a568-8e66d2dc59b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bad8db2-5b23-43c5-a568-8e66d2dc59b5-maas-api-xks-ca\") pod \"1bad8db2-5b23-43c5-a568-8e66d2dc59b5\" (UID: \"1bad8db2-5b23-43c5-a568-8e66d2dc59b5\") " Oct 30 16:38:06.568064 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:06.567961 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bad8db2-5b23-43c5-a568-8e66d2dc59b5-kube-api-access-jhqz7\" (UniqueName: \"kubernetes.io/projected/1bad8db2-5b23-43c5-a568-8e66d2dc59b5-kube-api-access-jhqz7\") pod \"1bad8db2-5b23-43c5-a568-8e66d2dc59b5\" (UID: \"1bad8db2-5b23-43c5-a568-8e66d2dc59b5\") " Oct 30 16:38:06.568164 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:06.568144 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bad8db2-5b23-43c5-a568-8e66d2dc59b5-maas-api-openshift-ca" pod "1bad8db2-5b23-43c5-a568-8e66d2dc59b5" (UID: "1bad8db2-5b23-43c5-a568-8e66d2dc59b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:38:06.568495 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:06.568476 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bad8db2-5b23-43c5-a568-8e66d2dc59b5-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:38:06.570118 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:06.570095 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bad8db2-5b23-43c5-a568-8e66d2dc59b5-maas-api-xks-ca" pod "1bad8db2-5b23-43c5-a568-8e66d2dc59b5" (UID: "1bad8db2-5b23-43c5-a568-8e66d2dc59b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:38:06.570228 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:06.570197 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bad8db2-5b23-43c5-a568-8e66d2dc59b5-kube-api-access-jhqz7" pod "1bad8db2-5b23-43c5-a568-8e66d2dc59b5" (UID: "1bad8db2-5b23-43c5-a568-8e66d2dc59b5"). InnerVolumeSpecName "kube-api-access-jhqz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:38:06.669451 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:06.669419 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bad8db2-5b23-43c5-a568-8e66d2dc59b5-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:38:06.669451 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:06.669446 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhqz7\" (UniqueName: \"kubernetes.io/projected/1bad8db2-5b23-43c5-a568-8e66d2dc59b5-kube-api-access-jhqz7\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:38:07.163893 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:07.163863 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-660c8fee-trm9h" Oct 30 16:38:07.163893 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:07.163876 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-660c8fee-trm9h" event={"ID":"1bad8db2-5b23-43c5-a568-8e66d2dc59b5","Type":"ContainerDied","Data":"35ed3262b9493fc75ab78ae9ed1bdab0176ba9ed7321229b09c027c921c4b091"} Oct 30 16:38:07.164065 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:07.163928 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35ed3262b9493fc75ab78ae9ed1bdab0176ba9ed7321229b09c027c921c4b091" Oct 30 16:38:08.883239 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:08.883194 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" Oct 30 16:38:08.895537 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:08.895514 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" Oct 30 16:38:09.055101 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:09.055068 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e7b30f3-qtsf9"] Oct 30 16:38:09.055619 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:09.055607 4493 cpu_manager.go:414] "containerMap: removing container" podUID="1bad8db2-5b23-43c5-a568-8e66d2dc59b5" containerName="revoke-keys" Oct 30 16:38:09.055668 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:09.055621 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bad8db2-5b23-43c5-a568-8e66d2dc59b5" containerName="revoke-keys" Oct 30 16:38:09.055750 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:09.055741 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bad8db2-5b23-43c5-a568-8e66d2dc59b5" containerName="revoke-keys" Oct 30 16:38:09.058842 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:09.058818 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e7b30f3-qtsf9" Oct 30 16:38:09.075665 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:09.075641 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:38:09.075767 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:09.075691 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:38:09.075767 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:09.075715 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:38:09.075767 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:09.075753 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 16:38:09.081419 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:09.081398 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e7b30f3-qtsf9"] Oct 30 16:38:09.190207 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:09.190181 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" Oct 30 16:38:09.202112 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:09.202088 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d2a02f4-8325-41c6-a341-df1267e99563-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e7b30f3-qtsf9\" (UID: \"8d2a02f4-8325-41c6-a341-df1267e99563\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e7b30f3-qtsf9" Oct 30 16:38:09.202277 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:09.202128 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmhdz\" (UniqueName: \"kubernetes.io/projected/8d2a02f4-8325-41c6-a341-df1267e99563-kube-api-access-qmhdz\") pod \"maas-api-revoke-sub-8e7b30f3-qtsf9\" (UID: \"8d2a02f4-8325-41c6-a341-df1267e99563\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e7b30f3-qtsf9" Oct 30 16:38:09.202343 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:09.202309 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d2a02f4-8325-41c6-a341-df1267e99563-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e7b30f3-qtsf9\" (UID: \"8d2a02f4-8325-41c6-a341-df1267e99563\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e7b30f3-qtsf9" Oct 30 16:38:09.244122 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:09.244090 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22135e92-mzp7l"] Oct 30 16:38:09.249891 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:09.249868 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22135e92-mzp7l" Oct 30 16:38:09.270875 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:09.270847 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22135e92-mzp7l"] Oct 30 16:38:09.304309 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:09.304277 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d2a02f4-8325-41c6-a341-df1267e99563-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e7b30f3-qtsf9\" (UID: \"8d2a02f4-8325-41c6-a341-df1267e99563\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e7b30f3-qtsf9" Oct 30 16:38:09.304478 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:09.304342 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmhdz\" (UniqueName: \"kubernetes.io/projected/8d2a02f4-8325-41c6-a341-df1267e99563-kube-api-access-qmhdz\") pod \"maas-api-revoke-sub-8e7b30f3-qtsf9\" (UID: \"8d2a02f4-8325-41c6-a341-df1267e99563\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e7b30f3-qtsf9" Oct 30 16:38:09.304478 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:09.304434 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d2a02f4-8325-41c6-a341-df1267e99563-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e7b30f3-qtsf9\" (UID: \"8d2a02f4-8325-41c6-a341-df1267e99563\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e7b30f3-qtsf9" Oct 30 16:38:09.305209 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:09.305185 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d2a02f4-8325-41c6-a341-df1267e99563-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e7b30f3-qtsf9\" (UID: \"8d2a02f4-8325-41c6-a341-df1267e99563\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e7b30f3-qtsf9" Oct 30 16:38:09.307494 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:09.307465 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d2a02f4-8325-41c6-a341-df1267e99563-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e7b30f3-qtsf9\" (UID: \"8d2a02f4-8325-41c6-a341-df1267e99563\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e7b30f3-qtsf9" Oct 30 16:38:09.327652 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:09.327627 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmhdz\" (UniqueName: \"kubernetes.io/projected/8d2a02f4-8325-41c6-a341-df1267e99563-kube-api-access-qmhdz\") pod \"maas-api-revoke-sub-8e7b30f3-qtsf9\" (UID: \"8d2a02f4-8325-41c6-a341-df1267e99563\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e7b30f3-qtsf9" Oct 30 16:38:09.373340 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:09.373307 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e7b30f3-qtsf9" Oct 30 16:38:09.405989 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:09.405950 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e20ce11b-ff37-4c9a-b8be-503604a1bac4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22135e92-mzp7l\" (UID: \"e20ce11b-ff37-4c9a-b8be-503604a1bac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22135e92-mzp7l" Oct 30 16:38:09.406129 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:09.406084 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9rws\" (UniqueName: \"kubernetes.io/projected/e20ce11b-ff37-4c9a-b8be-503604a1bac4-kube-api-access-p9rws\") pod \"maas-api-revoke-sub-22135e92-mzp7l\" (UID: \"e20ce11b-ff37-4c9a-b8be-503604a1bac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22135e92-mzp7l" Oct 30 16:38:09.406192 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:09.406179 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e20ce11b-ff37-4c9a-b8be-503604a1bac4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22135e92-mzp7l\" (UID: \"e20ce11b-ff37-4c9a-b8be-503604a1bac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22135e92-mzp7l" Oct 30 16:38:09.507949 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:09.507915 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e20ce11b-ff37-4c9a-b8be-503604a1bac4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22135e92-mzp7l\" (UID: \"e20ce11b-ff37-4c9a-b8be-503604a1bac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22135e92-mzp7l" Oct 30 16:38:09.508120 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:09.507959 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9rws\" (UniqueName: \"kubernetes.io/projected/e20ce11b-ff37-4c9a-b8be-503604a1bac4-kube-api-access-p9rws\") pod \"maas-api-revoke-sub-22135e92-mzp7l\" (UID: \"e20ce11b-ff37-4c9a-b8be-503604a1bac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22135e92-mzp7l" Oct 30 16:38:09.508120 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:09.507991 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e20ce11b-ff37-4c9a-b8be-503604a1bac4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22135e92-mzp7l\" (UID: \"e20ce11b-ff37-4c9a-b8be-503604a1bac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22135e92-mzp7l" Oct 30 16:38:09.508608 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:09.508589 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e20ce11b-ff37-4c9a-b8be-503604a1bac4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22135e92-mzp7l\" (UID: \"e20ce11b-ff37-4c9a-b8be-503604a1bac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22135e92-mzp7l" Oct 30 16:38:09.511409 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:09.511361 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e20ce11b-ff37-4c9a-b8be-503604a1bac4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22135e92-mzp7l\" (UID: \"e20ce11b-ff37-4c9a-b8be-503604a1bac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22135e92-mzp7l" Oct 30 16:38:09.543343 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:09.543319 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9rws\" (UniqueName: \"kubernetes.io/projected/e20ce11b-ff37-4c9a-b8be-503604a1bac4-kube-api-access-p9rws\") pod \"maas-api-revoke-sub-22135e92-mzp7l\" (UID: \"e20ce11b-ff37-4c9a-b8be-503604a1bac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22135e92-mzp7l" Oct 30 16:38:09.576465 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:09.576441 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22135e92-mzp7l" Oct 30 16:38:09.696148 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:09.695932 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e7b30f3-qtsf9"] Oct 30 16:38:09.893356 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:09.893326 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22135e92-mzp7l"] Oct 30 16:38:09.893721 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:38:09.893417 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode20ce11b_ff37_4c9a_b8be_503604a1bac4.slice/crio-efa7ae1926662b287ab7148555b824e25db66e646522d2069480d1391c80ad7a WatchSource:0}: Error finding container efa7ae1926662b287ab7148555b824e25db66e646522d2069480d1391c80ad7a: Status 404 returned error can't find the container with id efa7ae1926662b287ab7148555b824e25db66e646522d2069480d1391c80ad7a Oct 30 16:38:10.190104 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:10.189759 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e7b30f3-qtsf9" event={"ID":"8d2a02f4-8325-41c6-a341-df1267e99563","Type":"ContainerStarted","Data":"4705851b329f640def4ff8216f9f1ba1f7125fd9e6ddcebd77f72ad5437b3caa"} Oct 30 16:38:10.190104 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:10.189864 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e7b30f3-qtsf9" event={"ID":"8d2a02f4-8325-41c6-a341-df1267e99563","Type":"ContainerStarted","Data":"3654370883eb07d08b4b97b7f84883d92f63da58e9bfdbb231163b14d646e7f1"} Oct 30 16:38:10.191987 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:10.191959 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22135e92-mzp7l" event={"ID":"e20ce11b-ff37-4c9a-b8be-503604a1bac4","Type":"ContainerStarted","Data":"efa7ae1926662b287ab7148555b824e25db66e646522d2069480d1391c80ad7a"} Oct 30 16:38:10.234697 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:10.234649 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e7b30f3-qtsf9" podStartSLOduration=1.234636781 podStartE2EDuration="1.234636781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:38:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:38:10.23002832 +0000 UTC m=+917.457203421" watchObservedRunningTime="2026-10-30 16:38:10.234636781 +0000 UTC m=+917.461811902" Oct 30 16:38:11.200940 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:11.200910 4493 generic.go:356] "Generic (PLEG): container finished" podID="e20ce11b-ff37-4c9a-b8be-503604a1bac4" containerID="87f6939a0a65da3d7e966fc97e24f5813ac63836965edbeb6e1aabda8263903e" exitCode=0 Oct 30 16:38:11.201374 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:11.200987 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22135e92-mzp7l" event={"ID":"e20ce11b-ff37-4c9a-b8be-503604a1bac4","Type":"ContainerDied","Data":"87f6939a0a65da3d7e966fc97e24f5813ac63836965edbeb6e1aabda8263903e"} Oct 30 16:38:11.202872 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:11.202845 4493 generic.go:356] "Generic (PLEG): container finished" podID="8d2a02f4-8325-41c6-a341-df1267e99563" containerID="4705851b329f640def4ff8216f9f1ba1f7125fd9e6ddcebd77f72ad5437b3caa" exitCode=0 Oct 30 16:38:11.203027 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:11.202930 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e7b30f3-qtsf9" event={"ID":"8d2a02f4-8325-41c6-a341-df1267e99563","Type":"ContainerDied","Data":"4705851b329f640def4ff8216f9f1ba1f7125fd9e6ddcebd77f72ad5437b3caa"} Oct 30 16:38:12.389425 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:12.389399 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22135e92-mzp7l" Oct 30 16:38:12.432775 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:12.432724 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e7b30f3-qtsf9" Oct 30 16:38:12.448494 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:12.448460 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e20ce11b-ff37-4c9a-b8be-503604a1bac4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e20ce11b-ff37-4c9a-b8be-503604a1bac4-maas-api-xks-ca\") pod \"e20ce11b-ff37-4c9a-b8be-503604a1bac4\" (UID: \"e20ce11b-ff37-4c9a-b8be-503604a1bac4\") " Oct 30 16:38:12.448701 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:12.448511 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d2a02f4-8325-41c6-a341-df1267e99563-kube-api-access-qmhdz\" (UniqueName: \"kubernetes.io/projected/8d2a02f4-8325-41c6-a341-df1267e99563-kube-api-access-qmhdz\") pod \"8d2a02f4-8325-41c6-a341-df1267e99563\" (UID: \"8d2a02f4-8325-41c6-a341-df1267e99563\") " Oct 30 16:38:12.448701 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:12.448603 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e20ce11b-ff37-4c9a-b8be-503604a1bac4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e20ce11b-ff37-4c9a-b8be-503604a1bac4-maas-api-openshift-ca\") pod \"e20ce11b-ff37-4c9a-b8be-503604a1bac4\" (UID: \"e20ce11b-ff37-4c9a-b8be-503604a1bac4\") " Oct 30 16:38:12.448701 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:12.448698 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d2a02f4-8325-41c6-a341-df1267e99563-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d2a02f4-8325-41c6-a341-df1267e99563-maas-api-xks-ca\") pod \"8d2a02f4-8325-41c6-a341-df1267e99563\" (UID: \"8d2a02f4-8325-41c6-a341-df1267e99563\") " Oct 30 16:38:12.448922 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:12.448751 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d2a02f4-8325-41c6-a341-df1267e99563-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d2a02f4-8325-41c6-a341-df1267e99563-maas-api-openshift-ca\") pod \"8d2a02f4-8325-41c6-a341-df1267e99563\" (UID: \"8d2a02f4-8325-41c6-a341-df1267e99563\") " Oct 30 16:38:12.448922 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:12.448813 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e20ce11b-ff37-4c9a-b8be-503604a1bac4-kube-api-access-p9rws\" (UniqueName: \"kubernetes.io/projected/e20ce11b-ff37-4c9a-b8be-503604a1bac4-kube-api-access-p9rws\") pod \"e20ce11b-ff37-4c9a-b8be-503604a1bac4\" (UID: \"e20ce11b-ff37-4c9a-b8be-503604a1bac4\") " Oct 30 16:38:12.449855 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:12.449645 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e20ce11b-ff37-4c9a-b8be-503604a1bac4-maas-api-openshift-ca" pod "e20ce11b-ff37-4c9a-b8be-503604a1bac4" (UID: "e20ce11b-ff37-4c9a-b8be-503604a1bac4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:38:12.449855 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:12.449816 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d2a02f4-8325-41c6-a341-df1267e99563-maas-api-openshift-ca" pod "8d2a02f4-8325-41c6-a341-df1267e99563" (UID: "8d2a02f4-8325-41c6-a341-df1267e99563"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:38:12.451582 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:12.451555 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d2a02f4-8325-41c6-a341-df1267e99563-maas-api-xks-ca" pod "8d2a02f4-8325-41c6-a341-df1267e99563" (UID: "8d2a02f4-8325-41c6-a341-df1267e99563"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:38:12.451933 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:12.451910 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e20ce11b-ff37-4c9a-b8be-503604a1bac4-maas-api-xks-ca" pod "e20ce11b-ff37-4c9a-b8be-503604a1bac4" (UID: "e20ce11b-ff37-4c9a-b8be-503604a1bac4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:38:12.452022 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:12.451975 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e20ce11b-ff37-4c9a-b8be-503604a1bac4-kube-api-access-p9rws" pod "e20ce11b-ff37-4c9a-b8be-503604a1bac4" (UID: "e20ce11b-ff37-4c9a-b8be-503604a1bac4"). InnerVolumeSpecName "kube-api-access-p9rws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:38:12.453080 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:12.453046 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d2a02f4-8325-41c6-a341-df1267e99563-kube-api-access-qmhdz" pod "8d2a02f4-8325-41c6-a341-df1267e99563" (UID: "8d2a02f4-8325-41c6-a341-df1267e99563"). InnerVolumeSpecName "kube-api-access-qmhdz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:38:12.550546 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:12.550515 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d2a02f4-8325-41c6-a341-df1267e99563-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:38:12.550546 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:12.550546 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9rws\" (UniqueName: \"kubernetes.io/projected/e20ce11b-ff37-4c9a-b8be-503604a1bac4-kube-api-access-p9rws\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:38:12.550743 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:12.550556 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e20ce11b-ff37-4c9a-b8be-503604a1bac4-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:38:12.550743 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:12.550573 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmhdz\" (UniqueName: \"kubernetes.io/projected/8d2a02f4-8325-41c6-a341-df1267e99563-kube-api-access-qmhdz\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:38:12.550743 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:12.550588 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e20ce11b-ff37-4c9a-b8be-503604a1bac4-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:38:12.550743 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:12.550596 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d2a02f4-8325-41c6-a341-df1267e99563-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:38:13.224172 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:13.224087 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22135e92-mzp7l" Oct 30 16:38:13.224172 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:13.224116 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22135e92-mzp7l" event={"ID":"e20ce11b-ff37-4c9a-b8be-503604a1bac4","Type":"ContainerDied","Data":"efa7ae1926662b287ab7148555b824e25db66e646522d2069480d1391c80ad7a"} Oct 30 16:38:13.224391 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:13.224174 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="efa7ae1926662b287ab7148555b824e25db66e646522d2069480d1391c80ad7a" Oct 30 16:38:13.225822 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:13.225771 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e7b30f3-qtsf9" Oct 30 16:38:13.225822 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:13.225778 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e7b30f3-qtsf9" event={"ID":"8d2a02f4-8325-41c6-a341-df1267e99563","Type":"ContainerDied","Data":"3654370883eb07d08b4b97b7f84883d92f63da58e9bfdbb231163b14d646e7f1"} Oct 30 16:38:13.226037 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:13.225844 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3654370883eb07d08b4b97b7f84883d92f63da58e9bfdbb231163b14d646e7f1" Oct 30 16:38:13.463617 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:13.463580 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4ebd091-cj9n9"] Oct 30 16:38:13.464363 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:13.464342 4493 cpu_manager.go:414] "containerMap: removing container" podUID="8d2a02f4-8325-41c6-a341-df1267e99563" containerName="revoke-keys" Oct 30 16:38:13.464466 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:13.464364 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d2a02f4-8325-41c6-a341-df1267e99563" containerName="revoke-keys" Oct 30 16:38:13.464466 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:13.464381 4493 cpu_manager.go:414] "containerMap: removing container" podUID="e20ce11b-ff37-4c9a-b8be-503604a1bac4" containerName="revoke-keys" Oct 30 16:38:13.464466 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:13.464389 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e20ce11b-ff37-4c9a-b8be-503604a1bac4" containerName="revoke-keys" Oct 30 16:38:13.464617 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:13.464557 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d2a02f4-8325-41c6-a341-df1267e99563" containerName="revoke-keys" Oct 30 16:38:13.464617 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:13.464569 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="e20ce11b-ff37-4c9a-b8be-503604a1bac4" containerName="revoke-keys" Oct 30 16:38:13.468323 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:13.468303 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ebd091-cj9n9" Oct 30 16:38:13.483318 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:13.483288 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:38:13.483459 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:13.483387 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:38:13.483459 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:13.483421 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:38:13.483808 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:13.483772 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 16:38:13.490091 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:13.490060 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4ebd091-cj9n9"] Oct 30 16:38:13.562586 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:13.562551 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7379b269-37fa-48c8-bba5-8c3e74a3294c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4ebd091-cj9n9\" (UID: \"7379b269-37fa-48c8-bba5-8c3e74a3294c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ebd091-cj9n9" Oct 30 16:38:13.562751 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:13.562693 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7379b269-37fa-48c8-bba5-8c3e74a3294c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4ebd091-cj9n9\" (UID: \"7379b269-37fa-48c8-bba5-8c3e74a3294c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ebd091-cj9n9" Oct 30 16:38:13.562827 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:13.562816 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nrfs\" (UniqueName: \"kubernetes.io/projected/7379b269-37fa-48c8-bba5-8c3e74a3294c-kube-api-access-7nrfs\") pod \"maas-api-revoke-sub-c4ebd091-cj9n9\" (UID: \"7379b269-37fa-48c8-bba5-8c3e74a3294c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ebd091-cj9n9" Oct 30 16:38:13.664417 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:13.664383 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nrfs\" (UniqueName: \"kubernetes.io/projected/7379b269-37fa-48c8-bba5-8c3e74a3294c-kube-api-access-7nrfs\") pod \"maas-api-revoke-sub-c4ebd091-cj9n9\" (UID: \"7379b269-37fa-48c8-bba5-8c3e74a3294c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ebd091-cj9n9" Oct 30 16:38:13.664620 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:13.664428 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7379b269-37fa-48c8-bba5-8c3e74a3294c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4ebd091-cj9n9\" (UID: \"7379b269-37fa-48c8-bba5-8c3e74a3294c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ebd091-cj9n9" Oct 30 16:38:13.664620 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:13.664522 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7379b269-37fa-48c8-bba5-8c3e74a3294c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4ebd091-cj9n9\" (UID: \"7379b269-37fa-48c8-bba5-8c3e74a3294c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ebd091-cj9n9" Oct 30 16:38:13.665338 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:13.665320 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7379b269-37fa-48c8-bba5-8c3e74a3294c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4ebd091-cj9n9\" (UID: \"7379b269-37fa-48c8-bba5-8c3e74a3294c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ebd091-cj9n9" Oct 30 16:38:13.667071 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:13.667050 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7379b269-37fa-48c8-bba5-8c3e74a3294c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4ebd091-cj9n9\" (UID: \"7379b269-37fa-48c8-bba5-8c3e74a3294c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ebd091-cj9n9" Oct 30 16:38:13.687494 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:13.687469 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nrfs\" (UniqueName: \"kubernetes.io/projected/7379b269-37fa-48c8-bba5-8c3e74a3294c-kube-api-access-7nrfs\") pod \"maas-api-revoke-sub-c4ebd091-cj9n9\" (UID: \"7379b269-37fa-48c8-bba5-8c3e74a3294c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ebd091-cj9n9" Oct 30 16:38:13.784671 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:13.784599 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ebd091-cj9n9" Oct 30 16:38:14.099438 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:14.099405 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4ebd091-cj9n9"] Oct 30 16:38:14.099742 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:38:14.099716 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7379b269_37fa_48c8_bba5_8c3e74a3294c.slice/crio-c27aaa1250263b30c935b61795545966b28caaa9df8f8796c2254703e5bdb676 WatchSource:0}: Error finding container c27aaa1250263b30c935b61795545966b28caaa9df8f8796c2254703e5bdb676: Status 404 returned error can't find the container with id c27aaa1250263b30c935b61795545966b28caaa9df8f8796c2254703e5bdb676 Oct 30 16:38:14.234409 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:14.234377 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ebd091-cj9n9" event={"ID":"7379b269-37fa-48c8-bba5-8c3e74a3294c","Type":"ContainerStarted","Data":"c27aaa1250263b30c935b61795545966b28caaa9df8f8796c2254703e5bdb676"} Oct 30 16:38:15.246127 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:15.246091 4493 generic.go:356] "Generic (PLEG): container finished" podID="7379b269-37fa-48c8-bba5-8c3e74a3294c" containerID="e983a58cedcd385a4dd4de4b69aba1caca3afd90c3b18bf30a6979567b3bec39" exitCode=0 Oct 30 16:38:15.246647 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:15.246177 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ebd091-cj9n9" event={"ID":"7379b269-37fa-48c8-bba5-8c3e74a3294c","Type":"ContainerDied","Data":"e983a58cedcd385a4dd4de4b69aba1caca3afd90c3b18bf30a6979567b3bec39"} Oct 30 16:38:16.417591 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:16.417560 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ebd091-cj9n9" Oct 30 16:38:16.502350 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:16.502305 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7379b269-37fa-48c8-bba5-8c3e74a3294c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7379b269-37fa-48c8-bba5-8c3e74a3294c-maas-api-xks-ca\") pod \"7379b269-37fa-48c8-bba5-8c3e74a3294c\" (UID: \"7379b269-37fa-48c8-bba5-8c3e74a3294c\") " Oct 30 16:38:16.502539 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:16.502483 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7379b269-37fa-48c8-bba5-8c3e74a3294c-kube-api-access-7nrfs\" (UniqueName: \"kubernetes.io/projected/7379b269-37fa-48c8-bba5-8c3e74a3294c-kube-api-access-7nrfs\") pod \"7379b269-37fa-48c8-bba5-8c3e74a3294c\" (UID: \"7379b269-37fa-48c8-bba5-8c3e74a3294c\") " Oct 30 16:38:16.502619 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:16.502552 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7379b269-37fa-48c8-bba5-8c3e74a3294c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7379b269-37fa-48c8-bba5-8c3e74a3294c-maas-api-openshift-ca\") pod \"7379b269-37fa-48c8-bba5-8c3e74a3294c\" (UID: \"7379b269-37fa-48c8-bba5-8c3e74a3294c\") " Oct 30 16:38:16.503016 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:16.502984 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7379b269-37fa-48c8-bba5-8c3e74a3294c-maas-api-openshift-ca" pod "7379b269-37fa-48c8-bba5-8c3e74a3294c" (UID: "7379b269-37fa-48c8-bba5-8c3e74a3294c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:38:16.505128 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:16.505071 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7379b269-37fa-48c8-bba5-8c3e74a3294c-maas-api-xks-ca" pod "7379b269-37fa-48c8-bba5-8c3e74a3294c" (UID: "7379b269-37fa-48c8-bba5-8c3e74a3294c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:38:16.505225 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:16.505163 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7379b269-37fa-48c8-bba5-8c3e74a3294c-kube-api-access-7nrfs" pod "7379b269-37fa-48c8-bba5-8c3e74a3294c" (UID: "7379b269-37fa-48c8-bba5-8c3e74a3294c"). InnerVolumeSpecName "kube-api-access-7nrfs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:38:16.604865 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:16.604802 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nrfs\" (UniqueName: \"kubernetes.io/projected/7379b269-37fa-48c8-bba5-8c3e74a3294c-kube-api-access-7nrfs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:38:16.604865 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:16.604852 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7379b269-37fa-48c8-bba5-8c3e74a3294c-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:38:16.604865 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:16.604869 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7379b269-37fa-48c8-bba5-8c3e74a3294c-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:38:17.263549 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:17.263469 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ebd091-cj9n9" event={"ID":"7379b269-37fa-48c8-bba5-8c3e74a3294c","Type":"ContainerDied","Data":"c27aaa1250263b30c935b61795545966b28caaa9df8f8796c2254703e5bdb676"} Oct 30 16:38:17.263549 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:17.263528 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c27aaa1250263b30c935b61795545966b28caaa9df8f8796c2254703e5bdb676" Oct 30 16:38:17.263549 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:17.263479 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ebd091-cj9n9" Oct 30 16:38:18.037661 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:18.037622 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr"] Oct 30 16:38:18.039055 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:18.039026 4493 cpu_manager.go:414] "containerMap: removing container" podUID="7379b269-37fa-48c8-bba5-8c3e74a3294c" containerName="revoke-keys" Oct 30 16:38:18.039055 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:18.039053 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7379b269-37fa-48c8-bba5-8c3e74a3294c" containerName="revoke-keys" Oct 30 16:38:18.040236 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:18.040206 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="7379b269-37fa-48c8-bba5-8c3e74a3294c" containerName="revoke-keys" Oct 30 16:38:18.058141 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:18.057422 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr" Oct 30 16:38:18.074253 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:18.074223 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk"] Oct 30 16:38:18.084113 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:18.079254 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk" Oct 30 16:38:18.084113 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:18.079963 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr"] Oct 30 16:38:18.114281 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:18.111895 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk"] Oct 30 16:38:18.126542 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:18.126034 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dbdb8f4-a39a-4880-bc1b-525787aa6120-workload-certs\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr\" (UID: \"3dbdb8f4-a39a-4880-bc1b-525787aa6120\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr" Oct 30 16:38:18.126542 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:18.126093 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zks2h\" (UniqueName: \"kubernetes.io/projected/3dbdb8f4-a39a-4880-bc1b-525787aa6120-kube-api-access-zks2h\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr\" (UID: \"3dbdb8f4-a39a-4880-bc1b-525787aa6120\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr" Oct 30 16:38:18.126542 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:18.126127 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/320f1fb7-deb8-4a82-b468-f92066eca8a1-workload-socket\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk\" (UID: \"320f1fb7-deb8-4a82-b468-f92066eca8a1\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk" Oct 30 16:38:18.126542 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:18.126159 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dbdb8f4-a39a-4880-bc1b-525787aa6120-istio-token\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr\" (UID: \"3dbdb8f4-a39a-4880-bc1b-525787aa6120\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr" Oct 30 16:38:18.126542 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:18.126186 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc8rj\" (UniqueName: \"kubernetes.io/projected/320f1fb7-deb8-4a82-b468-f92066eca8a1-kube-api-access-mc8rj\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk\" (UID: \"320f1fb7-deb8-4a82-b468-f92066eca8a1\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk" Oct 30 16:38:18.126542 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:18.126218 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dbdb8f4-a39a-4880-bc1b-525787aa6120-istio-envoy\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr\" (UID: \"3dbdb8f4-a39a-4880-bc1b-525787aa6120\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr" Oct 30 16:38:18.126542 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:18.126269 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dbdb8f4-a39a-4880-bc1b-525787aa6120-istio-data\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr\" (UID: \"3dbdb8f4-a39a-4880-bc1b-525787aa6120\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr" Oct 30 16:38:18.126542 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:18.126305 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbdb8f4-a39a-4880-bc1b-525787aa6120-workload-socket\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr\" (UID: \"3dbdb8f4-a39a-4880-bc1b-525787aa6120\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr" Oct 30 16:38:18.126542 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:18.126330 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dbdb8f4-a39a-4880-bc1b-525787aa6120-istio-podinfo\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr\" (UID: \"3dbdb8f4-a39a-4880-bc1b-525787aa6120\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr" Oct 30 16:38:18.126542 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:18.126363 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/320f1fb7-deb8-4a82-b468-f92066eca8a1-istio-token\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk\" (UID: \"320f1fb7-deb8-4a82-b468-f92066eca8a1\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk" Oct 30 16:38:18.126542 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:18.126420 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/320f1fb7-deb8-4a82-b468-f92066eca8a1-istio-envoy\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk\" (UID: \"320f1fb7-deb8-4a82-b468-f92066eca8a1\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk" Oct 30 16:38:18.126542 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:18.126503 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/320f1fb7-deb8-4a82-b468-f92066eca8a1-workload-certs\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk\" (UID: \"320f1fb7-deb8-4a82-b468-f92066eca8a1\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk" Oct 30 16:38:18.126542 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:18.126549 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dbdb8f4-a39a-4880-bc1b-525787aa6120-istiod-ca-cert\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr\" (UID: \"3dbdb8f4-a39a-4880-bc1b-525787aa6120\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr" Oct 30 16:38:18.127273 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:18.126657 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/320f1fb7-deb8-4a82-b468-f92066eca8a1-istiod-ca-cert\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk\" (UID: \"320f1fb7-deb8-4a82-b468-f92066eca8a1\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk" Oct 30 16:38:18.127273 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:18.126718 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/320f1fb7-deb8-4a82-b468-f92066eca8a1-istio-podinfo\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk\" (UID: \"320f1fb7-deb8-4a82-b468-f92066eca8a1\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk" Oct 30 16:38:18.127273 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:18.126803 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/320f1fb7-deb8-4a82-b468-f92066eca8a1-istio-data\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk\" (UID: \"320f1fb7-deb8-4a82-b468-f92066eca8a1\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk" Oct 30 16:38:18.127273 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:18.126844 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbdb8f4-a39a-4880-bc1b-525787aa6120-credential-socket\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr\" (UID: \"3dbdb8f4-a39a-4880-bc1b-525787aa6120\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr" Oct 30 16:38:18.127273 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:18.126869 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/320f1fb7-deb8-4a82-b468-f92066eca8a1-credential-socket\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk\" (UID: \"320f1fb7-deb8-4a82-b468-f92066eca8a1\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk" Oct 30 16:38:18.228567 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:18.228534 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dbdb8f4-a39a-4880-bc1b-525787aa6120-istio-data\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr\" (UID: \"3dbdb8f4-a39a-4880-bc1b-525787aa6120\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr" Oct 30 16:38:18.228763 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:18.228587 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbdb8f4-a39a-4880-bc1b-525787aa6120-workload-socket\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr\" (UID: \"3dbdb8f4-a39a-4880-bc1b-525787aa6120\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr" Oct 30 16:38:18.228763 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:18.228616 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dbdb8f4-a39a-4880-bc1b-525787aa6120-istio-podinfo\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr\" (UID: \"3dbdb8f4-a39a-4880-bc1b-525787aa6120\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr" Oct 30 16:38:18.228763 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:18.228647 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/320f1fb7-deb8-4a82-b468-f92066eca8a1-istio-token\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk\" (UID: \"320f1fb7-deb8-4a82-b468-f92066eca8a1\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk" Oct 30 16:38:18.228763 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:18.228697 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/320f1fb7-deb8-4a82-b468-f92066eca8a1-istio-envoy\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk\" (UID: \"320f1fb7-deb8-4a82-b468-f92066eca8a1\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk" Oct 30 16:38:18.229051 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:18.228766 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/320f1fb7-deb8-4a82-b468-f92066eca8a1-workload-certs\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk\" (UID: \"320f1fb7-deb8-4a82-b468-f92066eca8a1\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk" Oct 30 16:38:18.229051 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:18.228829 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dbdb8f4-a39a-4880-bc1b-525787aa6120-istiod-ca-cert\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr\" (UID: \"3dbdb8f4-a39a-4880-bc1b-525787aa6120\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr" Oct 30 16:38:18.229051 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:18.228886 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/320f1fb7-deb8-4a82-b468-f92066eca8a1-istiod-ca-cert\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk\" (UID: \"320f1fb7-deb8-4a82-b468-f92066eca8a1\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk" Oct 30 16:38:18.229051 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:18.228940 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/320f1fb7-deb8-4a82-b468-f92066eca8a1-istio-podinfo\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk\" (UID: \"320f1fb7-deb8-4a82-b468-f92066eca8a1\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk" Oct 30 16:38:18.229051 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:18.228986 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dbdb8f4-a39a-4880-bc1b-525787aa6120-istio-data\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr\" (UID: \"3dbdb8f4-a39a-4880-bc1b-525787aa6120\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr" Oct 30 16:38:18.229051 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:18.228998 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/320f1fb7-deb8-4a82-b468-f92066eca8a1-istio-data\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk\" (UID: \"320f1fb7-deb8-4a82-b468-f92066eca8a1\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk" Oct 30 16:38:18.229051 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:18.229046 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbdb8f4-a39a-4880-bc1b-525787aa6120-credential-socket\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr\" (UID: \"3dbdb8f4-a39a-4880-bc1b-525787aa6120\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr" Oct 30 16:38:18.229051 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:18.229048 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbdb8f4-a39a-4880-bc1b-525787aa6120-workload-socket\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr\" (UID: \"3dbdb8f4-a39a-4880-bc1b-525787aa6120\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr" Oct 30 16:38:18.229458 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:18.229073 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/320f1fb7-deb8-4a82-b468-f92066eca8a1-credential-socket\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk\" (UID: \"320f1fb7-deb8-4a82-b468-f92066eca8a1\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk" Oct 30 16:38:18.229458 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:18.229123 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dbdb8f4-a39a-4880-bc1b-525787aa6120-workload-certs\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr\" (UID: \"3dbdb8f4-a39a-4880-bc1b-525787aa6120\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr" Oct 30 16:38:18.229458 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:18.229147 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zks2h\" (UniqueName: \"kubernetes.io/projected/3dbdb8f4-a39a-4880-bc1b-525787aa6120-kube-api-access-zks2h\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr\" (UID: \"3dbdb8f4-a39a-4880-bc1b-525787aa6120\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr" Oct 30 16:38:18.229458 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:18.229179 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/320f1fb7-deb8-4a82-b468-f92066eca8a1-workload-socket\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk\" (UID: \"320f1fb7-deb8-4a82-b468-f92066eca8a1\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk" Oct 30 16:38:18.229458 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:18.229209 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dbdb8f4-a39a-4880-bc1b-525787aa6120-istio-token\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr\" (UID: \"3dbdb8f4-a39a-4880-bc1b-525787aa6120\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr" Oct 30 16:38:18.229458 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:18.229236 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc8rj\" (UniqueName: \"kubernetes.io/projected/320f1fb7-deb8-4a82-b468-f92066eca8a1-kube-api-access-mc8rj\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk\" (UID: \"320f1fb7-deb8-4a82-b468-f92066eca8a1\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk" Oct 30 16:38:18.229458 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:18.229268 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dbdb8f4-a39a-4880-bc1b-525787aa6120-istio-envoy\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr\" (UID: \"3dbdb8f4-a39a-4880-bc1b-525787aa6120\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr" Oct 30 16:38:18.229458 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:18.229389 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/320f1fb7-deb8-4a82-b468-f92066eca8a1-istio-data\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk\" (UID: \"320f1fb7-deb8-4a82-b468-f92066eca8a1\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk" Oct 30 16:38:18.229458 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:18.229431 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/320f1fb7-deb8-4a82-b468-f92066eca8a1-workload-certs\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk\" (UID: \"320f1fb7-deb8-4a82-b468-f92066eca8a1\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk" Oct 30 16:38:18.229942 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:18.229674 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dbdb8f4-a39a-4880-bc1b-525787aa6120-workload-certs\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr\" (UID: \"3dbdb8f4-a39a-4880-bc1b-525787aa6120\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr" Oct 30 16:38:18.229942 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:18.229724 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/320f1fb7-deb8-4a82-b468-f92066eca8a1-istiod-ca-cert\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk\" (UID: \"320f1fb7-deb8-4a82-b468-f92066eca8a1\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk" Oct 30 16:38:18.230064 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:18.229970 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/320f1fb7-deb8-4a82-b468-f92066eca8a1-workload-socket\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk\" (UID: \"320f1fb7-deb8-4a82-b468-f92066eca8a1\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk" Oct 30 16:38:18.230487 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:18.230298 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dbdb8f4-a39a-4880-bc1b-525787aa6120-istiod-ca-cert\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr\" (UID: \"3dbdb8f4-a39a-4880-bc1b-525787aa6120\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr" Oct 30 16:38:18.230487 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:18.230404 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbdb8f4-a39a-4880-bc1b-525787aa6120-credential-socket\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr\" (UID: \"3dbdb8f4-a39a-4880-bc1b-525787aa6120\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr" Oct 30 16:38:18.230487 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:18.230426 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/320f1fb7-deb8-4a82-b468-f92066eca8a1-credential-socket\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk\" (UID: \"320f1fb7-deb8-4a82-b468-f92066eca8a1\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk" Oct 30 16:38:18.232145 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:18.232120 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/320f1fb7-deb8-4a82-b468-f92066eca8a1-istio-podinfo\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk\" (UID: \"320f1fb7-deb8-4a82-b468-f92066eca8a1\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk" Oct 30 16:38:18.232262 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:18.232197 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dbdb8f4-a39a-4880-bc1b-525787aa6120-istio-envoy\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr\" (UID: \"3dbdb8f4-a39a-4880-bc1b-525787aa6120\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr" Oct 30 16:38:18.232551 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:18.232533 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dbdb8f4-a39a-4880-bc1b-525787aa6120-istio-podinfo\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr\" (UID: \"3dbdb8f4-a39a-4880-bc1b-525787aa6120\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr" Oct 30 16:38:18.232631 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:18.232593 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/320f1fb7-deb8-4a82-b468-f92066eca8a1-istio-envoy\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk\" (UID: \"320f1fb7-deb8-4a82-b468-f92066eca8a1\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk" Oct 30 16:38:18.251700 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:18.251658 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/320f1fb7-deb8-4a82-b468-f92066eca8a1-istio-token\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk\" (UID: \"320f1fb7-deb8-4a82-b468-f92066eca8a1\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk" Oct 30 16:38:18.252906 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:18.252881 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zks2h\" (UniqueName: \"kubernetes.io/projected/3dbdb8f4-a39a-4880-bc1b-525787aa6120-kube-api-access-zks2h\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr\" (UID: \"3dbdb8f4-a39a-4880-bc1b-525787aa6120\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr" Oct 30 16:38:18.253213 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:18.253194 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dbdb8f4-a39a-4880-bc1b-525787aa6120-istio-token\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr\" (UID: \"3dbdb8f4-a39a-4880-bc1b-525787aa6120\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr" Oct 30 16:38:18.253330 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:18.253307 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc8rj\" (UniqueName: \"kubernetes.io/projected/320f1fb7-deb8-4a82-b468-f92066eca8a1-kube-api-access-mc8rj\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk\" (UID: \"320f1fb7-deb8-4a82-b468-f92066eca8a1\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk" Oct 30 16:38:18.373925 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:18.373831 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr" Oct 30 16:38:18.412670 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:18.412635 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk" Oct 30 16:38:18.701308 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:38:18.701275 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dbdb8f4_a39a_4880_bc1b_525787aa6120.slice/crio-bd11401bdfc77bf4189d8a1b36de0e88abde5e672b0fae35031a9c55e3af4375 WatchSource:0}: Error finding container bd11401bdfc77bf4189d8a1b36de0e88abde5e672b0fae35031a9c55e3af4375: Status 404 returned error can't find the container with id bd11401bdfc77bf4189d8a1b36de0e88abde5e672b0fae35031a9c55e3af4375 Oct 30 16:38:18.701704 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:18.701683 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr"] Oct 30 16:38:18.704062 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:18.704021 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:38:18.704163 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:18.704105 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:38:18.704163 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:18.704145 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:38:18.746932 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:18.746908 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk"] Oct 30 16:38:18.748468 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:38:18.748426 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod320f1fb7_deb8_4a82_b468_f92066eca8a1.slice/crio-e42145e35cd0f23618ba288b6817f69341fa221ac529d72a6ae4c59e0bef9b4e WatchSource:0}: Error finding container e42145e35cd0f23618ba288b6817f69341fa221ac529d72a6ae4c59e0bef9b4e: Status 404 returned error can't find the container with id e42145e35cd0f23618ba288b6817f69341fa221ac529d72a6ae4c59e0bef9b4e Oct 30 16:38:18.751116 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:18.751078 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:38:18.751206 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:18.751185 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:38:18.751242 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:18.751229 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:38:19.281584 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:19.281550 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk" event={"ID":"320f1fb7-deb8-4a82-b468-f92066eca8a1","Type":"ContainerStarted","Data":"24a87beb129a10f854bcadc0f61d76be963d76dbd791cb56e7a80316215ae361"} Oct 30 16:38:19.282037 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:19.281603 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk" event={"ID":"320f1fb7-deb8-4a82-b468-f92066eca8a1","Type":"ContainerStarted","Data":"e42145e35cd0f23618ba288b6817f69341fa221ac529d72a6ae4c59e0bef9b4e"} Oct 30 16:38:19.283404 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:19.283382 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr" event={"ID":"3dbdb8f4-a39a-4880-bc1b-525787aa6120","Type":"ContainerStarted","Data":"b7eb285df4bf73fa42f5e66c470871bf662853edc34e327416c81862cd367993"} Oct 30 16:38:19.283519 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:19.283425 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr" event={"ID":"3dbdb8f4-a39a-4880-bc1b-525787aa6120","Type":"ContainerStarted","Data":"bd11401bdfc77bf4189d8a1b36de0e88abde5e672b0fae35031a9c55e3af4375"} Oct 30 16:38:19.344606 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:19.344543 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk" podStartSLOduration=1.344522647 podStartE2EDuration="1.344522647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:38:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:38:19.339999164 +0000 UTC m=+926.567174265" watchObservedRunningTime="2026-10-30 16:38:19.344522647 +0000 UTC m=+926.571697747" Oct 30 16:38:19.374605 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:19.374578 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr" Oct 30 16:38:19.376303 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:19.376277 4493 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 30 16:38:19.376419 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:19.376322 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr" podUID="3dbdb8f4-a39a-4880-bc1b-525787aa6120" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 30 16:38:19.387970 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:19.387924 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr" podStartSLOduration=1.387909969 podStartE2EDuration="1.387909969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:38:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:38:19.384537523 +0000 UTC m=+926.611712625" watchObservedRunningTime="2026-10-30 16:38:19.387909969 +0000 UTC m=+926.615085069" Oct 30 16:38:19.413368 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:19.413338 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk" Oct 30 16:38:19.414934 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:19.414908 4493 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 30 16:38:19.415031 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:19.414953 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk" podUID="320f1fb7-deb8-4a82-b468-f92066eca8a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 30 16:38:20.377293 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:20.377252 4493 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 30 16:38:20.377907 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:20.377876 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr" podUID="3dbdb8f4-a39a-4880-bc1b-525787aa6120" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 30 16:38:20.414262 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:20.414222 4493 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 30 16:38:20.414574 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:20.414546 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk" podUID="320f1fb7-deb8-4a82-b468-f92066eca8a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 30 16:38:21.375809 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:21.375273 4493 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 30 16:38:21.375809 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:21.375357 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr" podUID="3dbdb8f4-a39a-4880-bc1b-525787aa6120" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 30 16:38:21.428824 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:21.413258 4493 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 30 16:38:21.428824 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:21.413319 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk" podUID="320f1fb7-deb8-4a82-b468-f92066eca8a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 30 16:38:22.381812 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:22.378987 4493 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 30 16:38:22.381812 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:22.379058 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr" podUID="3dbdb8f4-a39a-4880-bc1b-525787aa6120" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 30 16:38:22.416810 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:22.416537 4493 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 30 16:38:22.416810 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:22.416612 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk" podUID="320f1fb7-deb8-4a82-b468-f92066eca8a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 30 16:38:22.586809 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:22.585882 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" podUID="e43738f3-d55f-4c0a-a364-bf2ac76605c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:22.633509 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:22.633378 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" podUID="14805045-a727-41ba-8bd0-100d41f9f27d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 16:38:22.634771 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:22.634722 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" podUID="0eadc5d4-98c3-4bf2-b3c2-f73fe249182f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:22.711993 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:22.711945 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" podUID="e76172b1-659d-45bd-91c6-1a0ce4062335" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:22.823030 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:22.821883 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" podUID="08baad08-cd72-4305-b92d-c11a84ef276f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:22.870093 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:22.869323 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" podUID="80b9bd9a-30d6-45ff-b318-26fcdc608e5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:22.943070 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:22.943015 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" podUID="18b72648-f523-405e-b5cd-1266ff13f8f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:23.181103 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:23.181042 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" podUID="f3e1c8ff-fcc3-4a1f-9730-c617d57b5193" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:23.397257 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:23.397222 4493 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 30 16:38:23.397620 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:23.397560 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr" podUID="3dbdb8f4-a39a-4880-bc1b-525787aa6120" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 30 16:38:23.423296 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:23.423261 4493 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 30 16:38:23.423618 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:23.423565 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk" podUID="320f1fb7-deb8-4a82-b468-f92066eca8a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 30 16:38:23.497989 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:23.497925 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" podUID="e90f43d3-f5eb-45ab-af1e-340d0ce18966" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:23.586536 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:23.586496 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" podUID="e43738f3-d55f-4c0a-a364-bf2ac76605c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:23.587492 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:23.587470 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" Oct 30 16:38:23.635907 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:23.635866 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" podUID="0eadc5d4-98c3-4bf2-b3c2-f73fe249182f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:23.636449 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:23.636423 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" podUID="14805045-a727-41ba-8bd0-100d41f9f27d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 16:38:23.637324 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:23.637302 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" Oct 30 16:38:23.730874 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:23.727202 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" Oct 30 16:38:23.730874 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:23.718599 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" podUID="e76172b1-659d-45bd-91c6-1a0ce4062335" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:23.756822 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:23.756338 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" Oct 30 16:38:23.853945 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:23.853905 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" podUID="08baad08-cd72-4305-b92d-c11a84ef276f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:23.855032 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:23.855008 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" Oct 30 16:38:23.875856 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:23.874882 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" podUID="80b9bd9a-30d6-45ff-b318-26fcdc608e5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:23.877011 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:23.876349 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" Oct 30 16:38:23.961670 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:23.957958 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" podUID="18b72648-f523-405e-b5cd-1266ff13f8f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 30 16:38:23.961670 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:23.958848 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" Oct 30 16:38:24.190826 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:24.190762 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" podUID="f3e1c8ff-fcc3-4a1f-9730-c617d57b5193" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded" Oct 30 16:38:24.217830 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:24.192183 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" Oct 30 16:38:24.409729 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:24.409680 4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" podUID="14805045-a727-41ba-8bd0-100d41f9f27d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": context deadline exceeded" Oct 30 16:38:24.466952 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:24.466124 4493 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 30 16:38:24.467218 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:24.467187 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr" podUID="3dbdb8f4-a39a-4880-bc1b-525787aa6120" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 30 16:38:24.467369 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:24.466917 4493 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 30 16:38:24.467523 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:24.467505 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk" podUID="320f1fb7-deb8-4a82-b468-f92066eca8a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 30 16:38:24.503813 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:24.502959 4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" podUID="b5abe703-fee0-4185-aa22-88d1db06392b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": context deadline exceeded" Oct 30 16:38:24.503813 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:24.503112 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" podUID="e90f43d3-f5eb-45ab-af1e-340d0ce18966" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:24.519808 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:24.504546 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" Oct 30 16:38:24.596811 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:24.549664 4493 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:38:24.596811 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:24.549724 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br" podUID="b30c2fa0-a3d3-4942-9596-bb6b245a238f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:24.596811 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:24.580407 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" podUID="b5abe703-fee0-4185-aa22-88d1db06392b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:24.596811 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:24.591887 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" podUID="e43738f3-d55f-4c0a-a364-bf2ac76605c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:24.659860 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:24.659056 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" podUID="14805045-a727-41ba-8bd0-100d41f9f27d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:24.660825 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:24.660107 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" podUID="0eadc5d4-98c3-4bf2-b3c2-f73fe249182f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:24.736363 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:24.736265 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" podUID="e76172b1-659d-45bd-91c6-1a0ce4062335" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 16:38:24.862984 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:24.862940 4493 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:38:24.863248 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:24.863204 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9" podUID="b08f59e7-615a-4d20-ba53-ad4bcf0d5728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:24.863481 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:24.863460 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" podUID="08baad08-cd72-4305-b92d-c11a84ef276f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 16:38:24.875906 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:24.875860 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" podUID="80b9bd9a-30d6-45ff-b318-26fcdc608e5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 16:38:24.925810 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:24.917859 4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" podUID="e43738f3-d55f-4c0a-a364-bf2ac76605c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:25.039917 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:25.024689 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" podUID="18b72648-f523-405e-b5cd-1266ff13f8f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:25.203110 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:25.203068 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" podUID="f3e1c8ff-fcc3-4a1f-9730-c617d57b5193" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:25.406873 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:25.376913 4493 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 30 16:38:25.406873 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:25.376975 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr" podUID="3dbdb8f4-a39a-4880-bc1b-525787aa6120" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 30 16:38:25.426811 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:25.413744 4493 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 30 16:38:25.426811 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:25.413823 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk" podUID="320f1fb7-deb8-4a82-b468-f92066eca8a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 30 16:38:25.503930 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:25.503882 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" podUID="e90f43d3-f5eb-45ab-af1e-340d0ce18966" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:25.580992 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:25.580930 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" podUID="b5abe703-fee0-4185-aa22-88d1db06392b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Oct 30 16:38:25.583661 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:25.582156 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" Oct 30 16:38:25.595943 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:25.595870 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" podUID="e43738f3-d55f-4c0a-a364-bf2ac76605c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:25.661765 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:25.659897 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" podUID="14805045-a727-41ba-8bd0-100d41f9f27d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:25.661765 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:25.661025 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" podUID="0eadc5d4-98c3-4bf2-b3c2-f73fe249182f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:25.736929 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:25.736880 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" podUID="e76172b1-659d-45bd-91c6-1a0ce4062335" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:25.876938 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:25.876889 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" podUID="80b9bd9a-30d6-45ff-b318-26fcdc608e5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:25.904012 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:25.903970 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" podUID="08baad08-cd72-4305-b92d-c11a84ef276f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 16:38:26.029187 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:26.027484 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" podUID="18b72648-f523-405e-b5cd-1266ff13f8f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:26.155223 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:26.154873 4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" podUID="e76172b1-659d-45bd-91c6-1a0ce4062335" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:26.210066 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:26.208935 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" podUID="f3e1c8ff-fcc3-4a1f-9730-c617d57b5193" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:26.375691 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:26.375155 4493 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 30 16:38:26.375691 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:26.375221 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr" podUID="3dbdb8f4-a39a-4880-bc1b-525787aa6120" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 30 16:38:26.380202 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:26.380105 4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" podUID="e90f43d3-f5eb-45ab-af1e-340d0ce18966" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:26.419405 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:26.419291 4493 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 30 16:38:26.419405 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:26.419348 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk" podUID="320f1fb7-deb8-4a82-b468-f92066eca8a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 30 16:38:26.520197 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:26.520142 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" podUID="e90f43d3-f5eb-45ab-af1e-340d0ce18966" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:26.583944 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:26.583895 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" podUID="b5abe703-fee0-4185-aa22-88d1db06392b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:26.607947 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:26.607890 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" podUID="e43738f3-d55f-4c0a-a364-bf2ac76605c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:26.692841 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:26.677410 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" podUID="14805045-a727-41ba-8bd0-100d41f9f27d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 16:38:26.694824 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:26.693437 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" podUID="0eadc5d4-98c3-4bf2-b3c2-f73fe249182f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:26.743101 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:26.742408 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" podUID="e76172b1-659d-45bd-91c6-1a0ce4062335" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 16:38:26.893832 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:26.892590 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" podUID="80b9bd9a-30d6-45ff-b318-26fcdc608e5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 16:38:26.923819 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:26.923166 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" podUID="08baad08-cd72-4305-b92d-c11a84ef276f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:27.036889 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:27.032369 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" podUID="18b72648-f523-405e-b5cd-1266ff13f8f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:27.089317 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:27.088923 4493 patch_prober.go:29] interesting pod/e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:38:27.089317 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:27.088991 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm" podUID="0a035fea-221e-4b14-aa1f-68bd02b90794" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:27.267808 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:27.264590 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" podUID="f3e1c8ff-fcc3-4a1f-9730-c617d57b5193" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:27.393905 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:27.390276 4493 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 30 16:38:27.393905 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:27.390334 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr" podUID="3dbdb8f4-a39a-4880-bc1b-525787aa6120" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 30 16:38:27.415835 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:27.415801 4493 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 30 16:38:27.440916 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:27.440864 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk" podUID="320f1fb7-deb8-4a82-b468-f92066eca8a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 30 16:38:27.580978 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:27.580930 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" podUID="e90f43d3-f5eb-45ab-af1e-340d0ce18966" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:27.586017 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:27.585969 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" podUID="b5abe703-fee0-4185-aa22-88d1db06392b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:27.636811 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:27.622099 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" podUID="e43738f3-d55f-4c0a-a364-bf2ac76605c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:27.698867 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:27.695232 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" podUID="14805045-a727-41ba-8bd0-100d41f9f27d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 16:38:27.698867 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:27.697233 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" podUID="0eadc5d4-98c3-4bf2-b3c2-f73fe249182f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:27.699491 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:27.699469 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" Oct 30 16:38:27.701551 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:27.701512 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" Oct 30 16:38:27.702712 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:27.702692 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" Oct 30 16:38:27.746053 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:27.746003 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" podUID="e76172b1-659d-45bd-91c6-1a0ce4062335" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:27.984894 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:27.952373 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" Oct 30 16:38:27.984894 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:27.953640 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" Oct 30 16:38:27.984894 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:27.963438 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" Oct 30 16:38:27.991079 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:27.991052 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" Oct 30 16:38:27.995941 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:27.995915 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" Oct 30 16:38:28.087624 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:28.087394 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" Oct 30 16:38:28.106821 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:28.098769 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" Oct 30 16:38:28.381553 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:28.381467 4493 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 30 16:38:28.381769 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:28.381742 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr" podUID="3dbdb8f4-a39a-4880-bc1b-525787aa6120" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 30 16:38:28.477737 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:28.477702 4493 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 30 16:38:28.478039 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:28.478002 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk" podUID="320f1fb7-deb8-4a82-b468-f92066eca8a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 30 16:38:29.374431 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:29.374390 4493 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 30 16:38:29.374981 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:29.374458 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr" podUID="3dbdb8f4-a39a-4880-bc1b-525787aa6120" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 30 16:38:29.413243 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:29.413211 4493 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 30 16:38:29.413436 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:29.413275 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk" podUID="320f1fb7-deb8-4a82-b468-f92066eca8a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 30 16:38:29.495751 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:29.495710 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c87577a2-b4vtg"] Oct 30 16:38:29.505156 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:29.505127 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87577a2-b4vtg" Oct 30 16:38:29.526475 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:29.526448 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c87577a2-b4vtg"] Oct 30 16:38:29.530804 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:29.529900 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:38:29.530804 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:29.530048 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 16:38:29.530804 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:29.530122 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:38:29.534537 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:29.533289 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:38:29.670015 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:29.669970 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68135857-e1be-4c00-bfd2-aa3efcd8ce22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c87577a2-b4vtg\" (UID: \"68135857-e1be-4c00-bfd2-aa3efcd8ce22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87577a2-b4vtg" Oct 30 16:38:29.670191 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:29.670064 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68135857-e1be-4c00-bfd2-aa3efcd8ce22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c87577a2-b4vtg\" (UID: \"68135857-e1be-4c00-bfd2-aa3efcd8ce22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87577a2-b4vtg" Oct 30 16:38:29.670191 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:29.670168 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfrrm\" (UniqueName: \"kubernetes.io/projected/68135857-e1be-4c00-bfd2-aa3efcd8ce22-kube-api-access-zfrrm\") pod \"maas-api-revoke-sub-c87577a2-b4vtg\" (UID: \"68135857-e1be-4c00-bfd2-aa3efcd8ce22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87577a2-b4vtg" Oct 30 16:38:29.772116 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:29.772082 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68135857-e1be-4c00-bfd2-aa3efcd8ce22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c87577a2-b4vtg\" (UID: \"68135857-e1be-4c00-bfd2-aa3efcd8ce22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87577a2-b4vtg" Oct 30 16:38:29.772421 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:29.772399 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68135857-e1be-4c00-bfd2-aa3efcd8ce22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c87577a2-b4vtg\" (UID: \"68135857-e1be-4c00-bfd2-aa3efcd8ce22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87577a2-b4vtg" Oct 30 16:38:29.772635 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:29.772619 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfrrm\" (UniqueName: \"kubernetes.io/projected/68135857-e1be-4c00-bfd2-aa3efcd8ce22-kube-api-access-zfrrm\") pod \"maas-api-revoke-sub-c87577a2-b4vtg\" (UID: \"68135857-e1be-4c00-bfd2-aa3efcd8ce22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87577a2-b4vtg" Oct 30 16:38:29.773253 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:29.773226 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68135857-e1be-4c00-bfd2-aa3efcd8ce22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c87577a2-b4vtg\" (UID: \"68135857-e1be-4c00-bfd2-aa3efcd8ce22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87577a2-b4vtg" Oct 30 16:38:29.776083 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:29.775711 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68135857-e1be-4c00-bfd2-aa3efcd8ce22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c87577a2-b4vtg\" (UID: \"68135857-e1be-4c00-bfd2-aa3efcd8ce22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87577a2-b4vtg" Oct 30 16:38:29.813826 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:29.807047 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfrrm\" (UniqueName: \"kubernetes.io/projected/68135857-e1be-4c00-bfd2-aa3efcd8ce22-kube-api-access-zfrrm\") pod \"maas-api-revoke-sub-c87577a2-b4vtg\" (UID: \"68135857-e1be-4c00-bfd2-aa3efcd8ce22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87577a2-b4vtg" Oct 30 16:38:29.836123 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:29.836092 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87577a2-b4vtg" Oct 30 16:38:30.201540 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:30.201502 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c87577a2-b4vtg"] Oct 30 16:38:30.204122 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:38:30.204094 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68135857_e1be_4c00_bfd2_aa3efcd8ce22.slice/crio-1fa722a722fa352f5f10937a20c17f0ccd0c2366d4c8624632553285bee3adbc WatchSource:0}: Error finding container 1fa722a722fa352f5f10937a20c17f0ccd0c2366d4c8624632553285bee3adbc: Status 404 returned error can't find the container with id 1fa722a722fa352f5f10937a20c17f0ccd0c2366d4c8624632553285bee3adbc Oct 30 16:38:30.281828 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:30.281245 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ecbd336-sw9fp"] Oct 30 16:38:30.291845 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:30.288397 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ecbd336-sw9fp" Oct 30 16:38:30.328287 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:30.324209 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ecbd336-sw9fp"] Oct 30 16:38:30.389467 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:30.389427 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f88a9011-5708-4dbf-a1dd-24d44c6bde2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ecbd336-sw9fp\" (UID: \"f88a9011-5708-4dbf-a1dd-24d44c6bde2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ecbd336-sw9fp" Oct 30 16:38:30.390046 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:30.389486 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f88a9011-5708-4dbf-a1dd-24d44c6bde2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ecbd336-sw9fp\" (UID: \"f88a9011-5708-4dbf-a1dd-24d44c6bde2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ecbd336-sw9fp" Oct 30 16:38:30.390046 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:30.389565 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmnth\" (UniqueName: \"kubernetes.io/projected/f88a9011-5708-4dbf-a1dd-24d44c6bde2c-kube-api-access-wmnth\") pod \"maas-api-revoke-sub-0ecbd336-sw9fp\" (UID: \"f88a9011-5708-4dbf-a1dd-24d44c6bde2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ecbd336-sw9fp" Oct 30 16:38:30.448321 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:38:30.448293 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 16:38:30.490937 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:30.490850 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f88a9011-5708-4dbf-a1dd-24d44c6bde2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ecbd336-sw9fp\" (UID: \"f88a9011-5708-4dbf-a1dd-24d44c6bde2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ecbd336-sw9fp" Oct 30 16:38:30.490937 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:30.490895 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f88a9011-5708-4dbf-a1dd-24d44c6bde2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ecbd336-sw9fp\" (UID: \"f88a9011-5708-4dbf-a1dd-24d44c6bde2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ecbd336-sw9fp" Oct 30 16:38:30.491163 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:30.490962 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmnth\" (UniqueName: \"kubernetes.io/projected/f88a9011-5708-4dbf-a1dd-24d44c6bde2c-kube-api-access-wmnth\") pod \"maas-api-revoke-sub-0ecbd336-sw9fp\" (UID: \"f88a9011-5708-4dbf-a1dd-24d44c6bde2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ecbd336-sw9fp" Oct 30 16:38:30.491694 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:30.491666 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f88a9011-5708-4dbf-a1dd-24d44c6bde2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ecbd336-sw9fp\" (UID: \"f88a9011-5708-4dbf-a1dd-24d44c6bde2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ecbd336-sw9fp" Oct 30 16:38:30.493855 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:30.493832 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f88a9011-5708-4dbf-a1dd-24d44c6bde2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ecbd336-sw9fp\" (UID: \"f88a9011-5708-4dbf-a1dd-24d44c6bde2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ecbd336-sw9fp" Oct 30 16:38:30.517088 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:30.517057 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmnth\" (UniqueName: \"kubernetes.io/projected/f88a9011-5708-4dbf-a1dd-24d44c6bde2c-kube-api-access-wmnth\") pod \"maas-api-revoke-sub-0ecbd336-sw9fp\" (UID: \"f88a9011-5708-4dbf-a1dd-24d44c6bde2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ecbd336-sw9fp" Oct 30 16:38:30.566223 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:30.565148 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk" Oct 30 16:38:30.566223 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:30.566142 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk" Oct 30 16:38:30.567045 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:30.567021 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk" Oct 30 16:38:30.607886 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:30.607862 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ecbd336-sw9fp" Oct 30 16:38:30.842944 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:30.842841 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87577a2-b4vtg" event={"ID":"68135857-e1be-4c00-bfd2-aa3efcd8ce22","Type":"ContainerStarted","Data":"4497fbd4cbf4941d63f0a1d26c0bccebcb22b9e07fa2df5213ba8f13fe17ad8e"} Oct 30 16:38:30.842944 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:30.842896 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87577a2-b4vtg" event={"ID":"68135857-e1be-4c00-bfd2-aa3efcd8ce22","Type":"ContainerStarted","Data":"1fa722a722fa352f5f10937a20c17f0ccd0c2366d4c8624632553285bee3adbc"} Oct 30 16:38:30.886723 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:30.886675 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87577a2-b4vtg" podStartSLOduration=1.8866598190000001 podStartE2EDuration="1.886659819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:38:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:38:30.88338795 +0000 UTC m=+938.110563049" watchObservedRunningTime="2026-10-30 16:38:30.886659819 +0000 UTC m=+938.113834921" Oct 30 16:38:30.893894 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:30.893872 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr" Oct 30 16:38:30.895003 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:30.894552 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr" Oct 30 16:38:30.895776 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:30.895756 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr" Oct 30 16:38:31.000906 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:31.000865 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ecbd336-sw9fp"] Oct 30 16:38:31.001701 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:38:31.001671 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf88a9011_5708_4dbf_a1dd_24d44c6bde2c.slice/crio-18651354ef926329bfb6ec52867a435f56c9d6bfa40edfd514dc909435b42dc4 WatchSource:0}: Error finding container 18651354ef926329bfb6ec52867a435f56c9d6bfa40edfd514dc909435b42dc4: Status 404 returned error can't find the container with id 18651354ef926329bfb6ec52867a435f56c9d6bfa40edfd514dc909435b42dc4 Oct 30 16:38:31.852277 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:31.852241 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ecbd336-sw9fp" event={"ID":"f88a9011-5708-4dbf-a1dd-24d44c6bde2c","Type":"ContainerStarted","Data":"cd3c54eee9c1aec1c308fc1afd9f11de148d725d3a7d2cc352b5d27568e612d6"} Oct 30 16:38:31.852684 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:31.852302 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ecbd336-sw9fp" event={"ID":"f88a9011-5708-4dbf-a1dd-24d44c6bde2c","Type":"ContainerStarted","Data":"18651354ef926329bfb6ec52867a435f56c9d6bfa40edfd514dc909435b42dc4"} Oct 30 16:38:31.854185 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:31.854155 4493 generic.go:356] "Generic (PLEG): container finished" podID="68135857-e1be-4c00-bfd2-aa3efcd8ce22" containerID="4497fbd4cbf4941d63f0a1d26c0bccebcb22b9e07fa2df5213ba8f13fe17ad8e" exitCode=0 Oct 30 16:38:31.854294 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:31.854213 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87577a2-b4vtg" event={"ID":"68135857-e1be-4c00-bfd2-aa3efcd8ce22","Type":"ContainerDied","Data":"4497fbd4cbf4941d63f0a1d26c0bccebcb22b9e07fa2df5213ba8f13fe17ad8e"} Oct 30 16:38:31.893655 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:31.893629 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd5d978f-rwghq"] Oct 30 16:38:31.906464 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:31.906427 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd5d978f-rwghq" Oct 30 16:38:31.917356 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:31.916471 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ecbd336-sw9fp" podStartSLOduration=1.9164541339999999 podStartE2EDuration="1.916454134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:38:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:38:31.905296921 +0000 UTC m=+939.132472023" watchObservedRunningTime="2026-10-30 16:38:31.916454134 +0000 UTC m=+939.143629239" Oct 30 16:38:31.920962 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:31.920937 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd5d978f-rwghq"] Oct 30 16:38:32.027005 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:32.026878 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scgmk\" (UniqueName: \"kubernetes.io/projected/38b708d1-f21e-4197-88d1-63d3e85f9475-kube-api-access-scgmk\") pod \"maas-api-revoke-sub-fd5d978f-rwghq\" (UID: \"38b708d1-f21e-4197-88d1-63d3e85f9475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd5d978f-rwghq" Oct 30 16:38:32.027005 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:32.026958 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38b708d1-f21e-4197-88d1-63d3e85f9475-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd5d978f-rwghq\" (UID: \"38b708d1-f21e-4197-88d1-63d3e85f9475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd5d978f-rwghq" Oct 30 16:38:32.027340 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:32.027312 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38b708d1-f21e-4197-88d1-63d3e85f9475-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd5d978f-rwghq\" (UID: \"38b708d1-f21e-4197-88d1-63d3e85f9475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd5d978f-rwghq" Oct 30 16:38:32.128910 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:32.128829 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scgmk\" (UniqueName: \"kubernetes.io/projected/38b708d1-f21e-4197-88d1-63d3e85f9475-kube-api-access-scgmk\") pod \"maas-api-revoke-sub-fd5d978f-rwghq\" (UID: \"38b708d1-f21e-4197-88d1-63d3e85f9475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd5d978f-rwghq" Oct 30 16:38:32.128910 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:32.128883 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38b708d1-f21e-4197-88d1-63d3e85f9475-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd5d978f-rwghq\" (UID: \"38b708d1-f21e-4197-88d1-63d3e85f9475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd5d978f-rwghq" Oct 30 16:38:32.129108 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:32.128987 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38b708d1-f21e-4197-88d1-63d3e85f9475-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd5d978f-rwghq\" (UID: \"38b708d1-f21e-4197-88d1-63d3e85f9475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd5d978f-rwghq" Oct 30 16:38:32.129741 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:32.129669 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38b708d1-f21e-4197-88d1-63d3e85f9475-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd5d978f-rwghq\" (UID: \"38b708d1-f21e-4197-88d1-63d3e85f9475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd5d978f-rwghq" Oct 30 16:38:32.131623 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:32.131599 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38b708d1-f21e-4197-88d1-63d3e85f9475-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd5d978f-rwghq\" (UID: \"38b708d1-f21e-4197-88d1-63d3e85f9475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd5d978f-rwghq" Oct 30 16:38:32.152980 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:32.152960 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scgmk\" (UniqueName: \"kubernetes.io/projected/38b708d1-f21e-4197-88d1-63d3e85f9475-kube-api-access-scgmk\") pod \"maas-api-revoke-sub-fd5d978f-rwghq\" (UID: \"38b708d1-f21e-4197-88d1-63d3e85f9475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd5d978f-rwghq" Oct 30 16:38:32.223937 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:32.223913 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd5d978f-rwghq" Oct 30 16:38:32.543829 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:32.543777 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd5d978f-rwghq"] Oct 30 16:38:32.864968 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:32.864925 4493 generic.go:356] "Generic (PLEG): container finished" podID="f88a9011-5708-4dbf-a1dd-24d44c6bde2c" containerID="cd3c54eee9c1aec1c308fc1afd9f11de148d725d3a7d2cc352b5d27568e612d6" exitCode=0 Oct 30 16:38:32.865398 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:32.864981 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ecbd336-sw9fp" event={"ID":"f88a9011-5708-4dbf-a1dd-24d44c6bde2c","Type":"ContainerDied","Data":"cd3c54eee9c1aec1c308fc1afd9f11de148d725d3a7d2cc352b5d27568e612d6"} Oct 30 16:38:32.866666 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:32.866634 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd5d978f-rwghq" event={"ID":"38b708d1-f21e-4197-88d1-63d3e85f9475","Type":"ContainerStarted","Data":"4ba37e0ec5a3b82d4f53380912ec1f5f0c8cb16bb142e110e7e9099627592184"} Oct 30 16:38:33.054557 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:33.054530 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87577a2-b4vtg" Oct 30 16:38:33.146927 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:33.146899 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68135857-e1be-4c00-bfd2-aa3efcd8ce22-kube-api-access-zfrrm\" (UniqueName: \"kubernetes.io/projected/68135857-e1be-4c00-bfd2-aa3efcd8ce22-kube-api-access-zfrrm\") pod \"68135857-e1be-4c00-bfd2-aa3efcd8ce22\" (UID: \"68135857-e1be-4c00-bfd2-aa3efcd8ce22\") " Oct 30 16:38:33.147097 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:33.146960 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68135857-e1be-4c00-bfd2-aa3efcd8ce22-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68135857-e1be-4c00-bfd2-aa3efcd8ce22-maas-api-openshift-ca\") pod \"68135857-e1be-4c00-bfd2-aa3efcd8ce22\" (UID: \"68135857-e1be-4c00-bfd2-aa3efcd8ce22\") " Oct 30 16:38:33.147180 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:33.147122 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68135857-e1be-4c00-bfd2-aa3efcd8ce22-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68135857-e1be-4c00-bfd2-aa3efcd8ce22-maas-api-xks-ca\") pod \"68135857-e1be-4c00-bfd2-aa3efcd8ce22\" (UID: \"68135857-e1be-4c00-bfd2-aa3efcd8ce22\") " Oct 30 16:38:33.147398 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:33.147372 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68135857-e1be-4c00-bfd2-aa3efcd8ce22-maas-api-openshift-ca" pod "68135857-e1be-4c00-bfd2-aa3efcd8ce22" (UID: "68135857-e1be-4c00-bfd2-aa3efcd8ce22"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:38:33.148122 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:33.148066 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68135857-e1be-4c00-bfd2-aa3efcd8ce22-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:38:33.149438 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:33.149410 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68135857-e1be-4c00-bfd2-aa3efcd8ce22-kube-api-access-zfrrm" pod "68135857-e1be-4c00-bfd2-aa3efcd8ce22" (UID: "68135857-e1be-4c00-bfd2-aa3efcd8ce22"). InnerVolumeSpecName "kube-api-access-zfrrm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:38:33.149541 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:33.149525 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68135857-e1be-4c00-bfd2-aa3efcd8ce22-maas-api-xks-ca" pod "68135857-e1be-4c00-bfd2-aa3efcd8ce22" (UID: "68135857-e1be-4c00-bfd2-aa3efcd8ce22"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:38:33.250222 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:33.250184 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfrrm\" (UniqueName: \"kubernetes.io/projected/68135857-e1be-4c00-bfd2-aa3efcd8ce22-kube-api-access-zfrrm\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:38:33.250222 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:33.250226 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68135857-e1be-4c00-bfd2-aa3efcd8ce22-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:38:33.876916 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:33.876833 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87577a2-b4vtg" event={"ID":"68135857-e1be-4c00-bfd2-aa3efcd8ce22","Type":"ContainerDied","Data":"1fa722a722fa352f5f10937a20c17f0ccd0c2366d4c8624632553285bee3adbc"} Oct 30 16:38:33.877322 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:33.876925 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1fa722a722fa352f5f10937a20c17f0ccd0c2366d4c8624632553285bee3adbc" Oct 30 16:38:33.877322 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:33.876848 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c87577a2-b4vtg" Oct 30 16:38:33.878734 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:33.878703 4493 generic.go:356] "Generic (PLEG): container finished" podID="38b708d1-f21e-4197-88d1-63d3e85f9475" containerID="7421cb9f73f21cb023685a964d32ecc058ea7382123b473817912dbb5ca38a13" exitCode=0 Oct 30 16:38:33.878897 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:33.878767 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd5d978f-rwghq" event={"ID":"38b708d1-f21e-4197-88d1-63d3e85f9475","Type":"ContainerDied","Data":"7421cb9f73f21cb023685a964d32ecc058ea7382123b473817912dbb5ca38a13"} Oct 30 16:38:34.013182 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:34.013156 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ecbd336-sw9fp" Oct 30 16:38:34.162440 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:34.162414 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f88a9011-5708-4dbf-a1dd-24d44c6bde2c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f88a9011-5708-4dbf-a1dd-24d44c6bde2c-maas-api-openshift-ca\") pod \"f88a9011-5708-4dbf-a1dd-24d44c6bde2c\" (UID: \"f88a9011-5708-4dbf-a1dd-24d44c6bde2c\") " Oct 30 16:38:34.162586 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:34.162495 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f88a9011-5708-4dbf-a1dd-24d44c6bde2c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f88a9011-5708-4dbf-a1dd-24d44c6bde2c-maas-api-xks-ca\") pod \"f88a9011-5708-4dbf-a1dd-24d44c6bde2c\" (UID: \"f88a9011-5708-4dbf-a1dd-24d44c6bde2c\") " Oct 30 16:38:34.162656 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:34.162601 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f88a9011-5708-4dbf-a1dd-24d44c6bde2c-kube-api-access-wmnth\" (UniqueName: \"kubernetes.io/projected/f88a9011-5708-4dbf-a1dd-24d44c6bde2c-kube-api-access-wmnth\") pod \"f88a9011-5708-4dbf-a1dd-24d44c6bde2c\" (UID: \"f88a9011-5708-4dbf-a1dd-24d44c6bde2c\") " Oct 30 16:38:34.162879 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:34.162850 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f88a9011-5708-4dbf-a1dd-24d44c6bde2c-maas-api-openshift-ca" pod "f88a9011-5708-4dbf-a1dd-24d44c6bde2c" (UID: "f88a9011-5708-4dbf-a1dd-24d44c6bde2c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:38:34.163091 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:34.163074 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f88a9011-5708-4dbf-a1dd-24d44c6bde2c-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:38:34.164746 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:34.164720 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f88a9011-5708-4dbf-a1dd-24d44c6bde2c-maas-api-xks-ca" pod "f88a9011-5708-4dbf-a1dd-24d44c6bde2c" (UID: "f88a9011-5708-4dbf-a1dd-24d44c6bde2c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:38:34.164848 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:34.164774 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f88a9011-5708-4dbf-a1dd-24d44c6bde2c-kube-api-access-wmnth" pod "f88a9011-5708-4dbf-a1dd-24d44c6bde2c" (UID: "f88a9011-5708-4dbf-a1dd-24d44c6bde2c"). InnerVolumeSpecName "kube-api-access-wmnth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:38:34.263987 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:34.263959 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f88a9011-5708-4dbf-a1dd-24d44c6bde2c-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:38:34.263987 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:34.263985 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmnth\" (UniqueName: \"kubernetes.io/projected/f88a9011-5708-4dbf-a1dd-24d44c6bde2c-kube-api-access-wmnth\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:38:34.887413 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:34.887378 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ecbd336-sw9fp" Oct 30 16:38:34.887842 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:34.887432 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ecbd336-sw9fp" event={"ID":"f88a9011-5708-4dbf-a1dd-24d44c6bde2c","Type":"ContainerDied","Data":"18651354ef926329bfb6ec52867a435f56c9d6bfa40edfd514dc909435b42dc4"} Oct 30 16:38:34.887842 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:34.887490 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18651354ef926329bfb6ec52867a435f56c9d6bfa40edfd514dc909435b42dc4" Oct 30 16:38:35.072177 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:35.072147 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd5d978f-rwghq" Oct 30 16:38:35.175674 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:35.175647 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38b708d1-f21e-4197-88d1-63d3e85f9475-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38b708d1-f21e-4197-88d1-63d3e85f9475-maas-api-openshift-ca\") pod \"38b708d1-f21e-4197-88d1-63d3e85f9475\" (UID: \"38b708d1-f21e-4197-88d1-63d3e85f9475\") " Oct 30 16:38:35.175862 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:35.175741 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38b708d1-f21e-4197-88d1-63d3e85f9475-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38b708d1-f21e-4197-88d1-63d3e85f9475-maas-api-xks-ca\") pod \"38b708d1-f21e-4197-88d1-63d3e85f9475\" (UID: \"38b708d1-f21e-4197-88d1-63d3e85f9475\") " Oct 30 16:38:35.175862 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:35.175774 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38b708d1-f21e-4197-88d1-63d3e85f9475-kube-api-access-scgmk\" (UniqueName: \"kubernetes.io/projected/38b708d1-f21e-4197-88d1-63d3e85f9475-kube-api-access-scgmk\") pod \"38b708d1-f21e-4197-88d1-63d3e85f9475\" (UID: \"38b708d1-f21e-4197-88d1-63d3e85f9475\") " Oct 30 16:38:35.176112 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:35.176079 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38b708d1-f21e-4197-88d1-63d3e85f9475-maas-api-openshift-ca" pod "38b708d1-f21e-4197-88d1-63d3e85f9475" (UID: "38b708d1-f21e-4197-88d1-63d3e85f9475"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:38:35.176363 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:35.176342 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38b708d1-f21e-4197-88d1-63d3e85f9475-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:38:35.177916 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:35.177897 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38b708d1-f21e-4197-88d1-63d3e85f9475-maas-api-xks-ca" pod "38b708d1-f21e-4197-88d1-63d3e85f9475" (UID: "38b708d1-f21e-4197-88d1-63d3e85f9475"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:38:35.178042 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:35.178019 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38b708d1-f21e-4197-88d1-63d3e85f9475-kube-api-access-scgmk" pod "38b708d1-f21e-4197-88d1-63d3e85f9475" (UID: "38b708d1-f21e-4197-88d1-63d3e85f9475"). InnerVolumeSpecName "kube-api-access-scgmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:38:35.278026 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:35.277994 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38b708d1-f21e-4197-88d1-63d3e85f9475-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:38:35.278026 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:35.278020 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scgmk\" (UniqueName: \"kubernetes.io/projected/38b708d1-f21e-4197-88d1-63d3e85f9475-kube-api-access-scgmk\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:38:35.896888 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:35.896847 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd5d978f-rwghq" event={"ID":"38b708d1-f21e-4197-88d1-63d3e85f9475","Type":"ContainerDied","Data":"4ba37e0ec5a3b82d4f53380912ec1f5f0c8cb16bb142e110e7e9099627592184"} Oct 30 16:38:35.897283 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:35.896893 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd5d978f-rwghq" Oct 30 16:38:35.897283 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:35.896907 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ba37e0ec5a3b82d4f53380912ec1f5f0c8cb16bb142e110e7e9099627592184" Oct 30 16:38:36.322282 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:36.319220 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 16:38:38.683811 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:38.679827 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" podUID="14805045-a727-41ba-8bd0-100d41f9f27d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:39.033057 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:39.031886 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" podUID="18b72648-f523-405e-b5cd-1266ff13f8f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:39.207917 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:39.207871 4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" podUID="80b9bd9a-30d6-45ff-b318-26fcdc608e5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:39.228129 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:39.228086 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" podUID="f3e1c8ff-fcc3-4a1f-9730-c617d57b5193" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:39.525469 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:39.524869 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" podUID="e90f43d3-f5eb-45ab-af1e-340d0ce18966" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Oct 30 16:38:39.605897 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:39.604898 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" podUID="b5abe703-fee0-4185-aa22-88d1db06392b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:39.607812 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:39.606094 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" podUID="e43738f3-d55f-4c0a-a364-bf2ac76605c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 16:38:39.692811 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:39.684905 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" podUID="0eadc5d4-98c3-4bf2-b3c2-f73fe249182f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:39.692811 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:39.686312 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" podUID="14805045-a727-41ba-8bd0-100d41f9f27d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:39.692811 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:39.686663 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" Oct 30 16:38:39.748595 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:39.747888 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" podUID="e76172b1-659d-45bd-91c6-1a0ce4062335" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:39.940841 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:39.936874 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" podUID="80b9bd9a-30d6-45ff-b318-26fcdc608e5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:40.002809 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:39.994036 4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" podUID="f3e1c8ff-fcc3-4a1f-9730-c617d57b5193" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/health\": context deadline exceeded" Oct 30 16:38:40.060863 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:40.032626 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" podUID="18b72648-f523-405e-b5cd-1266ff13f8f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:40.060863 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:40.033078 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" Oct 30 16:38:40.285558 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:40.250990 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" podUID="f3e1c8ff-fcc3-4a1f-9730-c617d57b5193" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded" Oct 30 16:38:40.289746 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:40.286982 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" Oct 30 16:38:40.537380 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:40.537283 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" podUID="e90f43d3-f5eb-45ab-af1e-340d0ce18966" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:40.542848 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:40.539019 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" Oct 30 16:38:40.639812 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:40.614184 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" podUID="e43738f3-d55f-4c0a-a364-bf2ac76605c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:40.642420 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:40.642382 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" podUID="b5abe703-fee0-4185-aa22-88d1db06392b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Oct 30 16:38:40.643948 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:40.643929 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" Oct 30 16:38:40.644137 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:40.644122 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" Oct 30 16:38:40.696509 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:40.696465 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" podUID="14805045-a727-41ba-8bd0-100d41f9f27d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:40.699502 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:40.699469 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" podUID="0eadc5d4-98c3-4bf2-b3c2-f73fe249182f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:40.700681 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:40.700663 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" Oct 30 16:38:40.750060 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:40.750021 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" podUID="e76172b1-659d-45bd-91c6-1a0ce4062335" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:40.751769 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:40.751745 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" Oct 30 16:38:40.777079 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:40.777049 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" Oct 30 16:38:40.795817 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:40.795716 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc09e051-k8p8h"] Oct 30 16:38:40.797659 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:40.797632 4493 cpu_manager.go:414] "containerMap: removing container" podUID="68135857-e1be-4c00-bfd2-aa3efcd8ce22" containerName="revoke-keys" Oct 30 16:38:40.798387 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:40.798368 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68135857-e1be-4c00-bfd2-aa3efcd8ce22" containerName="revoke-keys" Oct 30 16:38:40.799100 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:40.799082 4493 cpu_manager.go:414] "containerMap: removing container" podUID="f88a9011-5708-4dbf-a1dd-24d44c6bde2c" containerName="revoke-keys" Oct 30 16:38:40.799237 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:40.799223 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f88a9011-5708-4dbf-a1dd-24d44c6bde2c" containerName="revoke-keys" Oct 30 16:38:40.802859 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:40.802841 4493 cpu_manager.go:414] "containerMap: removing container" podUID="38b708d1-f21e-4197-88d1-63d3e85f9475" containerName="revoke-keys" Oct 30 16:38:40.802991 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:40.802976 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38b708d1-f21e-4197-88d1-63d3e85f9475" containerName="revoke-keys" Oct 30 16:38:40.803372 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:40.803350 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="68135857-e1be-4c00-bfd2-aa3efcd8ce22" containerName="revoke-keys" Oct 30 16:38:40.803494 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:40.803481 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="f88a9011-5708-4dbf-a1dd-24d44c6bde2c" containerName="revoke-keys" Oct 30 16:38:40.803589 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:40.803577 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="38b708d1-f21e-4197-88d1-63d3e85f9475" containerName="revoke-keys" Oct 30 16:38:41.035289 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:41.032953 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" podUID="08baad08-cd72-4305-b92d-c11a84ef276f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 16:38:41.121072 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:41.073944 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" podUID="80b9bd9a-30d6-45ff-b318-26fcdc608e5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:41.137584 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:41.137543 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" Oct 30 16:38:41.138062 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:41.138043 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc09e051-k8p8h"] Oct 30 16:38:41.154554 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:41.077268 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" podUID="18b72648-f523-405e-b5cd-1266ff13f8f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 30 16:38:41.351809 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:41.341923 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc09e051-k8p8h" Oct 30 16:38:41.411858 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:41.408551 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:38:41.425498 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:41.425471 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:38:41.425705 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:41.423773 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 16:38:41.467549 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:41.467513 4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" podUID="18b72648-f523-405e-b5cd-1266ff13f8f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:41.479212 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:41.335099 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" podUID="f3e1c8ff-fcc3-4a1f-9730-c617d57b5193" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:41.530948 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:41.527397 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:38:41.543489 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:41.543303 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" Oct 30 16:38:41.543727 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:41.543709 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" Oct 30 16:38:41.590886 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:41.590843 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/457364de-ff92-4a03-a3f8-ae3a3268c487-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc09e051-k8p8h\" (UID: \"457364de-ff92-4a03-a3f8-ae3a3268c487\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc09e051-k8p8h" Oct 30 16:38:41.743157 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:41.743061 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/457364de-ff92-4a03-a3f8-ae3a3268c487-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc09e051-k8p8h\" (UID: \"457364de-ff92-4a03-a3f8-ae3a3268c487\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc09e051-k8p8h" Oct 30 16:38:41.767997 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:41.767964 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs5rf\" (UniqueName: \"kubernetes.io/projected/457364de-ff92-4a03-a3f8-ae3a3268c487-kube-api-access-hs5rf\") pod \"maas-api-revoke-sub-bc09e051-k8p8h\" (UID: \"457364de-ff92-4a03-a3f8-ae3a3268c487\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc09e051-k8p8h" Oct 30 16:38:41.768556 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:41.715818 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" podUID="e43738f3-d55f-4c0a-a364-bf2ac76605c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 16:38:41.769035 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:41.715837 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" podUID="b5abe703-fee0-4185-aa22-88d1db06392b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Oct 30 16:38:41.779802 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:41.714508 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" podUID="0eadc5d4-98c3-4bf2-b3c2-f73fe249182f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:41.779802 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:41.715767 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" podUID="14805045-a727-41ba-8bd0-100d41f9f27d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:41.872282 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:41.872250 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/457364de-ff92-4a03-a3f8-ae3a3268c487-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc09e051-k8p8h\" (UID: \"457364de-ff92-4a03-a3f8-ae3a3268c487\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc09e051-k8p8h" Oct 30 16:38:41.872719 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:41.872696 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/457364de-ff92-4a03-a3f8-ae3a3268c487-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc09e051-k8p8h\" (UID: \"457364de-ff92-4a03-a3f8-ae3a3268c487\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc09e051-k8p8h" Oct 30 16:38:41.872902 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:41.872884 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs5rf\" (UniqueName: \"kubernetes.io/projected/457364de-ff92-4a03-a3f8-ae3a3268c487-kube-api-access-hs5rf\") pod \"maas-api-revoke-sub-bc09e051-k8p8h\" (UID: \"457364de-ff92-4a03-a3f8-ae3a3268c487\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc09e051-k8p8h" Oct 30 16:38:41.874803 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:41.874763 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/457364de-ff92-4a03-a3f8-ae3a3268c487-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc09e051-k8p8h\" (UID: \"457364de-ff92-4a03-a3f8-ae3a3268c487\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc09e051-k8p8h" Oct 30 16:38:41.887237 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:41.887213 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" Oct 30 16:38:41.981857 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:41.976229 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/457364de-ff92-4a03-a3f8-ae3a3268c487-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc09e051-k8p8h\" (UID: \"457364de-ff92-4a03-a3f8-ae3a3268c487\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc09e051-k8p8h" Oct 30 16:38:42.017074 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:41.997920 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" Oct 30 16:38:42.021019 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:42.018208 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs5rf\" (UniqueName: \"kubernetes.io/projected/457364de-ff92-4a03-a3f8-ae3a3268c487-kube-api-access-hs5rf\") pod \"maas-api-revoke-sub-bc09e051-k8p8h\" (UID: \"457364de-ff92-4a03-a3f8-ae3a3268c487\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc09e051-k8p8h" Oct 30 16:38:42.028341 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:42.028302 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc09e051-k8p8h" Oct 30 16:38:42.186811 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:42.168765 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" podUID="18b72648-f523-405e-b5cd-1266ff13f8f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:42.186811 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:42.169139 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" podUID="08baad08-cd72-4305-b92d-c11a84ef276f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 16:38:42.186811 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:42.182754 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" Oct 30 16:38:42.196807 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:42.196396 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" Oct 30 16:38:42.240764 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:42.240737 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" Oct 30 16:38:42.334094 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:42.325619 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" Oct 30 16:38:42.459208 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:42.459176 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" Oct 30 16:38:42.535567 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:42.535540 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" Oct 30 16:38:42.971658 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:38:42.971623 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod457364de_ff92_4a03_a3f8_ae3a3268c487.slice/crio-6f5db33de47657c6cbdc0e029a5557e7c24b910ee6eaf9ce488d8e35669b5f41 WatchSource:0}: Error finding container 6f5db33de47657c6cbdc0e029a5557e7c24b910ee6eaf9ce488d8e35669b5f41: Status 404 returned error can't find the container with id 6f5db33de47657c6cbdc0e029a5557e7c24b910ee6eaf9ce488d8e35669b5f41 Oct 30 16:38:42.974181 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:42.974153 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc09e051-k8p8h"] Oct 30 16:38:43.786318 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:43.786223 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc09e051-k8p8h" event={"ID":"457364de-ff92-4a03-a3f8-ae3a3268c487","Type":"ContainerStarted","Data":"b4a0668b314d34526b46e7c7c8fd2a90709862187e7a2576275ed8977e8de0db"} Oct 30 16:38:43.786504 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:43.786318 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc09e051-k8p8h" event={"ID":"457364de-ff92-4a03-a3f8-ae3a3268c487","Type":"ContainerStarted","Data":"6f5db33de47657c6cbdc0e029a5557e7c24b910ee6eaf9ce488d8e35669b5f41"} Oct 30 16:38:43.829337 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:43.829280 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc09e051-k8p8h" podStartSLOduration=3.829267501 podStartE2EDuration="3.829267501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:38:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:38:43.827865537 +0000 UTC m=+951.055040637" watchObservedRunningTime="2026-10-30 16:38:43.829267501 +0000 UTC m=+951.056442601" Oct 30 16:38:44.834813 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:44.827015 4493 generic.go:356] "Generic (PLEG): container finished" podID="457364de-ff92-4a03-a3f8-ae3a3268c487" containerID="b4a0668b314d34526b46e7c7c8fd2a90709862187e7a2576275ed8977e8de0db" exitCode=0 Oct 30 16:38:44.834813 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:44.827093 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc09e051-k8p8h" event={"ID":"457364de-ff92-4a03-a3f8-ae3a3268c487","Type":"ContainerDied","Data":"b4a0668b314d34526b46e7c7c8fd2a90709862187e7a2576275ed8977e8de0db"} Oct 30 16:38:46.012368 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:46.012336 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc09e051-k8p8h" Oct 30 16:38:46.064915 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:46.064831 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/457364de-ff92-4a03-a3f8-ae3a3268c487-kube-api-access-hs5rf\" (UniqueName: \"kubernetes.io/projected/457364de-ff92-4a03-a3f8-ae3a3268c487-kube-api-access-hs5rf\") pod \"457364de-ff92-4a03-a3f8-ae3a3268c487\" (UID: \"457364de-ff92-4a03-a3f8-ae3a3268c487\") " Oct 30 16:38:46.064915 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:46.064871 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/457364de-ff92-4a03-a3f8-ae3a3268c487-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/457364de-ff92-4a03-a3f8-ae3a3268c487-maas-api-openshift-ca\") pod \"457364de-ff92-4a03-a3f8-ae3a3268c487\" (UID: \"457364de-ff92-4a03-a3f8-ae3a3268c487\") " Oct 30 16:38:46.065143 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:46.064998 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/457364de-ff92-4a03-a3f8-ae3a3268c487-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/457364de-ff92-4a03-a3f8-ae3a3268c487-maas-api-xks-ca\") pod \"457364de-ff92-4a03-a3f8-ae3a3268c487\" (UID: \"457364de-ff92-4a03-a3f8-ae3a3268c487\") " Oct 30 16:38:46.065292 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:46.065263 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/457364de-ff92-4a03-a3f8-ae3a3268c487-maas-api-openshift-ca" pod "457364de-ff92-4a03-a3f8-ae3a3268c487" (UID: "457364de-ff92-4a03-a3f8-ae3a3268c487"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:38:46.065882 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:46.065858 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/457364de-ff92-4a03-a3f8-ae3a3268c487-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:38:46.067361 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:46.067333 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/457364de-ff92-4a03-a3f8-ae3a3268c487-maas-api-xks-ca" pod "457364de-ff92-4a03-a3f8-ae3a3268c487" (UID: "457364de-ff92-4a03-a3f8-ae3a3268c487"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:38:46.067696 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:46.067662 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/457364de-ff92-4a03-a3f8-ae3a3268c487-kube-api-access-hs5rf" pod "457364de-ff92-4a03-a3f8-ae3a3268c487" (UID: "457364de-ff92-4a03-a3f8-ae3a3268c487"). InnerVolumeSpecName "kube-api-access-hs5rf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:38:46.166951 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:46.166918 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hs5rf\" (UniqueName: \"kubernetes.io/projected/457364de-ff92-4a03-a3f8-ae3a3268c487-kube-api-access-hs5rf\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:38:46.166951 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:46.166950 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/457364de-ff92-4a03-a3f8-ae3a3268c487-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:38:46.848152 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:46.848114 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc09e051-k8p8h" Oct 30 16:38:46.848324 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:46.848123 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc09e051-k8p8h" event={"ID":"457364de-ff92-4a03-a3f8-ae3a3268c487","Type":"ContainerDied","Data":"6f5db33de47657c6cbdc0e029a5557e7c24b910ee6eaf9ce488d8e35669b5f41"} Oct 30 16:38:46.848324 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:46.848236 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f5db33de47657c6cbdc0e029a5557e7c24b910ee6eaf9ce488d8e35669b5f41" Oct 30 16:38:50.207076 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:50.207045 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb8efadc-cpd4t"] Oct 30 16:38:50.207629 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:50.207615 4493 cpu_manager.go:414] "containerMap: removing container" podUID="457364de-ff92-4a03-a3f8-ae3a3268c487" containerName="revoke-keys" Oct 30 16:38:50.207674 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:50.207630 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="457364de-ff92-4a03-a3f8-ae3a3268c487" containerName="revoke-keys" Oct 30 16:38:50.207712 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:50.207708 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="457364de-ff92-4a03-a3f8-ae3a3268c487" containerName="revoke-keys" Oct 30 16:38:50.211446 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:50.211430 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb8efadc-cpd4t" Oct 30 16:38:50.232044 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:50.231997 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:38:50.243762 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:50.243738 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:38:50.243925 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:50.243901 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 16:38:50.243992 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:50.243978 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:38:50.252344 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:50.252320 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb8efadc-cpd4t"] Oct 30 16:38:50.322645 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:50.322601 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pfqs\" (UniqueName: \"kubernetes.io/projected/1b9353ef-0d40-4b4e-b398-2c97692001bf-kube-api-access-9pfqs\") pod \"maas-api-revoke-sub-cb8efadc-cpd4t\" (UID: \"1b9353ef-0d40-4b4e-b398-2c97692001bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb8efadc-cpd4t" Oct 30 16:38:50.322645 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:50.322640 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b9353ef-0d40-4b4e-b398-2c97692001bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb8efadc-cpd4t\" (UID: \"1b9353ef-0d40-4b4e-b398-2c97692001bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb8efadc-cpd4t" Oct 30 16:38:50.322908 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:50.322816 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b9353ef-0d40-4b4e-b398-2c97692001bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb8efadc-cpd4t\" (UID: \"1b9353ef-0d40-4b4e-b398-2c97692001bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb8efadc-cpd4t" Oct 30 16:38:50.424018 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:50.423977 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pfqs\" (UniqueName: \"kubernetes.io/projected/1b9353ef-0d40-4b4e-b398-2c97692001bf-kube-api-access-9pfqs\") pod \"maas-api-revoke-sub-cb8efadc-cpd4t\" (UID: \"1b9353ef-0d40-4b4e-b398-2c97692001bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb8efadc-cpd4t" Oct 30 16:38:50.424018 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:50.424020 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b9353ef-0d40-4b4e-b398-2c97692001bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb8efadc-cpd4t\" (UID: \"1b9353ef-0d40-4b4e-b398-2c97692001bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb8efadc-cpd4t" Oct 30 16:38:50.424275 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:50.424212 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b9353ef-0d40-4b4e-b398-2c97692001bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb8efadc-cpd4t\" (UID: \"1b9353ef-0d40-4b4e-b398-2c97692001bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb8efadc-cpd4t" Oct 30 16:38:50.424863 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:50.424841 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b9353ef-0d40-4b4e-b398-2c97692001bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb8efadc-cpd4t\" (UID: \"1b9353ef-0d40-4b4e-b398-2c97692001bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb8efadc-cpd4t" Oct 30 16:38:50.426628 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:50.426600 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b9353ef-0d40-4b4e-b398-2c97692001bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb8efadc-cpd4t\" (UID: \"1b9353ef-0d40-4b4e-b398-2c97692001bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb8efadc-cpd4t" Oct 30 16:38:50.447461 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:50.447433 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pfqs\" (UniqueName: \"kubernetes.io/projected/1b9353ef-0d40-4b4e-b398-2c97692001bf-kube-api-access-9pfqs\") pod \"maas-api-revoke-sub-cb8efadc-cpd4t\" (UID: \"1b9353ef-0d40-4b4e-b398-2c97692001bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb8efadc-cpd4t" Oct 30 16:38:50.524900 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:50.524826 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb8efadc-cpd4t" Oct 30 16:38:50.831941 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:38:50.831909 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b9353ef_0d40_4b4e_b398_2c97692001bf.slice/crio-d2e1df7cfbac813f4ad3c83a08ffaaaa0da95a922facf5041ee010efcbe93405 WatchSource:0}: Error finding container d2e1df7cfbac813f4ad3c83a08ffaaaa0da95a922facf5041ee010efcbe93405: Status 404 returned error can't find the container with id d2e1df7cfbac813f4ad3c83a08ffaaaa0da95a922facf5041ee010efcbe93405 Oct 30 16:38:50.832079 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:50.831956 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb8efadc-cpd4t"] Oct 30 16:38:50.878848 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:50.878810 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb8efadc-cpd4t" event={"ID":"1b9353ef-0d40-4b4e-b398-2c97692001bf","Type":"ContainerStarted","Data":"d2e1df7cfbac813f4ad3c83a08ffaaaa0da95a922facf5041ee010efcbe93405"} Oct 30 16:38:51.722401 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:51.722375 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-696f6503-n2chf"] Oct 30 16:38:51.726549 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:51.726528 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-696f6503-n2chf" Oct 30 16:38:51.749731 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:51.749704 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-696f6503-n2chf"] Oct 30 16:38:51.844639 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:51.844603 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b379edd5-f6eb-4c61-9b59-1f9a36f0959d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-696f6503-n2chf\" (UID: \"b379edd5-f6eb-4c61-9b59-1f9a36f0959d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-696f6503-n2chf" Oct 30 16:38:51.844639 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:51.844642 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxxpp\" (UniqueName: \"kubernetes.io/projected/b379edd5-f6eb-4c61-9b59-1f9a36f0959d-kube-api-access-lxxpp\") pod \"maas-api-revoke-sub-696f6503-n2chf\" (UID: \"b379edd5-f6eb-4c61-9b59-1f9a36f0959d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-696f6503-n2chf" Oct 30 16:38:51.844884 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:51.844847 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b379edd5-f6eb-4c61-9b59-1f9a36f0959d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-696f6503-n2chf\" (UID: \"b379edd5-f6eb-4c61-9b59-1f9a36f0959d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-696f6503-n2chf" Oct 30 16:38:51.889353 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:51.889318 4493 generic.go:356] "Generic (PLEG): container finished" podID="1b9353ef-0d40-4b4e-b398-2c97692001bf" containerID="14ef523e96acac85ecf93a7e53fdade8930f2230fe5774d4f6cacb767e86f78b" exitCode=0 Oct 30 16:38:51.889531 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:51.889399 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb8efadc-cpd4t" event={"ID":"1b9353ef-0d40-4b4e-b398-2c97692001bf","Type":"ContainerDied","Data":"14ef523e96acac85ecf93a7e53fdade8930f2230fe5774d4f6cacb767e86f78b"} Oct 30 16:38:51.945977 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:51.945940 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b379edd5-f6eb-4c61-9b59-1f9a36f0959d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-696f6503-n2chf\" (UID: \"b379edd5-f6eb-4c61-9b59-1f9a36f0959d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-696f6503-n2chf" Oct 30 16:38:51.946134 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:51.945980 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxxpp\" (UniqueName: \"kubernetes.io/projected/b379edd5-f6eb-4c61-9b59-1f9a36f0959d-kube-api-access-lxxpp\") pod \"maas-api-revoke-sub-696f6503-n2chf\" (UID: \"b379edd5-f6eb-4c61-9b59-1f9a36f0959d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-696f6503-n2chf" Oct 30 16:38:51.946134 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:51.946069 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b379edd5-f6eb-4c61-9b59-1f9a36f0959d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-696f6503-n2chf\" (UID: \"b379edd5-f6eb-4c61-9b59-1f9a36f0959d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-696f6503-n2chf" Oct 30 16:38:51.946900 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:51.946735 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b379edd5-f6eb-4c61-9b59-1f9a36f0959d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-696f6503-n2chf\" (UID: \"b379edd5-f6eb-4c61-9b59-1f9a36f0959d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-696f6503-n2chf" Oct 30 16:38:51.948460 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:51.948440 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b379edd5-f6eb-4c61-9b59-1f9a36f0959d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-696f6503-n2chf\" (UID: \"b379edd5-f6eb-4c61-9b59-1f9a36f0959d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-696f6503-n2chf" Oct 30 16:38:51.969349 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:51.969296 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxxpp\" (UniqueName: \"kubernetes.io/projected/b379edd5-f6eb-4c61-9b59-1f9a36f0959d-kube-api-access-lxxpp\") pod \"maas-api-revoke-sub-696f6503-n2chf\" (UID: \"b379edd5-f6eb-4c61-9b59-1f9a36f0959d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-696f6503-n2chf" Oct 30 16:38:52.094215 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:52.094184 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-696f6503-n2chf" Oct 30 16:38:52.400208 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:38:52.400181 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb379edd5_f6eb_4c61_9b59_1f9a36f0959d.slice/crio-fb9593077c414a62f4a3f816c5bf593d0e7cff775d53a6523d1d490785cfd23d WatchSource:0}: Error finding container fb9593077c414a62f4a3f816c5bf593d0e7cff775d53a6523d1d490785cfd23d: Status 404 returned error can't find the container with id fb9593077c414a62f4a3f816c5bf593d0e7cff775d53a6523d1d490785cfd23d Oct 30 16:38:52.400517 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:52.400497 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-696f6503-n2chf"] Oct 30 16:38:52.900862 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:52.900822 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-696f6503-n2chf" event={"ID":"b379edd5-f6eb-4c61-9b59-1f9a36f0959d","Type":"ContainerStarted","Data":"55ceeb47a0dc74d42a6a41c072201a12ba10be9f7bbf954d3cbad0e69d0fa47c"} Oct 30 16:38:52.901369 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:52.900889 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-696f6503-n2chf" event={"ID":"b379edd5-f6eb-4c61-9b59-1f9a36f0959d","Type":"ContainerStarted","Data":"fb9593077c414a62f4a3f816c5bf593d0e7cff775d53a6523d1d490785cfd23d"} Oct 30 16:38:52.941855 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:52.941764 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-696f6503-n2chf" podStartSLOduration=1.941748971 podStartE2EDuration="1.941748971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:38:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:38:52.940768311 +0000 UTC m=+960.167943412" watchObservedRunningTime="2026-10-30 16:38:52.941748971 +0000 UTC m=+960.168924070" Oct 30 16:38:53.080655 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:53.080627 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb8efadc-cpd4t" Oct 30 16:38:53.164351 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:53.164321 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b9353ef-0d40-4b4e-b398-2c97692001bf-kube-api-access-9pfqs\" (UniqueName: \"kubernetes.io/projected/1b9353ef-0d40-4b4e-b398-2c97692001bf-kube-api-access-9pfqs\") pod \"1b9353ef-0d40-4b4e-b398-2c97692001bf\" (UID: \"1b9353ef-0d40-4b4e-b398-2c97692001bf\") " Oct 30 16:38:53.164519 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:53.164396 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b9353ef-0d40-4b4e-b398-2c97692001bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b9353ef-0d40-4b4e-b398-2c97692001bf-maas-api-xks-ca\") pod \"1b9353ef-0d40-4b4e-b398-2c97692001bf\" (UID: \"1b9353ef-0d40-4b4e-b398-2c97692001bf\") " Oct 30 16:38:53.164519 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:53.164488 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b9353ef-0d40-4b4e-b398-2c97692001bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b9353ef-0d40-4b4e-b398-2c97692001bf-maas-api-openshift-ca\") pod \"1b9353ef-0d40-4b4e-b398-2c97692001bf\" (UID: \"1b9353ef-0d40-4b4e-b398-2c97692001bf\") " Oct 30 16:38:53.164902 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:53.164875 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b9353ef-0d40-4b4e-b398-2c97692001bf-maas-api-openshift-ca" pod "1b9353ef-0d40-4b4e-b398-2c97692001bf" (UID: "1b9353ef-0d40-4b4e-b398-2c97692001bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:38:53.165091 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:53.164994 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b9353ef-0d40-4b4e-b398-2c97692001bf-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:38:53.166730 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:53.166708 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b9353ef-0d40-4b4e-b398-2c97692001bf-kube-api-access-9pfqs" pod "1b9353ef-0d40-4b4e-b398-2c97692001bf" (UID: "1b9353ef-0d40-4b4e-b398-2c97692001bf"). InnerVolumeSpecName "kube-api-access-9pfqs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:38:53.166831 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:53.166777 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b9353ef-0d40-4b4e-b398-2c97692001bf-maas-api-xks-ca" pod "1b9353ef-0d40-4b4e-b398-2c97692001bf" (UID: "1b9353ef-0d40-4b4e-b398-2c97692001bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:38:53.183716 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:53.183691 4493 scope.go:122] "RemoveContainer" containerID="8fd84d82bc6ef4ef3bbe5870c46c3a3fa547e6442e61e50e2d2d8dc9a5c76150" Oct 30 16:38:53.194750 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:53.194734 4493 scope.go:122] "RemoveContainer" containerID="b0ba9d1791f004d2c10ecc3081717a460f7b1ac2ead7104e0fa02513a96c008c" Oct 30 16:38:53.205698 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:53.205679 4493 scope.go:122] "RemoveContainer" containerID="579000dd3e8d320d666c49789abeb9627373b19b2d7eb4ec8c7b21bc381035bd" Oct 30 16:38:53.266727 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:53.266702 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pfqs\" (UniqueName: \"kubernetes.io/projected/1b9353ef-0d40-4b4e-b398-2c97692001bf-kube-api-access-9pfqs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:38:53.266821 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:53.266731 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b9353ef-0d40-4b4e-b398-2c97692001bf-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:38:53.910398 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:53.910375 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb8efadc-cpd4t" Oct 30 16:38:53.910866 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:53.910414 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb8efadc-cpd4t" event={"ID":"1b9353ef-0d40-4b4e-b398-2c97692001bf","Type":"ContainerDied","Data":"d2e1df7cfbac813f4ad3c83a08ffaaaa0da95a922facf5041ee010efcbe93405"} Oct 30 16:38:53.910866 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:53.910503 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2e1df7cfbac813f4ad3c83a08ffaaaa0da95a922facf5041ee010efcbe93405" Oct 30 16:38:53.912441 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:53.912416 4493 generic.go:356] "Generic (PLEG): container finished" podID="b379edd5-f6eb-4c61-9b59-1f9a36f0959d" containerID="55ceeb47a0dc74d42a6a41c072201a12ba10be9f7bbf954d3cbad0e69d0fa47c" exitCode=0 Oct 30 16:38:53.912590 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:53.912468 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-696f6503-n2chf" event={"ID":"b379edd5-f6eb-4c61-9b59-1f9a36f0959d","Type":"ContainerDied","Data":"55ceeb47a0dc74d42a6a41c072201a12ba10be9f7bbf954d3cbad0e69d0fa47c"} Oct 30 16:38:55.086327 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:55.086303 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-696f6503-n2chf" Oct 30 16:38:55.190978 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:55.190900 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b379edd5-f6eb-4c61-9b59-1f9a36f0959d-kube-api-access-lxxpp\" (UniqueName: \"kubernetes.io/projected/b379edd5-f6eb-4c61-9b59-1f9a36f0959d-kube-api-access-lxxpp\") pod \"b379edd5-f6eb-4c61-9b59-1f9a36f0959d\" (UID: \"b379edd5-f6eb-4c61-9b59-1f9a36f0959d\") " Oct 30 16:38:55.191127 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:55.191065 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b379edd5-f6eb-4c61-9b59-1f9a36f0959d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b379edd5-f6eb-4c61-9b59-1f9a36f0959d-maas-api-openshift-ca\") pod \"b379edd5-f6eb-4c61-9b59-1f9a36f0959d\" (UID: \"b379edd5-f6eb-4c61-9b59-1f9a36f0959d\") " Oct 30 16:38:55.191169 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:55.191138 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b379edd5-f6eb-4c61-9b59-1f9a36f0959d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b379edd5-f6eb-4c61-9b59-1f9a36f0959d-maas-api-xks-ca\") pod \"b379edd5-f6eb-4c61-9b59-1f9a36f0959d\" (UID: \"b379edd5-f6eb-4c61-9b59-1f9a36f0959d\") " Oct 30 16:38:55.191442 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:55.191417 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b379edd5-f6eb-4c61-9b59-1f9a36f0959d-maas-api-openshift-ca" pod "b379edd5-f6eb-4c61-9b59-1f9a36f0959d" (UID: "b379edd5-f6eb-4c61-9b59-1f9a36f0959d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:38:55.191620 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:55.191605 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b379edd5-f6eb-4c61-9b59-1f9a36f0959d-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:38:55.193175 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:55.193146 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b379edd5-f6eb-4c61-9b59-1f9a36f0959d-kube-api-access-lxxpp" pod "b379edd5-f6eb-4c61-9b59-1f9a36f0959d" (UID: "b379edd5-f6eb-4c61-9b59-1f9a36f0959d"). InnerVolumeSpecName "kube-api-access-lxxpp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:38:55.193306 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:55.193284 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b379edd5-f6eb-4c61-9b59-1f9a36f0959d-maas-api-xks-ca" pod "b379edd5-f6eb-4c61-9b59-1f9a36f0959d" (UID: "b379edd5-f6eb-4c61-9b59-1f9a36f0959d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:38:55.292773 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:55.292737 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b379edd5-f6eb-4c61-9b59-1f9a36f0959d-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:38:55.292976 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:55.292772 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxxpp\" (UniqueName: \"kubernetes.io/projected/b379edd5-f6eb-4c61-9b59-1f9a36f0959d-kube-api-access-lxxpp\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:38:55.929077 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:55.929047 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-696f6503-n2chf" Oct 30 16:38:55.929077 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:55.929064 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-696f6503-n2chf" event={"ID":"b379edd5-f6eb-4c61-9b59-1f9a36f0959d","Type":"ContainerDied","Data":"fb9593077c414a62f4a3f816c5bf593d0e7cff775d53a6523d1d490785cfd23d"} Oct 30 16:38:55.929287 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:38:55.929098 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb9593077c414a62f4a3f816c5bf593d0e7cff775d53a6523d1d490785cfd23d" Oct 30 16:39:00.342953 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:39:00.342916 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 16:39:02.861811 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:02.859943 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa13dec8-dvsd4"] Oct 30 16:39:02.861811 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:02.860996 4493 cpu_manager.go:414] "containerMap: removing container" podUID="b379edd5-f6eb-4c61-9b59-1f9a36f0959d" containerName="revoke-keys" Oct 30 16:39:02.861811 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:02.861021 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b379edd5-f6eb-4c61-9b59-1f9a36f0959d" containerName="revoke-keys" Oct 30 16:39:02.861811 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:02.861056 4493 cpu_manager.go:414] "containerMap: removing container" podUID="1b9353ef-0d40-4b4e-b398-2c97692001bf" containerName="revoke-keys" Oct 30 16:39:02.861811 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:02.861064 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b9353ef-0d40-4b4e-b398-2c97692001bf" containerName="revoke-keys" Oct 30 16:39:02.861811 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:02.861198 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="b379edd5-f6eb-4c61-9b59-1f9a36f0959d" containerName="revoke-keys" Oct 30 16:39:02.861811 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:02.861212 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b9353ef-0d40-4b4e-b398-2c97692001bf" containerName="revoke-keys" Oct 30 16:39:02.893309 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:02.888083 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa13dec8-dvsd4" Oct 30 16:39:02.918327 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:02.918095 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:39:02.918327 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:02.918228 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 16:39:02.946118 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:02.946087 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:39:02.959413 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:02.946407 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:39:03.062701 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:03.062659 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rbxv\" (UniqueName: \"kubernetes.io/projected/3fd39b8a-e318-4412-b153-ee3c8dfae71b-kube-api-access-2rbxv\") pod \"maas-api-revoke-sub-aa13dec8-dvsd4\" (UID: \"3fd39b8a-e318-4412-b153-ee3c8dfae71b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa13dec8-dvsd4" Oct 30 16:39:03.063124 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:03.063094 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fd39b8a-e318-4412-b153-ee3c8dfae71b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa13dec8-dvsd4\" (UID: \"3fd39b8a-e318-4412-b153-ee3c8dfae71b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa13dec8-dvsd4" Oct 30 16:39:03.063452 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:03.063424 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fd39b8a-e318-4412-b153-ee3c8dfae71b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa13dec8-dvsd4\" (UID: \"3fd39b8a-e318-4412-b153-ee3c8dfae71b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa13dec8-dvsd4" Oct 30 16:39:03.204865 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:03.173918 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fd39b8a-e318-4412-b153-ee3c8dfae71b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa13dec8-dvsd4\" (UID: \"3fd39b8a-e318-4412-b153-ee3c8dfae71b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa13dec8-dvsd4" Oct 30 16:39:03.204865 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:03.174052 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fd39b8a-e318-4412-b153-ee3c8dfae71b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa13dec8-dvsd4\" (UID: \"3fd39b8a-e318-4412-b153-ee3c8dfae71b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa13dec8-dvsd4" Oct 30 16:39:03.204865 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:03.174168 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rbxv\" (UniqueName: \"kubernetes.io/projected/3fd39b8a-e318-4412-b153-ee3c8dfae71b-kube-api-access-2rbxv\") pod \"maas-api-revoke-sub-aa13dec8-dvsd4\" (UID: \"3fd39b8a-e318-4412-b153-ee3c8dfae71b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa13dec8-dvsd4" Oct 30 16:39:03.204865 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:03.189957 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fd39b8a-e318-4412-b153-ee3c8dfae71b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa13dec8-dvsd4\" (UID: \"3fd39b8a-e318-4412-b153-ee3c8dfae71b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa13dec8-dvsd4" Oct 30 16:39:03.233819 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:03.227558 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rbxv\" (UniqueName: \"kubernetes.io/projected/3fd39b8a-e318-4412-b153-ee3c8dfae71b-kube-api-access-2rbxv\") pod \"maas-api-revoke-sub-aa13dec8-dvsd4\" (UID: \"3fd39b8a-e318-4412-b153-ee3c8dfae71b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa13dec8-dvsd4" Oct 30 16:39:03.238024 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:03.237992 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fd39b8a-e318-4412-b153-ee3c8dfae71b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa13dec8-dvsd4\" (UID: \"3fd39b8a-e318-4412-b153-ee3c8dfae71b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa13dec8-dvsd4" Oct 30 16:39:03.266823 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:03.266340 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa13dec8-dvsd4"] Oct 30 16:39:03.266823 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:03.266495 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" podUID="e43738f3-d55f-4c0a-a364-bf2ac76605c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:03.335811 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:03.332888 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" podUID="08baad08-cd72-4305-b92d-c11a84ef276f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:03.335811 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:03.334436 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa13dec8-dvsd4" Oct 30 16:39:03.367945 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:03.367895 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" podUID="e76172b1-659d-45bd-91c6-1a0ce4062335" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:03.482088 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:03.481985 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" podUID="f3e1c8ff-fcc3-4a1f-9730-c617d57b5193" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:03.701929 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:03.701881 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" podUID="14805045-a727-41ba-8bd0-100d41f9f27d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:03.773019 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:03.772903 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" podUID="b5abe703-fee0-4185-aa22-88d1db06392b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:03.780919 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:03.780869 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" podUID="e90f43d3-f5eb-45ab-af1e-340d0ce18966" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:03.888810 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:03.885882 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" podUID="80b9bd9a-30d6-45ff-b318-26fcdc608e5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:04.172127 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:04.172079 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" podUID="18b72648-f523-405e-b5cd-1266ff13f8f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:04.221171 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:04.221105 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" podUID="0eadc5d4-98c3-4bf2-b3c2-f73fe249182f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:04.316698 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:04.314946 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" podUID="e43738f3-d55f-4c0a-a364-bf2ac76605c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:04.316698 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:04.315381 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" Oct 30 16:39:04.334281 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:04.334234 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" podUID="08baad08-cd72-4305-b92d-c11a84ef276f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:04.339218 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:04.339189 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" Oct 30 16:39:04.483247 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:04.464245 4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" podUID="14805045-a727-41ba-8bd0-100d41f9f27d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:04.483247 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:04.464630 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" podUID="e76172b1-659d-45bd-91c6-1a0ce4062335" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:04.483247 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:04.465006 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" Oct 30 16:39:04.552876 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:04.534882 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" podUID="f3e1c8ff-fcc3-4a1f-9730-c617d57b5193" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:04.552876 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:04.535359 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" Oct 30 16:39:04.552876 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:04.536332 4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" podUID="b5abe703-fee0-4185-aa22-88d1db06392b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:04.831865 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:04.816902 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" podUID="e90f43d3-f5eb-45ab-af1e-340d0ce18966" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:04.831865 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:04.817654 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" Oct 30 16:39:04.900810 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:04.875391 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" podUID="b5abe703-fee0-4185-aa22-88d1db06392b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:04.901882 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:04.901844 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" podUID="14805045-a727-41ba-8bd0-100d41f9f27d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 16:39:04.910808 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:04.902767 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" Oct 30 16:39:04.925810 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:04.911065 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" Oct 30 16:39:04.928991 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:04.909713 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" podUID="80b9bd9a-30d6-45ff-b318-26fcdc608e5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 16:39:04.938819 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:04.909017 4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" podUID="e43738f3-d55f-4c0a-a364-bf2ac76605c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:05.175371 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:05.175326 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" podUID="18b72648-f523-405e-b5cd-1266ff13f8f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:05.439811 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:05.438878 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" podUID="0eadc5d4-98c3-4bf2-b3c2-f73fe249182f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:05.611807 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:05.575017 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" podUID="f3e1c8ff-fcc3-4a1f-9730-c617d57b5193" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:05.676053 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:05.577077 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" podUID="08baad08-cd72-4305-b92d-c11a84ef276f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:05.676053 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:05.577296 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" podUID="e43738f3-d55f-4c0a-a364-bf2ac76605c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:05.676053 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:05.662127 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" podUID="e76172b1-659d-45bd-91c6-1a0ce4062335" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:05.806860 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:05.794627 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" Oct 30 16:39:05.810967 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:05.810942 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" Oct 30 16:39:05.835810 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:05.811217 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" Oct 30 16:39:05.878618 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:05.878571 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" podUID="e90f43d3-f5eb-45ab-af1e-340d0ce18966" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:05.903250 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:05.903209 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" podUID="14805045-a727-41ba-8bd0-100d41f9f27d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 16:39:05.907809 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:05.904652 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" podUID="b5abe703-fee0-4185-aa22-88d1db06392b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Oct 30 16:39:05.957808 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:05.956025 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" podUID="80b9bd9a-30d6-45ff-b318-26fcdc608e5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:06.008727 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:05.995568 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa13dec8-dvsd4" event={"ID":"3fd39b8a-e318-4412-b153-ee3c8dfae71b","Type":"ContainerStarted","Data":"8330cbf79434d4732155cbf3cbd521cac98d34bcbab3781ff17cc463e69fd9e1"} Oct 30 16:39:06.160297 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:06.140897 4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" podUID="e76172b1-659d-45bd-91c6-1a0ce4062335" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:06.251080 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:06.251034 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" podUID="18b72648-f523-405e-b5cd-1266ff13f8f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:06.387437 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:06.387399 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa13dec8-dvsd4"] Oct 30 16:39:06.443007 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:06.442889 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" podUID="0eadc5d4-98c3-4bf2-b3c2-f73fe249182f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:06.584947 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:06.584889 4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" podUID="e90f43d3-f5eb-45ab-af1e-340d0ce18966" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:06.671829 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:06.670966 4493 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:39:06.671829 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:06.671085 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn" podUID="84009ea2-157b-45c6-b4bd-6df55fc7c72d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:06.770505 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:06.770410 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" podUID="08baad08-cd72-4305-b92d-c11a84ef276f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:06.784500 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:06.784461 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" podUID="e76172b1-659d-45bd-91c6-1a0ce4062335" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 16:39:06.806933 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:06.805910 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" podUID="f3e1c8ff-fcc3-4a1f-9730-c617d57b5193" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:06.806933 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:06.806083 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" podUID="e43738f3-d55f-4c0a-a364-bf2ac76605c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:06.879630 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:06.879587 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" podUID="e90f43d3-f5eb-45ab-af1e-340d0ce18966" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:06.904876 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:06.904823 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" podUID="b5abe703-fee0-4185-aa22-88d1db06392b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:06.905718 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:06.905686 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" podUID="14805045-a727-41ba-8bd0-100d41f9f27d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 16:39:06.984844 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:06.971677 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" podUID="80b9bd9a-30d6-45ff-b318-26fcdc608e5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 16:39:07.275950 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:07.271943 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" podUID="18b72648-f523-405e-b5cd-1266ff13f8f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:07.338697 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:07.336166 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" Oct 30 16:39:07.370822 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:07.364363 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" Oct 30 16:39:07.869830 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:07.869731 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" podUID="e76172b1-659d-45bd-91c6-1a0ce4062335" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 16:39:07.915960 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:07.915904 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" podUID="e43738f3-d55f-4c0a-a364-bf2ac76605c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:07.959894 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:07.959750 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" podUID="e90f43d3-f5eb-45ab-af1e-340d0ce18966" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:07.972257 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:07.972202 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" podUID="f3e1c8ff-fcc3-4a1f-9730-c617d57b5193" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:08.019807 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:08.019156 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" podUID="80b9bd9a-30d6-45ff-b318-26fcdc608e5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 16:39:08.038829 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:08.030983 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" podUID="b5abe703-fee0-4185-aa22-88d1db06392b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Oct 30 16:39:08.038829 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:08.031337 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" podUID="14805045-a727-41ba-8bd0-100d41f9f27d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 16:39:08.104886 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:08.084049 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-6w2sv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:39:08.104886 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:08.084118 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6w2sv" podUID="59ae5769-a6cf-444d-947e-313ff39785a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:08.297628 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:08.297598 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" Oct 30 16:39:08.348999 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:08.336272 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" podUID="18b72648-f523-405e-b5cd-1266ff13f8f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:08.535812 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:08.523380 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-d9rhc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:39:08.535812 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:08.523438 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-d9rhc" podUID="0476eb7d-91e3-4f24-bfbc-245775c8e93b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:08.732377 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:08.711045 4493 generic.go:356] "Generic (PLEG): container finished" podID="3fd39b8a-e318-4412-b153-ee3c8dfae71b" containerID="40442b7bedd4ae44381042fbd2abb87d8c2b211e2ba7dca3fa8a7aa843e6b5d5" exitCode=0 Oct 30 16:39:08.732624 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:08.711215 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa13dec8-dvsd4" event={"ID":"3fd39b8a-e318-4412-b153-ee3c8dfae71b","Type":"ContainerDied","Data":"40442b7bedd4ae44381042fbd2abb87d8c2b211e2ba7dca3fa8a7aa843e6b5d5"} Oct 30 16:39:08.773517 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:08.773487 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" Oct 30 16:39:08.959072 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:08.959032 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" podUID="e76172b1-659d-45bd-91c6-1a0ce4062335" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:09.003896 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:08.963083 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" podUID="e90f43d3-f5eb-45ab-af1e-340d0ce18966" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:09.072933 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:08.994871 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" podUID="e43738f3-d55f-4c0a-a364-bf2ac76605c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:09.074811 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:09.036211 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" podUID="14805045-a727-41ba-8bd0-100d41f9f27d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 16:39:09.091859 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:09.068107 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" podUID="80b9bd9a-30d6-45ff-b318-26fcdc608e5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:09.156817 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:09.156743 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" podUID="08baad08-cd72-4305-b92d-c11a84ef276f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:09.296146 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:09.296041 4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" podUID="80b9bd9a-30d6-45ff-b318-26fcdc608e5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": context deadline exceeded" Oct 30 16:39:09.324051 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:09.324023 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" Oct 30 16:39:09.363829 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:09.341142 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" podUID="18b72648-f523-405e-b5cd-1266ff13f8f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:09.372484 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:09.372134 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" Oct 30 16:39:09.378080 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:09.378057 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" Oct 30 16:39:09.389956 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:09.389936 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" Oct 30 16:39:09.418717 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:09.418690 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" Oct 30 16:39:09.423670 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:09.423651 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" Oct 30 16:39:09.998596 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:09.998568 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa13dec8-dvsd4" Oct 30 16:39:10.163927 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:10.163894 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fd39b8a-e318-4412-b153-ee3c8dfae71b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fd39b8a-e318-4412-b153-ee3c8dfae71b-maas-api-openshift-ca\") pod \"3fd39b8a-e318-4412-b153-ee3c8dfae71b\" (UID: \"3fd39b8a-e318-4412-b153-ee3c8dfae71b\") " Oct 30 16:39:10.164106 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:10.164001 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fd39b8a-e318-4412-b153-ee3c8dfae71b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fd39b8a-e318-4412-b153-ee3c8dfae71b-maas-api-xks-ca\") pod \"3fd39b8a-e318-4412-b153-ee3c8dfae71b\" (UID: \"3fd39b8a-e318-4412-b153-ee3c8dfae71b\") " Oct 30 16:39:10.164106 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:10.164044 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fd39b8a-e318-4412-b153-ee3c8dfae71b-kube-api-access-2rbxv\" (UniqueName: \"kubernetes.io/projected/3fd39b8a-e318-4412-b153-ee3c8dfae71b-kube-api-access-2rbxv\") pod \"3fd39b8a-e318-4412-b153-ee3c8dfae71b\" (UID: \"3fd39b8a-e318-4412-b153-ee3c8dfae71b\") " Oct 30 16:39:10.164254 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:10.164232 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fd39b8a-e318-4412-b153-ee3c8dfae71b-maas-api-openshift-ca" pod "3fd39b8a-e318-4412-b153-ee3c8dfae71b" (UID: "3fd39b8a-e318-4412-b153-ee3c8dfae71b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:39:10.164558 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:10.164535 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fd39b8a-e318-4412-b153-ee3c8dfae71b-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:39:10.166356 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:10.166328 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fd39b8a-e318-4412-b153-ee3c8dfae71b-maas-api-xks-ca" pod "3fd39b8a-e318-4412-b153-ee3c8dfae71b" (UID: "3fd39b8a-e318-4412-b153-ee3c8dfae71b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:39:10.166504 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:10.166349 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fd39b8a-e318-4412-b153-ee3c8dfae71b-kube-api-access-2rbxv" pod "3fd39b8a-e318-4412-b153-ee3c8dfae71b" (UID: "3fd39b8a-e318-4412-b153-ee3c8dfae71b"). InnerVolumeSpecName "kube-api-access-2rbxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:39:10.265372 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:10.265337 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fd39b8a-e318-4412-b153-ee3c8dfae71b-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:39:10.265372 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:10.265370 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rbxv\" (UniqueName: \"kubernetes.io/projected/3fd39b8a-e318-4412-b153-ee3c8dfae71b-kube-api-access-2rbxv\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:39:10.775844 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:10.775810 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa13dec8-dvsd4" Oct 30 16:39:10.775844 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:10.775823 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa13dec8-dvsd4" event={"ID":"3fd39b8a-e318-4412-b153-ee3c8dfae71b","Type":"ContainerDied","Data":"8330cbf79434d4732155cbf3cbd521cac98d34bcbab3781ff17cc463e69fd9e1"} Oct 30 16:39:10.776038 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:10.775916 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8330cbf79434d4732155cbf3cbd521cac98d34bcbab3781ff17cc463e69fd9e1" Oct 30 16:39:24.961136 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:24.961104 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 16:39:27.772952 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:27.772917 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4"] Oct 30 16:39:27.773749 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:27.773728 4493 cpu_manager.go:414] "containerMap: removing container" podUID="3fd39b8a-e318-4412-b153-ee3c8dfae71b" containerName="revoke-keys" Oct 30 16:39:27.773884 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:27.773751 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fd39b8a-e318-4412-b153-ee3c8dfae71b" containerName="revoke-keys" Oct 30 16:39:27.773945 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:27.773916 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fd39b8a-e318-4412-b153-ee3c8dfae71b" containerName="revoke-keys" Oct 30 16:39:27.778140 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:27.778120 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4" Oct 30 16:39:27.819840 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:27.819799 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da5ec9e6-8379-46ee-a2d9-2360237ccf2d-workload-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4\" (UID: \"da5ec9e6-8379-46ee-a2d9-2360237ccf2d\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4" Oct 30 16:39:27.820039 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:27.819849 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da5ec9e6-8379-46ee-a2d9-2360237ccf2d-istio-token\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4\" (UID: \"da5ec9e6-8379-46ee-a2d9-2360237ccf2d\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4" Oct 30 16:39:27.820039 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:27.819930 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da5ec9e6-8379-46ee-a2d9-2360237ccf2d-istio-data\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4\" (UID: \"da5ec9e6-8379-46ee-a2d9-2360237ccf2d\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4" Oct 30 16:39:27.820039 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:27.820008 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da5ec9e6-8379-46ee-a2d9-2360237ccf2d-istiod-ca-cert\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4\" (UID: \"da5ec9e6-8379-46ee-a2d9-2360237ccf2d\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4" Oct 30 16:39:27.820213 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:27.820055 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da5ec9e6-8379-46ee-a2d9-2360237ccf2d-credential-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4\" (UID: \"da5ec9e6-8379-46ee-a2d9-2360237ccf2d\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4" Oct 30 16:39:27.820213 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:27.820109 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da5ec9e6-8379-46ee-a2d9-2360237ccf2d-istio-envoy\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4\" (UID: \"da5ec9e6-8379-46ee-a2d9-2360237ccf2d\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4" Oct 30 16:39:27.820213 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:27.820155 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da5ec9e6-8379-46ee-a2d9-2360237ccf2d-istio-podinfo\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4\" (UID: \"da5ec9e6-8379-46ee-a2d9-2360237ccf2d\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4" Oct 30 16:39:27.820371 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:27.820214 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da5ec9e6-8379-46ee-a2d9-2360237ccf2d-workload-certs\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4\" (UID: \"da5ec9e6-8379-46ee-a2d9-2360237ccf2d\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4" Oct 30 16:39:27.820371 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:27.820239 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snk48\" (UniqueName: \"kubernetes.io/projected/da5ec9e6-8379-46ee-a2d9-2360237ccf2d-kube-api-access-snk48\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4\" (UID: \"da5ec9e6-8379-46ee-a2d9-2360237ccf2d\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4" Oct 30 16:39:27.821281 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:27.821144 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4"] Oct 30 16:39:27.921918 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:27.921883 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da5ec9e6-8379-46ee-a2d9-2360237ccf2d-workload-certs\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4\" (UID: \"da5ec9e6-8379-46ee-a2d9-2360237ccf2d\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4" Oct 30 16:39:27.921918 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:27.921914 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snk48\" (UniqueName: \"kubernetes.io/projected/da5ec9e6-8379-46ee-a2d9-2360237ccf2d-kube-api-access-snk48\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4\" (UID: \"da5ec9e6-8379-46ee-a2d9-2360237ccf2d\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4" Oct 30 16:39:27.922136 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:27.922044 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da5ec9e6-8379-46ee-a2d9-2360237ccf2d-workload-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4\" (UID: \"da5ec9e6-8379-46ee-a2d9-2360237ccf2d\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4" Oct 30 16:39:27.922136 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:27.922063 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da5ec9e6-8379-46ee-a2d9-2360237ccf2d-istio-token\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4\" (UID: \"da5ec9e6-8379-46ee-a2d9-2360237ccf2d\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4" Oct 30 16:39:27.922239 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:27.922137 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da5ec9e6-8379-46ee-a2d9-2360237ccf2d-istio-data\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4\" (UID: \"da5ec9e6-8379-46ee-a2d9-2360237ccf2d\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4" Oct 30 16:39:27.922239 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:27.922164 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da5ec9e6-8379-46ee-a2d9-2360237ccf2d-istiod-ca-cert\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4\" (UID: \"da5ec9e6-8379-46ee-a2d9-2360237ccf2d\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4" Oct 30 16:39:27.922239 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:27.922197 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da5ec9e6-8379-46ee-a2d9-2360237ccf2d-credential-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4\" (UID: \"da5ec9e6-8379-46ee-a2d9-2360237ccf2d\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4" Oct 30 16:39:27.922386 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:27.922245 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da5ec9e6-8379-46ee-a2d9-2360237ccf2d-istio-envoy\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4\" (UID: \"da5ec9e6-8379-46ee-a2d9-2360237ccf2d\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4" Oct 30 16:39:27.922386 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:27.922282 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da5ec9e6-8379-46ee-a2d9-2360237ccf2d-workload-certs\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4\" (UID: \"da5ec9e6-8379-46ee-a2d9-2360237ccf2d\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4" Oct 30 16:39:27.922386 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:27.922292 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da5ec9e6-8379-46ee-a2d9-2360237ccf2d-istio-podinfo\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4\" (UID: \"da5ec9e6-8379-46ee-a2d9-2360237ccf2d\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4" Oct 30 16:39:27.922524 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:27.922430 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da5ec9e6-8379-46ee-a2d9-2360237ccf2d-workload-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4\" (UID: \"da5ec9e6-8379-46ee-a2d9-2360237ccf2d\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4" Oct 30 16:39:27.922524 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:27.922459 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da5ec9e6-8379-46ee-a2d9-2360237ccf2d-istio-data\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4\" (UID: \"da5ec9e6-8379-46ee-a2d9-2360237ccf2d\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4" Oct 30 16:39:27.922635 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:27.922608 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da5ec9e6-8379-46ee-a2d9-2360237ccf2d-credential-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4\" (UID: \"da5ec9e6-8379-46ee-a2d9-2360237ccf2d\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4" Oct 30 16:39:27.923014 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:27.922992 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da5ec9e6-8379-46ee-a2d9-2360237ccf2d-istiod-ca-cert\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4\" (UID: \"da5ec9e6-8379-46ee-a2d9-2360237ccf2d\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4" Oct 30 16:39:27.924574 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:27.924553 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da5ec9e6-8379-46ee-a2d9-2360237ccf2d-istio-envoy\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4\" (UID: \"da5ec9e6-8379-46ee-a2d9-2360237ccf2d\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4" Oct 30 16:39:27.924713 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:27.924695 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da5ec9e6-8379-46ee-a2d9-2360237ccf2d-istio-podinfo\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4\" (UID: \"da5ec9e6-8379-46ee-a2d9-2360237ccf2d\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4" Oct 30 16:39:27.944732 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:27.944711 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da5ec9e6-8379-46ee-a2d9-2360237ccf2d-istio-token\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4\" (UID: \"da5ec9e6-8379-46ee-a2d9-2360237ccf2d\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4" Oct 30 16:39:27.944869 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:27.944807 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snk48\" (UniqueName: \"kubernetes.io/projected/da5ec9e6-8379-46ee-a2d9-2360237ccf2d-kube-api-access-snk48\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4\" (UID: \"da5ec9e6-8379-46ee-a2d9-2360237ccf2d\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4" Oct 30 16:39:28.093212 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:28.093118 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4" Oct 30 16:39:28.425074 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:39:28.425047 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda5ec9e6_8379_46ee_a2d9_2360237ccf2d.slice/crio-3189fba18e047d98e5d906d05678a4f1de1520a183a6b5ff7bb25e95c54fa5f8 WatchSource:0}: Error finding container 3189fba18e047d98e5d906d05678a4f1de1520a183a6b5ff7bb25e95c54fa5f8: Status 404 returned error can't find the container with id 3189fba18e047d98e5d906d05678a4f1de1520a183a6b5ff7bb25e95c54fa5f8 Oct 30 16:39:28.426343 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:28.426318 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4"] Oct 30 16:39:28.427896 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:28.427858 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:39:28.427980 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:28.427924 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:39:28.427980 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:28.427952 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:39:29.162602 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:29.162563 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4" event={"ID":"da5ec9e6-8379-46ee-a2d9-2360237ccf2d","Type":"ContainerStarted","Data":"d42783125009f74712772bb9386d5740e54727c4ede5e78a24b84706230eecea"} Oct 30 16:39:29.163026 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:29.162628 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4" event={"ID":"da5ec9e6-8379-46ee-a2d9-2360237ccf2d","Type":"ContainerStarted","Data":"3189fba18e047d98e5d906d05678a4f1de1520a183a6b5ff7bb25e95c54fa5f8"} Oct 30 16:39:29.233079 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:29.233005 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4" podStartSLOduration=2.232987312 podStartE2EDuration="2.232987312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:39:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:39:29.23011594 +0000 UTC m=+996.457291045" watchObservedRunningTime="2026-10-30 16:39:29.232987312 +0000 UTC m=+996.460162469" Oct 30 16:39:30.093821 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:30.093790 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4" Oct 30 16:39:30.095440 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:30.095414 4493 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Oct 30 16:39:30.095572 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:30.095467 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4" podUID="da5ec9e6-8379-46ee-a2d9-2360237ccf2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Oct 30 16:39:30.356173 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:39:30.356089 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 16:39:31.093518 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:31.093486 4493 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Oct 30 16:39:31.093699 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:31.093542 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4" podUID="da5ec9e6-8379-46ee-a2d9-2360237ccf2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Oct 30 16:39:32.094216 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:32.094186 4493 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Oct 30 16:39:32.094615 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:32.094243 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4" podUID="da5ec9e6-8379-46ee-a2d9-2360237ccf2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Oct 30 16:39:33.094352 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:33.094319 4493 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Oct 30 16:39:33.094961 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:33.094924 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4" podUID="da5ec9e6-8379-46ee-a2d9-2360237ccf2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Oct 30 16:39:34.094117 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:34.094080 4493 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Oct 30 16:39:34.094297 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:34.094166 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4" podUID="da5ec9e6-8379-46ee-a2d9-2360237ccf2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Oct 30 16:39:35.094470 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:35.094433 4493 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body= Oct 30 16:39:35.094976 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:35.094506 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4" podUID="da5ec9e6-8379-46ee-a2d9-2360237ccf2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" Oct 30 16:39:36.098428 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:36.098399 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4" Oct 30 16:39:36.098908 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:36.098890 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4" Oct 30 16:39:36.099752 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:36.099729 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4" Oct 30 16:39:38.154842 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:38.153596 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5kzmj"] Oct 30 16:39:38.182354 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:38.182322 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5kzmj"] Oct 30 16:39:38.182531 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:38.182459 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5kzmj" Oct 30 16:39:38.207762 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:38.205201 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 16:39:38.207762 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:38.205216 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 16:39:38.207762 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:38.205276 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 16:39:38.207762 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:38.205422 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-pmfs7\"" Oct 30 16:39:38.287515 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:38.287481 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/439163d6-6cb9-4b15-bcae-2f6f6cdecde2-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5kzmj\" (UID: \"439163d6-6cb9-4b15-bcae-2f6f6cdecde2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5kzmj" Oct 30 16:39:38.287740 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:38.287718 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hknxx\" (UniqueName: \"kubernetes.io/projected/439163d6-6cb9-4b15-bcae-2f6f6cdecde2-kube-api-access-hknxx\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5kzmj\" (UID: \"439163d6-6cb9-4b15-bcae-2f6f6cdecde2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5kzmj" Oct 30 16:39:38.287849 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:38.287762 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/439163d6-6cb9-4b15-bcae-2f6f6cdecde2-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5kzmj\" (UID: \"439163d6-6cb9-4b15-bcae-2f6f6cdecde2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5kzmj" Oct 30 16:39:38.287917 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:38.287869 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/439163d6-6cb9-4b15-bcae-2f6f6cdecde2-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5kzmj\" (UID: \"439163d6-6cb9-4b15-bcae-2f6f6cdecde2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5kzmj" Oct 30 16:39:38.287964 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:38.287926 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/439163d6-6cb9-4b15-bcae-2f6f6cdecde2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5kzmj\" (UID: \"439163d6-6cb9-4b15-bcae-2f6f6cdecde2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5kzmj" Oct 30 16:39:38.288091 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:38.288066 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/439163d6-6cb9-4b15-bcae-2f6f6cdecde2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5kzmj\" (UID: \"439163d6-6cb9-4b15-bcae-2f6f6cdecde2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5kzmj" Oct 30 16:39:38.288154 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:38.288125 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/439163d6-6cb9-4b15-bcae-2f6f6cdecde2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5kzmj\" (UID: \"439163d6-6cb9-4b15-bcae-2f6f6cdecde2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5kzmj" Oct 30 16:39:38.389312 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:38.389278 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/439163d6-6cb9-4b15-bcae-2f6f6cdecde2-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5kzmj\" (UID: \"439163d6-6cb9-4b15-bcae-2f6f6cdecde2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5kzmj" Oct 30 16:39:38.389511 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:38.389448 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hknxx\" (UniqueName: \"kubernetes.io/projected/439163d6-6cb9-4b15-bcae-2f6f6cdecde2-kube-api-access-hknxx\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5kzmj\" (UID: \"439163d6-6cb9-4b15-bcae-2f6f6cdecde2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5kzmj" Oct 30 16:39:38.389511 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:38.389474 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/439163d6-6cb9-4b15-bcae-2f6f6cdecde2-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5kzmj\" (UID: \"439163d6-6cb9-4b15-bcae-2f6f6cdecde2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5kzmj" Oct 30 16:39:38.389511 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:38.389495 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/439163d6-6cb9-4b15-bcae-2f6f6cdecde2-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5kzmj\" (UID: \"439163d6-6cb9-4b15-bcae-2f6f6cdecde2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5kzmj" Oct 30 16:39:38.389674 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:38.389531 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/439163d6-6cb9-4b15-bcae-2f6f6cdecde2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5kzmj\" (UID: \"439163d6-6cb9-4b15-bcae-2f6f6cdecde2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5kzmj" Oct 30 16:39:38.389674 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:38.389604 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/439163d6-6cb9-4b15-bcae-2f6f6cdecde2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5kzmj\" (UID: \"439163d6-6cb9-4b15-bcae-2f6f6cdecde2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5kzmj" Oct 30 16:39:38.389674 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:38.389657 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/439163d6-6cb9-4b15-bcae-2f6f6cdecde2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5kzmj\" (UID: \"439163d6-6cb9-4b15-bcae-2f6f6cdecde2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5kzmj" Oct 30 16:39:38.390694 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:38.390664 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/439163d6-6cb9-4b15-bcae-2f6f6cdecde2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5kzmj\" (UID: \"439163d6-6cb9-4b15-bcae-2f6f6cdecde2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5kzmj" Oct 30 16:39:38.392429 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:38.392409 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/439163d6-6cb9-4b15-bcae-2f6f6cdecde2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5kzmj\" (UID: \"439163d6-6cb9-4b15-bcae-2f6f6cdecde2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5kzmj" Oct 30 16:39:38.392536 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:38.392518 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/439163d6-6cb9-4b15-bcae-2f6f6cdecde2-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5kzmj\" (UID: \"439163d6-6cb9-4b15-bcae-2f6f6cdecde2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5kzmj" Oct 30 16:39:38.392864 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:38.392844 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/439163d6-6cb9-4b15-bcae-2f6f6cdecde2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5kzmj\" (UID: \"439163d6-6cb9-4b15-bcae-2f6f6cdecde2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5kzmj" Oct 30 16:39:38.392922 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:38.392837 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/439163d6-6cb9-4b15-bcae-2f6f6cdecde2-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5kzmj\" (UID: \"439163d6-6cb9-4b15-bcae-2f6f6cdecde2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5kzmj" Oct 30 16:39:38.412444 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:38.412396 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/439163d6-6cb9-4b15-bcae-2f6f6cdecde2-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5kzmj\" (UID: \"439163d6-6cb9-4b15-bcae-2f6f6cdecde2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5kzmj" Oct 30 16:39:38.416128 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:38.416104 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hknxx\" (UniqueName: \"kubernetes.io/projected/439163d6-6cb9-4b15-bcae-2f6f6cdecde2-kube-api-access-hknxx\") pod \"istiod-openshift-gateway-7ddd9cfdc8-5kzmj\" (UID: \"439163d6-6cb9-4b15-bcae-2f6f6cdecde2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5kzmj" Oct 30 16:39:38.513241 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:38.513215 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5kzmj" Oct 30 16:39:38.858984 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:39:38.858950 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod439163d6_6cb9_4b15_bcae_2f6f6cdecde2.slice/crio-e32eb5e5358db9c173618433ddc5839863830e9c9a42420aae08e41203d8848e WatchSource:0}: Error finding container e32eb5e5358db9c173618433ddc5839863830e9c9a42420aae08e41203d8848e: Status 404 returned error can't find the container with id e32eb5e5358db9c173618433ddc5839863830e9c9a42420aae08e41203d8848e Oct 30 16:39:38.859397 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:38.859375 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5kzmj"] Oct 30 16:39:38.861300 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:38.861281 4493 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 16:39:39.260734 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:39.260696 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5kzmj" event={"ID":"439163d6-6cb9-4b15-bcae-2f6f6cdecde2","Type":"ContainerStarted","Data":"e32eb5e5358db9c173618433ddc5839863830e9c9a42420aae08e41203d8848e"} Oct 30 16:39:41.949466 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:41.949432 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:39:41.949773 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:41.949496 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:39:42.298771 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:42.298681 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5kzmj" event={"ID":"439163d6-6cb9-4b15-bcae-2f6f6cdecde2","Type":"ContainerStarted","Data":"521b68db462ca55a97f83f15e37756bec7728705237c57a8a856d1fde46d9187"} Oct 30 16:39:42.298960 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:42.298862 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5kzmj" Oct 30 16:39:42.399045 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:42.398985 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5kzmj" podStartSLOduration=1.312083699 podStartE2EDuration="4.398968232s" totalImagesPullingTime="3.086884533s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:39:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:39:42.380044658 +0000 UTC m=+1009.607219775" watchObservedRunningTime="2026-10-30 16:39:42.398968232 +0000 UTC m=+1009.626143338" Oct 30 16:39:43.308651 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:43.308620 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5kzmj" Oct 30 16:39:49.050021 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:49.049988 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47f5178c-v8mkn"] Oct 30 16:39:49.058930 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:49.058902 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47f5178c-v8mkn"] Oct 30 16:39:50.936033 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:50.935997 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d3e0d8b-2248-472d-8bbf-90504460b614" path="/var/lib/kubelet/pods/8d3e0d8b-2248-472d-8bbf-90504460b614/volumes" Oct 30 16:39:51.713806 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:51.705224 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 16:39:53.254761 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:53.254633 4493 scope.go:122] "RemoveContainer" containerID="c33ba945c787a7d93de0178ec45729069d73adb4ff8f538a7bb378421453511a" Oct 30 16:39:55.044572 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:55.044535 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e296dac2-l6sr7"] Oct 30 16:39:55.055660 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:55.055629 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e296dac2-l6sr7"] Oct 30 16:39:56.951201 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:56.951172 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb7182a3-151d-42f6-ad4e-9ddce1ce44fe" path="/var/lib/kubelet/pods/eb7182a3-151d-42f6-ad4e-9ddce1ce44fe/volumes" Oct 30 16:39:59.034817 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:59.034770 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ca6bcd5-t8r4g"] Oct 30 16:39:59.046160 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:39:59.046138 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ca6bcd5-t8r4g"] Oct 30 16:40:00.342116 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:40:00.342083 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 16:40:00.934860 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:00.934811 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72ed491e-d203-4084-ac8a-dc459c564583" path="/var/lib/kubelet/pods/72ed491e-d203-4084-ac8a-dc459c564583/volumes" Oct 30 16:40:04.038026 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:04.037989 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04fa06a9-wqm7n"] Oct 30 16:40:04.051488 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:04.051455 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04fa06a9-wqm7n"] Oct 30 16:40:04.933879 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:04.933845 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3f871ac-8b1f-4dc5-bb98-2c0f09507776" path="/var/lib/kubelet/pods/f3f871ac-8b1f-4dc5-bb98-2c0f09507776/volumes" Oct 30 16:40:06.034540 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:06.034507 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6040abad-gwb7g"] Oct 30 16:40:06.045899 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:06.045874 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6040abad-gwb7g"] Oct 30 16:40:06.937052 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:06.937011 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec7bcd15-b4d3-4e0f-beec-a8d843f006ac" path="/var/lib/kubelet/pods/ec7bcd15-b4d3-4e0f-beec-a8d843f006ac/volumes" Oct 30 16:40:09.035920 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:09.035887 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe1b2b03-v2nfc"] Oct 30 16:40:09.049733 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:09.049698 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe1b2b03-v2nfc"] Oct 30 16:40:10.935393 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:10.935360 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c866da74-9e9e-4e46-a33c-5f641f2dbb90" path="/var/lib/kubelet/pods/c866da74-9e9e-4e46-a33c-5f641f2dbb90/volumes" Oct 30 16:40:13.129660 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:13.129618 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-020b099b-wxsr9"] Oct 30 16:40:13.134715 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:13.134695 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-020b099b-wxsr9" Oct 30 16:40:13.147214 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:13.147189 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fcd7b01-59e6-43c3-a720-054782533d20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-020b099b-wxsr9\" (UID: \"1fcd7b01-59e6-43c3-a720-054782533d20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-020b099b-wxsr9" Oct 30 16:40:13.147339 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:13.147242 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8trzm\" (UniqueName: \"kubernetes.io/projected/1fcd7b01-59e6-43c3-a720-054782533d20-kube-api-access-8trzm\") pod \"maas-api-revoke-sub-020b099b-wxsr9\" (UID: \"1fcd7b01-59e6-43c3-a720-054782533d20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-020b099b-wxsr9" Oct 30 16:40:13.147505 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:13.147483 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fcd7b01-59e6-43c3-a720-054782533d20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-020b099b-wxsr9\" (UID: \"1fcd7b01-59e6-43c3-a720-054782533d20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-020b099b-wxsr9" Oct 30 16:40:13.155185 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:13.155158 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-020b099b-wxsr9"] Oct 30 16:40:13.164050 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:13.164019 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:40:13.164188 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:13.164176 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 16:40:13.164267 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:13.164234 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:40:13.164267 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:13.164263 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:40:13.248628 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:13.248585 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fcd7b01-59e6-43c3-a720-054782533d20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-020b099b-wxsr9\" (UID: \"1fcd7b01-59e6-43c3-a720-054782533d20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-020b099b-wxsr9" Oct 30 16:40:13.248864 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:13.248637 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8trzm\" (UniqueName: \"kubernetes.io/projected/1fcd7b01-59e6-43c3-a720-054782533d20-kube-api-access-8trzm\") pod \"maas-api-revoke-sub-020b099b-wxsr9\" (UID: \"1fcd7b01-59e6-43c3-a720-054782533d20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-020b099b-wxsr9" Oct 30 16:40:13.248864 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:13.248754 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fcd7b01-59e6-43c3-a720-054782533d20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-020b099b-wxsr9\" (UID: \"1fcd7b01-59e6-43c3-a720-054782533d20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-020b099b-wxsr9" Oct 30 16:40:13.249443 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:13.249419 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fcd7b01-59e6-43c3-a720-054782533d20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-020b099b-wxsr9\" (UID: \"1fcd7b01-59e6-43c3-a720-054782533d20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-020b099b-wxsr9" Oct 30 16:40:13.251330 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:13.251303 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fcd7b01-59e6-43c3-a720-054782533d20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-020b099b-wxsr9\" (UID: \"1fcd7b01-59e6-43c3-a720-054782533d20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-020b099b-wxsr9" Oct 30 16:40:13.276566 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:13.276517 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8trzm\" (UniqueName: \"kubernetes.io/projected/1fcd7b01-59e6-43c3-a720-054782533d20-kube-api-access-8trzm\") pod \"maas-api-revoke-sub-020b099b-wxsr9\" (UID: \"1fcd7b01-59e6-43c3-a720-054782533d20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-020b099b-wxsr9" Oct 30 16:40:13.448995 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:13.448960 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-020b099b-wxsr9" Oct 30 16:40:13.765675 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:13.765637 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-020b099b-wxsr9"] Oct 30 16:40:13.766755 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:40:13.766717 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fcd7b01_59e6_43c3_a720_054782533d20.slice/crio-bc41299e1014fc6071de6e198dd942d9db87895ca095dedc2e4215939e768eb1 WatchSource:0}: Error finding container bc41299e1014fc6071de6e198dd942d9db87895ca095dedc2e4215939e768eb1: Status 404 returned error can't find the container with id bc41299e1014fc6071de6e198dd942d9db87895ca095dedc2e4215939e768eb1 Oct 30 16:40:13.832056 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:13.832016 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-020b099b-wxsr9" event={"ID":"1fcd7b01-59e6-43c3-a720-054782533d20","Type":"ContainerStarted","Data":"bc41299e1014fc6071de6e198dd942d9db87895ca095dedc2e4215939e768eb1"} Oct 30 16:40:14.845906 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:14.845866 4493 generic.go:356] "Generic (PLEG): container finished" podID="1fcd7b01-59e6-43c3-a720-054782533d20" containerID="00e675ce42e43826328445bb71b1af0002ea4b4988a95182465df3931af992bf" exitCode=0 Oct 30 16:40:14.846592 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:14.845974 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-020b099b-wxsr9" event={"ID":"1fcd7b01-59e6-43c3-a720-054782533d20","Type":"ContainerDied","Data":"00e675ce42e43826328445bb71b1af0002ea4b4988a95182465df3931af992bf"} Oct 30 16:40:16.018084 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:16.018061 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-020b099b-wxsr9" Oct 30 16:40:16.089996 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:16.089955 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fcd7b01-59e6-43c3-a720-054782533d20-kube-api-access-8trzm\" (UniqueName: \"kubernetes.io/projected/1fcd7b01-59e6-43c3-a720-054782533d20-kube-api-access-8trzm\") pod \"1fcd7b01-59e6-43c3-a720-054782533d20\" (UID: \"1fcd7b01-59e6-43c3-a720-054782533d20\") " Oct 30 16:40:16.090183 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:16.090022 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fcd7b01-59e6-43c3-a720-054782533d20-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fcd7b01-59e6-43c3-a720-054782533d20-maas-api-xks-ca\") pod \"1fcd7b01-59e6-43c3-a720-054782533d20\" (UID: \"1fcd7b01-59e6-43c3-a720-054782533d20\") " Oct 30 16:40:16.090183 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:16.090048 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fcd7b01-59e6-43c3-a720-054782533d20-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fcd7b01-59e6-43c3-a720-054782533d20-maas-api-openshift-ca\") pod \"1fcd7b01-59e6-43c3-a720-054782533d20\" (UID: \"1fcd7b01-59e6-43c3-a720-054782533d20\") " Oct 30 16:40:16.090483 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:16.090457 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fcd7b01-59e6-43c3-a720-054782533d20-maas-api-openshift-ca" pod "1fcd7b01-59e6-43c3-a720-054782533d20" (UID: "1fcd7b01-59e6-43c3-a720-054782533d20"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:40:16.090614 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:16.090594 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fcd7b01-59e6-43c3-a720-054782533d20-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:40:16.092249 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:16.092221 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fcd7b01-59e6-43c3-a720-054782533d20-kube-api-access-8trzm" pod "1fcd7b01-59e6-43c3-a720-054782533d20" (UID: "1fcd7b01-59e6-43c3-a720-054782533d20"). InnerVolumeSpecName "kube-api-access-8trzm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:40:16.092337 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:16.092252 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fcd7b01-59e6-43c3-a720-054782533d20-maas-api-xks-ca" pod "1fcd7b01-59e6-43c3-a720-054782533d20" (UID: "1fcd7b01-59e6-43c3-a720-054782533d20"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:40:16.191391 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:16.191361 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8trzm\" (UniqueName: \"kubernetes.io/projected/1fcd7b01-59e6-43c3-a720-054782533d20-kube-api-access-8trzm\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:40:16.191391 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:16.191389 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fcd7b01-59e6-43c3-a720-054782533d20-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:40:16.864859 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:16.864833 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-020b099b-wxsr9" Oct 30 16:40:16.864859 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:16.864847 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-020b099b-wxsr9" event={"ID":"1fcd7b01-59e6-43c3-a720-054782533d20","Type":"ContainerDied","Data":"bc41299e1014fc6071de6e198dd942d9db87895ca095dedc2e4215939e768eb1"} Oct 30 16:40:16.865073 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:16.864932 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc41299e1014fc6071de6e198dd942d9db87895ca095dedc2e4215939e768eb1" Oct 30 16:40:17.033291 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:17.033237 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58f6b168-jvvbc"] Oct 30 16:40:17.037771 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:17.037742 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58f6b168-jvvbc"] Oct 30 16:40:18.935874 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:18.935837 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bed50be1-2f29-48f7-b2a2-2a02be02535c" path="/var/lib/kubelet/pods/bed50be1-2f29-48f7-b2a2-2a02be02535c/volumes" Oct 30 16:40:20.933303 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:20.933274 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae0011a0-2b8t5"] Oct 30 16:40:20.933845 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:20.933831 4493 cpu_manager.go:414] "containerMap: removing container" podUID="1fcd7b01-59e6-43c3-a720-054782533d20" containerName="revoke-keys" Oct 30 16:40:20.933897 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:20.933846 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fcd7b01-59e6-43c3-a720-054782533d20" containerName="revoke-keys" Oct 30 16:40:20.933944 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:20.933933 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fcd7b01-59e6-43c3-a720-054782533d20" containerName="revoke-keys" Oct 30 16:40:20.937822 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:20.937806 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae0011a0-2b8t5" Oct 30 16:40:20.962731 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:20.962707 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae0011a0-2b8t5"] Oct 30 16:40:20.966305 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:20.965889 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:40:20.966305 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:20.965982 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:40:20.966305 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:20.966012 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:40:20.966305 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:20.966089 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 16:40:21.056089 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:21.056051 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e69b09a-3937-46d0-864d-929d6f53620b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae0011a0-2b8t5\" (UID: \"1e69b09a-3937-46d0-864d-929d6f53620b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae0011a0-2b8t5" Oct 30 16:40:21.056271 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:21.056101 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e69b09a-3937-46d0-864d-929d6f53620b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae0011a0-2b8t5\" (UID: \"1e69b09a-3937-46d0-864d-929d6f53620b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae0011a0-2b8t5" Oct 30 16:40:21.056442 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:21.056418 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5gps\" (UniqueName: \"kubernetes.io/projected/1e69b09a-3937-46d0-864d-929d6f53620b-kube-api-access-d5gps\") pod \"maas-api-revoke-sub-ae0011a0-2b8t5\" (UID: \"1e69b09a-3937-46d0-864d-929d6f53620b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae0011a0-2b8t5" Oct 30 16:40:21.158058 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:21.158030 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5gps\" (UniqueName: \"kubernetes.io/projected/1e69b09a-3937-46d0-864d-929d6f53620b-kube-api-access-d5gps\") pod \"maas-api-revoke-sub-ae0011a0-2b8t5\" (UID: \"1e69b09a-3937-46d0-864d-929d6f53620b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae0011a0-2b8t5" Oct 30 16:40:21.158203 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:21.158136 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e69b09a-3937-46d0-864d-929d6f53620b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae0011a0-2b8t5\" (UID: \"1e69b09a-3937-46d0-864d-929d6f53620b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae0011a0-2b8t5" Oct 30 16:40:21.158203 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:21.158173 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e69b09a-3937-46d0-864d-929d6f53620b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae0011a0-2b8t5\" (UID: \"1e69b09a-3937-46d0-864d-929d6f53620b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae0011a0-2b8t5" Oct 30 16:40:21.158899 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:21.158871 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e69b09a-3937-46d0-864d-929d6f53620b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae0011a0-2b8t5\" (UID: \"1e69b09a-3937-46d0-864d-929d6f53620b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae0011a0-2b8t5" Oct 30 16:40:21.160698 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:21.160673 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e69b09a-3937-46d0-864d-929d6f53620b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae0011a0-2b8t5\" (UID: \"1e69b09a-3937-46d0-864d-929d6f53620b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae0011a0-2b8t5" Oct 30 16:40:21.182136 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:21.182108 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5gps\" (UniqueName: \"kubernetes.io/projected/1e69b09a-3937-46d0-864d-929d6f53620b-kube-api-access-d5gps\") pod \"maas-api-revoke-sub-ae0011a0-2b8t5\" (UID: \"1e69b09a-3937-46d0-864d-929d6f53620b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae0011a0-2b8t5" Oct 30 16:40:21.252110 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:21.252029 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae0011a0-2b8t5" Oct 30 16:40:21.563275 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:40:21.563242 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e69b09a_3937_46d0_864d_929d6f53620b.slice/crio-ad9a9e6e770f84212123e7b079f85f8f0ff2b1bf06776fcada4d01a094ab2995 WatchSource:0}: Error finding container ad9a9e6e770f84212123e7b079f85f8f0ff2b1bf06776fcada4d01a094ab2995: Status 404 returned error can't find the container with id ad9a9e6e770f84212123e7b079f85f8f0ff2b1bf06776fcada4d01a094ab2995 Oct 30 16:40:21.563404 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:21.563303 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae0011a0-2b8t5"] Oct 30 16:40:21.910801 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:21.910745 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae0011a0-2b8t5" event={"ID":"1e69b09a-3937-46d0-864d-929d6f53620b","Type":"ContainerStarted","Data":"ae3732987a32634279ae045339a6a93875ebe1f95bbfb0dadc77d355298e2c26"} Oct 30 16:40:21.910957 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:21.910831 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae0011a0-2b8t5" event={"ID":"1e69b09a-3937-46d0-864d-929d6f53620b","Type":"ContainerStarted","Data":"ad9a9e6e770f84212123e7b079f85f8f0ff2b1bf06776fcada4d01a094ab2995"} Oct 30 16:40:21.953456 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:21.953402 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae0011a0-2b8t5" podStartSLOduration=1.953389378 podStartE2EDuration="1.953389378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:40:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:40:21.95070483 +0000 UTC m=+1049.177879931" watchObservedRunningTime="2026-10-30 16:40:21.953389378 +0000 UTC m=+1049.180564478" Oct 30 16:40:22.924743 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:22.924711 4493 generic.go:356] "Generic (PLEG): container finished" podID="1e69b09a-3937-46d0-864d-929d6f53620b" containerID="ae3732987a32634279ae045339a6a93875ebe1f95bbfb0dadc77d355298e2c26" exitCode=0 Oct 30 16:40:22.924953 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:22.924817 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae0011a0-2b8t5" event={"ID":"1e69b09a-3937-46d0-864d-929d6f53620b","Type":"ContainerDied","Data":"ae3732987a32634279ae045339a6a93875ebe1f95bbfb0dadc77d355298e2c26"} Oct 30 16:40:24.143337 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:24.143299 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae0011a0-2b8t5" Oct 30 16:40:24.212684 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:24.212603 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e69b09a-3937-46d0-864d-929d6f53620b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e69b09a-3937-46d0-864d-929d6f53620b-maas-api-openshift-ca\") pod \"1e69b09a-3937-46d0-864d-929d6f53620b\" (UID: \"1e69b09a-3937-46d0-864d-929d6f53620b\") " Oct 30 16:40:24.212868 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:24.212753 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e69b09a-3937-46d0-864d-929d6f53620b-kube-api-access-d5gps\" (UniqueName: \"kubernetes.io/projected/1e69b09a-3937-46d0-864d-929d6f53620b-kube-api-access-d5gps\") pod \"1e69b09a-3937-46d0-864d-929d6f53620b\" (UID: \"1e69b09a-3937-46d0-864d-929d6f53620b\") " Oct 30 16:40:24.212868 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:24.212825 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e69b09a-3937-46d0-864d-929d6f53620b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e69b09a-3937-46d0-864d-929d6f53620b-maas-api-xks-ca\") pod \"1e69b09a-3937-46d0-864d-929d6f53620b\" (UID: \"1e69b09a-3937-46d0-864d-929d6f53620b\") " Oct 30 16:40:24.213062 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:24.213026 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e69b09a-3937-46d0-864d-929d6f53620b-maas-api-openshift-ca" pod "1e69b09a-3937-46d0-864d-929d6f53620b" (UID: "1e69b09a-3937-46d0-864d-929d6f53620b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:40:24.213301 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:24.213283 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e69b09a-3937-46d0-864d-929d6f53620b-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:40:24.215817 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:24.215122 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e69b09a-3937-46d0-864d-929d6f53620b-maas-api-xks-ca" pod "1e69b09a-3937-46d0-864d-929d6f53620b" (UID: "1e69b09a-3937-46d0-864d-929d6f53620b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:40:24.215817 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:24.215452 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e69b09a-3937-46d0-864d-929d6f53620b-kube-api-access-d5gps" pod "1e69b09a-3937-46d0-864d-929d6f53620b" (UID: "1e69b09a-3937-46d0-864d-929d6f53620b"). InnerVolumeSpecName "kube-api-access-d5gps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:40:24.314611 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:24.314584 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5gps\" (UniqueName: \"kubernetes.io/projected/1e69b09a-3937-46d0-864d-929d6f53620b-kube-api-access-d5gps\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:40:24.314611 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:24.314611 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e69b09a-3937-46d0-864d-929d6f53620b-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:40:24.944068 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:24.944046 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae0011a0-2b8t5" Oct 30 16:40:24.944191 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:24.944090 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae0011a0-2b8t5" event={"ID":"1e69b09a-3937-46d0-864d-929d6f53620b","Type":"ContainerDied","Data":"ad9a9e6e770f84212123e7b079f85f8f0ff2b1bf06776fcada4d01a094ab2995"} Oct 30 16:40:24.944191 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:24.944138 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad9a9e6e770f84212123e7b079f85f8f0ff2b1bf06776fcada4d01a094ab2995" Oct 30 16:40:27.040145 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:27.040112 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b57f3178-4jjhq"] Oct 30 16:40:27.051791 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:27.051764 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b57f3178-4jjhq"] Oct 30 16:40:28.934744 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:28.934706 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39f93bd8-f7dc-4ce9-9e91-5e012b71cc5c" path="/var/lib/kubelet/pods/39f93bd8-f7dc-4ce9-9e91-5e012b71cc5c/volumes" Oct 30 16:40:30.342136 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:40:30.342096 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 16:40:32.024819 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:32.021358 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q"] Oct 30 16:40:32.024819 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:32.022276 4493 cpu_manager.go:414] "containerMap: removing container" podUID="1e69b09a-3937-46d0-864d-929d6f53620b" containerName="revoke-keys" Oct 30 16:40:32.024819 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:32.022295 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e69b09a-3937-46d0-864d-929d6f53620b" containerName="revoke-keys" Oct 30 16:40:32.024819 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:32.022453 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e69b09a-3937-46d0-864d-929d6f53620b" containerName="revoke-keys" Oct 30 16:40:32.029416 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:32.028649 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q" Oct 30 16:40:32.064465 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:32.064417 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-7410c7-openshift-default-dockercfg-rdh4l\"" Oct 30 16:40:32.087817 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:32.087752 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q"] Oct 30 16:40:32.127413 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:32.127378 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/386d43ba-11f5-440a-ab13-d6b444bacaae-istio-envoy\") pod \"e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q\" (UID: \"386d43ba-11f5-440a-ab13-d6b444bacaae\") " pod="openshift-ingress/e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q" Oct 30 16:40:32.127735 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:32.127710 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/386d43ba-11f5-440a-ab13-d6b444bacaae-istio-podinfo\") pod \"e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q\" (UID: \"386d43ba-11f5-440a-ab13-d6b444bacaae\") " pod="openshift-ingress/e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q" Oct 30 16:40:32.127930 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:32.127912 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/386d43ba-11f5-440a-ab13-d6b444bacaae-istio-token\") pod \"e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q\" (UID: \"386d43ba-11f5-440a-ab13-d6b444bacaae\") " pod="openshift-ingress/e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q" Oct 30 16:40:32.128082 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:32.128068 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/386d43ba-11f5-440a-ab13-d6b444bacaae-istiod-ca-cert\") pod \"e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q\" (UID: \"386d43ba-11f5-440a-ab13-d6b444bacaae\") " pod="openshift-ingress/e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q" Oct 30 16:40:32.128241 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:32.128225 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxvhz\" (UniqueName: \"kubernetes.io/projected/386d43ba-11f5-440a-ab13-d6b444bacaae-kube-api-access-lxvhz\") pod \"e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q\" (UID: \"386d43ba-11f5-440a-ab13-d6b444bacaae\") " pod="openshift-ingress/e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q" Oct 30 16:40:32.128403 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:32.128389 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/386d43ba-11f5-440a-ab13-d6b444bacaae-credential-socket\") pod \"e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q\" (UID: \"386d43ba-11f5-440a-ab13-d6b444bacaae\") " pod="openshift-ingress/e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q" Oct 30 16:40:32.128549 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:32.128536 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/386d43ba-11f5-440a-ab13-d6b444bacaae-workload-certs\") pod \"e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q\" (UID: \"386d43ba-11f5-440a-ab13-d6b444bacaae\") " pod="openshift-ingress/e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q" Oct 30 16:40:32.128688 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:32.128675 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/386d43ba-11f5-440a-ab13-d6b444bacaae-istio-data\") pod \"e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q\" (UID: \"386d43ba-11f5-440a-ab13-d6b444bacaae\") " pod="openshift-ingress/e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q" Oct 30 16:40:32.128879 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:32.128865 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/386d43ba-11f5-440a-ab13-d6b444bacaae-workload-socket\") pod \"e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q\" (UID: \"386d43ba-11f5-440a-ab13-d6b444bacaae\") " pod="openshift-ingress/e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q" Oct 30 16:40:32.230819 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:32.230143 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/386d43ba-11f5-440a-ab13-d6b444bacaae-istio-data\") pod \"e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q\" (UID: \"386d43ba-11f5-440a-ab13-d6b444bacaae\") " pod="openshift-ingress/e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q" Oct 30 16:40:32.230819 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:32.230299 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/386d43ba-11f5-440a-ab13-d6b444bacaae-workload-socket\") pod \"e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q\" (UID: \"386d43ba-11f5-440a-ab13-d6b444bacaae\") " pod="openshift-ingress/e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q" Oct 30 16:40:32.230819 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:32.230385 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/386d43ba-11f5-440a-ab13-d6b444bacaae-istio-envoy\") pod \"e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q\" (UID: \"386d43ba-11f5-440a-ab13-d6b444bacaae\") " pod="openshift-ingress/e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q" Oct 30 16:40:32.230819 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:32.230414 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/386d43ba-11f5-440a-ab13-d6b444bacaae-istio-podinfo\") pod \"e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q\" (UID: \"386d43ba-11f5-440a-ab13-d6b444bacaae\") " pod="openshift-ingress/e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q" Oct 30 16:40:32.230819 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:32.230443 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/386d43ba-11f5-440a-ab13-d6b444bacaae-istio-token\") pod \"e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q\" (UID: \"386d43ba-11f5-440a-ab13-d6b444bacaae\") " pod="openshift-ingress/e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q" Oct 30 16:40:32.230819 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:32.230475 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/386d43ba-11f5-440a-ab13-d6b444bacaae-istiod-ca-cert\") pod \"e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q\" (UID: \"386d43ba-11f5-440a-ab13-d6b444bacaae\") " pod="openshift-ingress/e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q" Oct 30 16:40:32.230819 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:32.230543 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxvhz\" (UniqueName: \"kubernetes.io/projected/386d43ba-11f5-440a-ab13-d6b444bacaae-kube-api-access-lxvhz\") pod \"e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q\" (UID: \"386d43ba-11f5-440a-ab13-d6b444bacaae\") " pod="openshift-ingress/e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q" Oct 30 16:40:32.230819 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:32.230606 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/386d43ba-11f5-440a-ab13-d6b444bacaae-credential-socket\") pod \"e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q\" (UID: \"386d43ba-11f5-440a-ab13-d6b444bacaae\") " pod="openshift-ingress/e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q" Oct 30 16:40:32.230819 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:32.230675 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/386d43ba-11f5-440a-ab13-d6b444bacaae-workload-certs\") pod \"e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q\" (UID: \"386d43ba-11f5-440a-ab13-d6b444bacaae\") " pod="openshift-ingress/e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q" Oct 30 16:40:32.231459 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:32.231131 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/386d43ba-11f5-440a-ab13-d6b444bacaae-workload-certs\") pod \"e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q\" (UID: \"386d43ba-11f5-440a-ab13-d6b444bacaae\") " pod="openshift-ingress/e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q" Oct 30 16:40:32.234811 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:32.231717 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/386d43ba-11f5-440a-ab13-d6b444bacaae-istio-data\") pod \"e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q\" (UID: \"386d43ba-11f5-440a-ab13-d6b444bacaae\") " pod="openshift-ingress/e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q" Oct 30 16:40:32.234811 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:32.231996 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/386d43ba-11f5-440a-ab13-d6b444bacaae-workload-socket\") pod \"e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q\" (UID: \"386d43ba-11f5-440a-ab13-d6b444bacaae\") " pod="openshift-ingress/e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q" Oct 30 16:40:32.243192 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:32.243164 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/386d43ba-11f5-440a-ab13-d6b444bacaae-istio-envoy\") pod \"e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q\" (UID: \"386d43ba-11f5-440a-ab13-d6b444bacaae\") " pod="openshift-ingress/e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q" Oct 30 16:40:32.243953 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:32.243929 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/386d43ba-11f5-440a-ab13-d6b444bacaae-credential-socket\") pod \"e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q\" (UID: \"386d43ba-11f5-440a-ab13-d6b444bacaae\") " pod="openshift-ingress/e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q" Oct 30 16:40:32.244096 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:32.243997 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/386d43ba-11f5-440a-ab13-d6b444bacaae-istiod-ca-cert\") pod \"e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q\" (UID: \"386d43ba-11f5-440a-ab13-d6b444bacaae\") " pod="openshift-ingress/e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q" Oct 30 16:40:32.283821 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:32.277716 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/386d43ba-11f5-440a-ab13-d6b444bacaae-istio-podinfo\") pod \"e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q\" (UID: \"386d43ba-11f5-440a-ab13-d6b444bacaae\") " pod="openshift-ingress/e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q" Oct 30 16:40:32.291357 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:32.291326 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/386d43ba-11f5-440a-ab13-d6b444bacaae-istio-token\") pod \"e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q\" (UID: \"386d43ba-11f5-440a-ab13-d6b444bacaae\") " pod="openshift-ingress/e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q" Oct 30 16:40:32.299870 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:32.299804 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxvhz\" (UniqueName: \"kubernetes.io/projected/386d43ba-11f5-440a-ab13-d6b444bacaae-kube-api-access-lxvhz\") pod \"e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q\" (UID: \"386d43ba-11f5-440a-ab13-d6b444bacaae\") " pod="openshift-ingress/e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q" Oct 30 16:40:32.373935 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:32.373900 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q" Oct 30 16:40:32.728764 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:40:32.728735 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod386d43ba_11f5_440a_ab13_d6b444bacaae.slice/crio-5102848f6b5ba6a3fba3aa0eccf4fa37f9066158c70cef98110a8245d1954056 WatchSource:0}: Error finding container 5102848f6b5ba6a3fba3aa0eccf4fa37f9066158c70cef98110a8245d1954056: Status 404 returned error can't find the container with id 5102848f6b5ba6a3fba3aa0eccf4fa37f9066158c70cef98110a8245d1954056 Oct 30 16:40:32.730171 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:32.730142 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q"] Oct 30 16:40:32.732123 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:32.732077 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:40:32.732242 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:32.732170 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:40:32.732242 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:32.732214 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:40:33.059868 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:33.054999 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q" event={"ID":"386d43ba-11f5-440a-ab13-d6b444bacaae","Type":"ContainerStarted","Data":"9239688fe498dfdcd8b715a6e0d1b6caaffe02cad2c44bf435e781bec69579de"} Oct 30 16:40:33.059868 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:33.055164 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q" event={"ID":"386d43ba-11f5-440a-ab13-d6b444bacaae","Type":"ContainerStarted","Data":"5102848f6b5ba6a3fba3aa0eccf4fa37f9066158c70cef98110a8245d1954056"} Oct 30 16:40:33.136805 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:33.136656 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q" podStartSLOduration=2.136616781 podStartE2EDuration="2.136616781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:40:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:40:33.120299949 +0000 UTC m=+1060.347475063" watchObservedRunningTime="2026-10-30 16:40:33.136616781 +0000 UTC m=+1060.363791882" Oct 30 16:40:33.375128 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:33.375042 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q" Oct 30 16:40:33.376994 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:33.376967 4493 patch_prober.go:29] interesting pod/e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 16:40:33.377168 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:33.377022 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q" podUID="386d43ba-11f5-440a-ab13-d6b444bacaae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 16:40:34.379088 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:34.379050 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q" Oct 30 16:40:34.456215 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:34.456181 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 16:40:35.072927 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:35.072896 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q" Oct 30 16:40:35.074018 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:35.073997 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q" Oct 30 16:40:36.776020 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:36.774898 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" podUID="e76172b1-659d-45bd-91c6-1a0ce4062335" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:36.859808 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:36.858990 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" podUID="e43738f3-d55f-4c0a-a364-bf2ac76605c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:39.152131 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:39.152094 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8295b2-dhz5d"] Oct 30 16:40:39.156800 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:39.156763 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8295b2-dhz5d" Oct 30 16:40:39.172825 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:39.172800 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:40:39.172962 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:39.172828 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:40:39.173167 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:39.173148 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:40:39.183045 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:39.183010 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dfac39d-8c50-469d-a0e0-38fdc16889d5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8295b2-dhz5d\" (UID: \"5dfac39d-8c50-469d-a0e0-38fdc16889d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8295b2-dhz5d" Oct 30 16:40:39.183287 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:39.183265 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwzm9\" (UniqueName: \"kubernetes.io/projected/5dfac39d-8c50-469d-a0e0-38fdc16889d5-kube-api-access-wwzm9\") pod \"maas-api-revoke-keys-e2e-api-a-8295b2-dhz5d\" (UID: \"5dfac39d-8c50-469d-a0e0-38fdc16889d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8295b2-dhz5d" Oct 30 16:40:39.183427 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:39.183412 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dfac39d-8c50-469d-a0e0-38fdc16889d5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8295b2-dhz5d\" (UID: \"5dfac39d-8c50-469d-a0e0-38fdc16889d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8295b2-dhz5d" Oct 30 16:40:39.184748 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:39.184725 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 16:40:39.192509 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:39.192481 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8295b2-dhz5d"] Oct 30 16:40:39.285198 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:39.285162 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dfac39d-8c50-469d-a0e0-38fdc16889d5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8295b2-dhz5d\" (UID: \"5dfac39d-8c50-469d-a0e0-38fdc16889d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8295b2-dhz5d" Oct 30 16:40:39.285944 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:39.285217 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwzm9\" (UniqueName: \"kubernetes.io/projected/5dfac39d-8c50-469d-a0e0-38fdc16889d5-kube-api-access-wwzm9\") pod \"maas-api-revoke-keys-e2e-api-a-8295b2-dhz5d\" (UID: \"5dfac39d-8c50-469d-a0e0-38fdc16889d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8295b2-dhz5d" Oct 30 16:40:39.285944 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:39.285267 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dfac39d-8c50-469d-a0e0-38fdc16889d5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8295b2-dhz5d\" (UID: \"5dfac39d-8c50-469d-a0e0-38fdc16889d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8295b2-dhz5d" Oct 30 16:40:39.285944 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:39.285924 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dfac39d-8c50-469d-a0e0-38fdc16889d5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8295b2-dhz5d\" (UID: \"5dfac39d-8c50-469d-a0e0-38fdc16889d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8295b2-dhz5d" Oct 30 16:40:39.287854 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:39.287830 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dfac39d-8c50-469d-a0e0-38fdc16889d5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8295b2-dhz5d\" (UID: \"5dfac39d-8c50-469d-a0e0-38fdc16889d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8295b2-dhz5d" Oct 30 16:40:39.310118 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:39.310094 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwzm9\" (UniqueName: \"kubernetes.io/projected/5dfac39d-8c50-469d-a0e0-38fdc16889d5-kube-api-access-wwzm9\") pod \"maas-api-revoke-keys-e2e-api-a-8295b2-dhz5d\" (UID: \"5dfac39d-8c50-469d-a0e0-38fdc16889d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8295b2-dhz5d" Oct 30 16:40:39.471851 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:39.471738 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8295b2-dhz5d" Oct 30 16:40:39.789227 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:40:39.789194 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dfac39d_8c50_469d_a0e0_38fdc16889d5.slice/crio-455ec0166e868ee84b0bcc35d2fe77b84c16275b74c3343f54bd4453b25a57e3 WatchSource:0}: Error finding container 455ec0166e868ee84b0bcc35d2fe77b84c16275b74c3343f54bd4453b25a57e3: Status 404 returned error can't find the container with id 455ec0166e868ee84b0bcc35d2fe77b84c16275b74c3343f54bd4453b25a57e3 Oct 30 16:40:39.789545 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:39.789523 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8295b2-dhz5d"] Oct 30 16:40:40.191610 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:40.191578 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8295b2-dhz5d" event={"ID":"5dfac39d-8c50-469d-a0e0-38fdc16889d5","Type":"ContainerStarted","Data":"455ec0166e868ee84b0bcc35d2fe77b84c16275b74c3343f54bd4453b25a57e3"} Oct 30 16:40:41.201885 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:41.201775 4493 generic.go:356] "Generic (PLEG): container finished" podID="5dfac39d-8c50-469d-a0e0-38fdc16889d5" containerID="04220862c312a9077b0163a12af7aca0c11644531dcde2dd3d82564c70dd1a78" exitCode=0 Oct 30 16:40:41.201885 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:41.201817 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8295b2-dhz5d" event={"ID":"5dfac39d-8c50-469d-a0e0-38fdc16889d5","Type":"ContainerDied","Data":"04220862c312a9077b0163a12af7aca0c11644531dcde2dd3d82564c70dd1a78"} Oct 30 16:40:42.448244 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:42.448214 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8295b2-dhz5d" Oct 30 16:40:42.558443 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:42.558308 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5dfac39d-8c50-469d-a0e0-38fdc16889d5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dfac39d-8c50-469d-a0e0-38fdc16889d5-maas-api-openshift-ca\") pod \"5dfac39d-8c50-469d-a0e0-38fdc16889d5\" (UID: \"5dfac39d-8c50-469d-a0e0-38fdc16889d5\") " Oct 30 16:40:42.558647 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:42.558628 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5dfac39d-8c50-469d-a0e0-38fdc16889d5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dfac39d-8c50-469d-a0e0-38fdc16889d5-maas-api-xks-ca\") pod \"5dfac39d-8c50-469d-a0e0-38fdc16889d5\" (UID: \"5dfac39d-8c50-469d-a0e0-38fdc16889d5\") " Oct 30 16:40:42.558731 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:42.558673 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dfac39d-8c50-469d-a0e0-38fdc16889d5-maas-api-openshift-ca" pod "5dfac39d-8c50-469d-a0e0-38fdc16889d5" (UID: "5dfac39d-8c50-469d-a0e0-38fdc16889d5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:40:42.559533 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:42.559460 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dfac39d-8c50-469d-a0e0-38fdc16889d5-kube-api-access-wwzm9\" (UniqueName: \"kubernetes.io/projected/5dfac39d-8c50-469d-a0e0-38fdc16889d5-kube-api-access-wwzm9\") pod \"5dfac39d-8c50-469d-a0e0-38fdc16889d5\" (UID: \"5dfac39d-8c50-469d-a0e0-38fdc16889d5\") " Oct 30 16:40:42.561874 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:42.561840 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dfac39d-8c50-469d-a0e0-38fdc16889d5-kube-api-access-wwzm9" pod "5dfac39d-8c50-469d-a0e0-38fdc16889d5" (UID: "5dfac39d-8c50-469d-a0e0-38fdc16889d5"). InnerVolumeSpecName "kube-api-access-wwzm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:40:42.563155 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:42.563131 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5dfac39d-8c50-469d-a0e0-38fdc16889d5-maas-api-xks-ca" pod "5dfac39d-8c50-469d-a0e0-38fdc16889d5" (UID: "5dfac39d-8c50-469d-a0e0-38fdc16889d5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:40:42.565174 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:42.565150 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dfac39d-8c50-469d-a0e0-38fdc16889d5-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:40:42.565296 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:42.565180 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwzm9\" (UniqueName: \"kubernetes.io/projected/5dfac39d-8c50-469d-a0e0-38fdc16889d5-kube-api-access-wwzm9\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:40:42.565296 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:42.565197 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dfac39d-8c50-469d-a0e0-38fdc16889d5-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:40:43.037542 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:43.037509 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a8fd3e5-rmwfd"] Oct 30 16:40:43.047154 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:43.047125 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a8fd3e5-rmwfd"] Oct 30 16:40:43.229572 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:43.229498 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8295b2-dhz5d" Oct 30 16:40:43.229572 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:43.229531 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8295b2-dhz5d" event={"ID":"5dfac39d-8c50-469d-a0e0-38fdc16889d5","Type":"ContainerDied","Data":"455ec0166e868ee84b0bcc35d2fe77b84c16275b74c3343f54bd4453b25a57e3"} Oct 30 16:40:43.229755 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:43.229576 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="455ec0166e868ee84b0bcc35d2fe77b84c16275b74c3343f54bd4453b25a57e3" Oct 30 16:40:44.934183 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:44.934151 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0768bd59-6767-4d80-9982-abed675e7c27" path="/var/lib/kubelet/pods/0768bd59-6767-4d80-9982-abed675e7c27/volumes" Oct 30 16:40:45.699527 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:45.699494 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b09f1-48lhp"] Oct 30 16:40:45.700133 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:45.700119 4493 cpu_manager.go:414] "containerMap: removing container" podUID="5dfac39d-8c50-469d-a0e0-38fdc16889d5" containerName="revoke-keys" Oct 30 16:40:45.700187 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:45.700135 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dfac39d-8c50-469d-a0e0-38fdc16889d5" containerName="revoke-keys" Oct 30 16:40:45.700258 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:45.700249 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dfac39d-8c50-469d-a0e0-38fdc16889d5" containerName="revoke-keys" Oct 30 16:40:45.704970 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:45.704949 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b09f1-48lhp" Oct 30 16:40:45.731759 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:45.731732 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:40:45.732032 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:45.732011 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:40:45.739900 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:45.739875 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b09f1-48lhp"] Oct 30 16:40:45.806186 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:45.806150 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9abe55b2-ad36-4d9c-8b40-001d1c3b5b48-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0b09f1-48lhp\" (UID: \"9abe55b2-ad36-4d9c-8b40-001d1c3b5b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b09f1-48lhp" Oct 30 16:40:45.806319 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:45.806297 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89swn\" (UniqueName: \"kubernetes.io/projected/9abe55b2-ad36-4d9c-8b40-001d1c3b5b48-kube-api-access-89swn\") pod \"maas-api-revoke-keys-e2e-api-a-0b09f1-48lhp\" (UID: \"9abe55b2-ad36-4d9c-8b40-001d1c3b5b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b09f1-48lhp" Oct 30 16:40:45.806439 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:45.806412 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9abe55b2-ad36-4d9c-8b40-001d1c3b5b48-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0b09f1-48lhp\" (UID: \"9abe55b2-ad36-4d9c-8b40-001d1c3b5b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b09f1-48lhp" Oct 30 16:40:45.815488 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:45.815461 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:40:45.815488 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:45.815474 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 16:40:45.907857 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:45.907831 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89swn\" (UniqueName: \"kubernetes.io/projected/9abe55b2-ad36-4d9c-8b40-001d1c3b5b48-kube-api-access-89swn\") pod \"maas-api-revoke-keys-e2e-api-a-0b09f1-48lhp\" (UID: \"9abe55b2-ad36-4d9c-8b40-001d1c3b5b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b09f1-48lhp" Oct 30 16:40:45.908043 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:45.907879 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9abe55b2-ad36-4d9c-8b40-001d1c3b5b48-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0b09f1-48lhp\" (UID: \"9abe55b2-ad36-4d9c-8b40-001d1c3b5b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b09f1-48lhp" Oct 30 16:40:45.908105 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:45.908054 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9abe55b2-ad36-4d9c-8b40-001d1c3b5b48-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0b09f1-48lhp\" (UID: \"9abe55b2-ad36-4d9c-8b40-001d1c3b5b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b09f1-48lhp" Oct 30 16:40:45.908690 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:45.908661 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9abe55b2-ad36-4d9c-8b40-001d1c3b5b48-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0b09f1-48lhp\" (UID: \"9abe55b2-ad36-4d9c-8b40-001d1c3b5b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b09f1-48lhp" Oct 30 16:40:45.910456 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:45.910436 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9abe55b2-ad36-4d9c-8b40-001d1c3b5b48-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0b09f1-48lhp\" (UID: \"9abe55b2-ad36-4d9c-8b40-001d1c3b5b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b09f1-48lhp" Oct 30 16:40:45.931074 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:45.931045 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89swn\" (UniqueName: \"kubernetes.io/projected/9abe55b2-ad36-4d9c-8b40-001d1c3b5b48-kube-api-access-89swn\") pod \"maas-api-revoke-keys-e2e-api-a-0b09f1-48lhp\" (UID: \"9abe55b2-ad36-4d9c-8b40-001d1c3b5b48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b09f1-48lhp" Oct 30 16:40:46.019702 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:46.019613 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b09f1-48lhp" Oct 30 16:40:46.330999 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:40:46.330964 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9abe55b2_ad36_4d9c_8b40_001d1c3b5b48.slice/crio-4c940ce09870b136089a2f822d92839d5b2dedf4ab243a1544d3db7591982127 WatchSource:0}: Error finding container 4c940ce09870b136089a2f822d92839d5b2dedf4ab243a1544d3db7591982127: Status 404 returned error can't find the container with id 4c940ce09870b136089a2f822d92839d5b2dedf4ab243a1544d3db7591982127 Oct 30 16:40:46.331418 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:46.331396 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b09f1-48lhp"] Oct 30 16:40:47.267589 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:47.267547 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b09f1-48lhp" event={"ID":"9abe55b2-ad36-4d9c-8b40-001d1c3b5b48","Type":"ContainerStarted","Data":"0da036f25ac77111e7837f047bd4609bfe61641de2fdfcb1cfca0a9efb0915cf"} Oct 30 16:40:47.268065 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:47.267616 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b09f1-48lhp" event={"ID":"9abe55b2-ad36-4d9c-8b40-001d1c3b5b48","Type":"ContainerStarted","Data":"4c940ce09870b136089a2f822d92839d5b2dedf4ab243a1544d3db7591982127"} Oct 30 16:40:47.313580 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:47.313520 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b09f1-48lhp" podStartSLOduration=2.31349996 podStartE2EDuration="2.31349996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:40:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:40:47.307644647 +0000 UTC m=+1074.534819761" watchObservedRunningTime="2026-10-30 16:40:47.31349996 +0000 UTC m=+1074.540675061" Oct 30 16:40:48.277116 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:48.277077 4493 generic.go:356] "Generic (PLEG): container finished" podID="9abe55b2-ad36-4d9c-8b40-001d1c3b5b48" containerID="0da036f25ac77111e7837f047bd4609bfe61641de2fdfcb1cfca0a9efb0915cf" exitCode=0 Oct 30 16:40:48.277507 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:48.277160 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b09f1-48lhp" event={"ID":"9abe55b2-ad36-4d9c-8b40-001d1c3b5b48","Type":"ContainerDied","Data":"0da036f25ac77111e7837f047bd4609bfe61641de2fdfcb1cfca0a9efb0915cf"} Oct 30 16:40:49.446672 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:49.446647 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b09f1-48lhp" Oct 30 16:40:49.573212 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:49.573136 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9abe55b2-ad36-4d9c-8b40-001d1c3b5b48-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9abe55b2-ad36-4d9c-8b40-001d1c3b5b48-maas-api-xks-ca\") pod \"9abe55b2-ad36-4d9c-8b40-001d1c3b5b48\" (UID: \"9abe55b2-ad36-4d9c-8b40-001d1c3b5b48\") " Oct 30 16:40:49.573212 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:49.573187 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9abe55b2-ad36-4d9c-8b40-001d1c3b5b48-kube-api-access-89swn\" (UniqueName: \"kubernetes.io/projected/9abe55b2-ad36-4d9c-8b40-001d1c3b5b48-kube-api-access-89swn\") pod \"9abe55b2-ad36-4d9c-8b40-001d1c3b5b48\" (UID: \"9abe55b2-ad36-4d9c-8b40-001d1c3b5b48\") " Oct 30 16:40:49.573446 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:49.573226 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9abe55b2-ad36-4d9c-8b40-001d1c3b5b48-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9abe55b2-ad36-4d9c-8b40-001d1c3b5b48-maas-api-openshift-ca\") pod \"9abe55b2-ad36-4d9c-8b40-001d1c3b5b48\" (UID: \"9abe55b2-ad36-4d9c-8b40-001d1c3b5b48\") " Oct 30 16:40:49.573685 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:49.573661 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9abe55b2-ad36-4d9c-8b40-001d1c3b5b48-maas-api-openshift-ca" pod "9abe55b2-ad36-4d9c-8b40-001d1c3b5b48" (UID: "9abe55b2-ad36-4d9c-8b40-001d1c3b5b48"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:40:49.575442 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:49.575416 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9abe55b2-ad36-4d9c-8b40-001d1c3b5b48-maas-api-xks-ca" pod "9abe55b2-ad36-4d9c-8b40-001d1c3b5b48" (UID: "9abe55b2-ad36-4d9c-8b40-001d1c3b5b48"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:40:49.575442 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:49.575429 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9abe55b2-ad36-4d9c-8b40-001d1c3b5b48-kube-api-access-89swn" pod "9abe55b2-ad36-4d9c-8b40-001d1c3b5b48" (UID: "9abe55b2-ad36-4d9c-8b40-001d1c3b5b48"). InnerVolumeSpecName "kube-api-access-89swn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:40:49.674543 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:49.674496 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9abe55b2-ad36-4d9c-8b40-001d1c3b5b48-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:40:49.674543 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:49.674538 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89swn\" (UniqueName: \"kubernetes.io/projected/9abe55b2-ad36-4d9c-8b40-001d1c3b5b48-kube-api-access-89swn\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:40:49.674725 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:49.674584 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9abe55b2-ad36-4d9c-8b40-001d1c3b5b48-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:40:50.336811 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:50.333291 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b09f1-48lhp" event={"ID":"9abe55b2-ad36-4d9c-8b40-001d1c3b5b48","Type":"ContainerDied","Data":"4c940ce09870b136089a2f822d92839d5b2dedf4ab243a1544d3db7591982127"} Oct 30 16:40:50.336811 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:50.333358 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c940ce09870b136089a2f822d92839d5b2dedf4ab243a1544d3db7591982127" Oct 30 16:40:50.336811 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:50.333424 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b09f1-48lhp" Oct 30 16:40:53.299504 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:53.299473 4493 scope.go:122] "RemoveContainer" containerID="b2ca2aba6b860ae3720203ab373e415d3281529b8839249465b2a1df759ca9fe" Oct 30 16:40:53.312774 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:40:53.312746 4493 scope.go:122] "RemoveContainer" containerID="6c052f30460205f0b4c9bb45d4698085e5ecd45e2068fcf6f4b9ab3f9f5a5c25" Oct 30 16:41:00.346109 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:41:00.346070 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 16:41:10.203940 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:10.203907 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 16:41:20.496092 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:20.495006 4493 patch_prober.go:29] interesting pod/e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": read tcp 10.134.0.2:51468->10.134.0.115:15021: read: connection reset by peer" start-of-body= Oct 30 16:41:20.496092 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:20.495065 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q" podUID="386d43ba-11f5-440a-ab13-d6b444bacaae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": read tcp 10.134.0.2:51468->10.134.0.115:15021: read: connection reset by peer" Oct 30 16:41:20.620526 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:20.620489 4493 generic.go:356] "Generic (PLEG): container finished" podID="386d43ba-11f5-440a-ab13-d6b444bacaae" containerID="9239688fe498dfdcd8b715a6e0d1b6caaffe02cad2c44bf435e781bec69579de" exitCode=0 Oct 30 16:41:20.620698 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:20.620561 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q" event={"ID":"386d43ba-11f5-440a-ab13-d6b444bacaae","Type":"ContainerDied","Data":"9239688fe498dfdcd8b715a6e0d1b6caaffe02cad2c44bf435e781bec69579de"} Oct 30 16:41:20.621096 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:20.621078 4493 scope.go:122] "RemoveContainer" containerID="9239688fe498dfdcd8b715a6e0d1b6caaffe02cad2c44bf435e781bec69579de" Oct 30 16:41:20.622772 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:20.622746 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:41:20.622863 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:20.622841 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:41:20.622909 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:20.622877 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:41:21.639106 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:21.639073 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q" event={"ID":"386d43ba-11f5-440a-ab13-d6b444bacaae","Type":"ContainerStarted","Data":"5bac4d79cfebef6bc5690e3bc9ecd8f76d365fda174b01c0750d948f63fad880"} Oct 30 16:41:22.376840 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:22.376190 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q" Oct 30 16:41:22.376840 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:22.376316 4493 patch_prober.go:29] interesting pod/e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 16:41:22.376840 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:22.376372 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q" podUID="386d43ba-11f5-440a-ab13-d6b444bacaae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 16:41:23.036827 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:23.036771 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a49e5776-qv4xq"] Oct 30 16:41:23.045609 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:23.045584 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a49e5776-qv4xq"] Oct 30 16:41:23.377319 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:23.377230 4493 patch_prober.go:29] interesting pod/e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 16:41:23.377835 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:23.377761 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q" podUID="386d43ba-11f5-440a-ab13-d6b444bacaae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 16:41:24.375185 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:24.375150 4493 patch_prober.go:29] interesting pod/e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 16:41:24.375701 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:24.375201 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q" podUID="386d43ba-11f5-440a-ab13-d6b444bacaae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 16:41:24.939201 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:24.939162 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72b50e48-9b41-4994-b2e2-4e328586edb2" path="/var/lib/kubelet/pods/72b50e48-9b41-4994-b2e2-4e328586edb2/volumes" Oct 30 16:41:25.375025 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:25.374942 4493 patch_prober.go:29] interesting pod/e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 16:41:25.375025 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:25.374992 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q" podUID="386d43ba-11f5-440a-ab13-d6b444bacaae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 16:41:26.374940 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:26.374908 4493 patch_prober.go:29] interesting pod/e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 16:41:26.375334 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:26.374958 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q" podUID="386d43ba-11f5-440a-ab13-d6b444bacaae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 16:41:27.378945 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:27.378918 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q" Oct 30 16:41:27.379447 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:27.379428 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q" Oct 30 16:41:27.380209 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:27.380185 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q" Oct 30 16:41:27.993501 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:27.993466 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82381960-t9wgm"] Oct 30 16:41:27.994440 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:27.994414 4493 cpu_manager.go:414] "containerMap: removing container" podUID="9abe55b2-ad36-4d9c-8b40-001d1c3b5b48" containerName="revoke-keys" Oct 30 16:41:27.994440 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:27.994439 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9abe55b2-ad36-4d9c-8b40-001d1c3b5b48" containerName="revoke-keys" Oct 30 16:41:27.994626 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:27.994611 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="9abe55b2-ad36-4d9c-8b40-001d1c3b5b48" containerName="revoke-keys" Oct 30 16:41:27.998438 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:27.998422 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82381960-t9wgm" Oct 30 16:41:28.022935 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:28.022889 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82381960-t9wgm"] Oct 30 16:41:28.028288 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:28.028261 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:41:28.028913 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:28.028890 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:41:28.033292 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:28.029013 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 16:41:28.033292 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:28.029092 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:41:28.095475 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:28.095442 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f35f8ab5-990b-4031-8b8e-16cec0f201a7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-82381960-t9wgm\" (UID: \"f35f8ab5-990b-4031-8b8e-16cec0f201a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82381960-t9wgm" Oct 30 16:41:28.095672 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:28.095529 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgqnx\" (UniqueName: \"kubernetes.io/projected/f35f8ab5-990b-4031-8b8e-16cec0f201a7-kube-api-access-zgqnx\") pod \"maas-api-revoke-keys-e2e-mt-82381960-t9wgm\" (UID: \"f35f8ab5-990b-4031-8b8e-16cec0f201a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82381960-t9wgm" Oct 30 16:41:28.095672 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:28.095632 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f35f8ab5-990b-4031-8b8e-16cec0f201a7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-82381960-t9wgm\" (UID: \"f35f8ab5-990b-4031-8b8e-16cec0f201a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82381960-t9wgm" Oct 30 16:41:28.197558 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:28.197520 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f35f8ab5-990b-4031-8b8e-16cec0f201a7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-82381960-t9wgm\" (UID: \"f35f8ab5-990b-4031-8b8e-16cec0f201a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82381960-t9wgm" Oct 30 16:41:28.197728 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:28.197638 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgqnx\" (UniqueName: \"kubernetes.io/projected/f35f8ab5-990b-4031-8b8e-16cec0f201a7-kube-api-access-zgqnx\") pod \"maas-api-revoke-keys-e2e-mt-82381960-t9wgm\" (UID: \"f35f8ab5-990b-4031-8b8e-16cec0f201a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82381960-t9wgm" Oct 30 16:41:28.197728 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:28.197688 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f35f8ab5-990b-4031-8b8e-16cec0f201a7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-82381960-t9wgm\" (UID: \"f35f8ab5-990b-4031-8b8e-16cec0f201a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82381960-t9wgm" Oct 30 16:41:28.198320 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:28.198291 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f35f8ab5-990b-4031-8b8e-16cec0f201a7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-82381960-t9wgm\" (UID: \"f35f8ab5-990b-4031-8b8e-16cec0f201a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82381960-t9wgm" Oct 30 16:41:28.200307 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:28.200286 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f35f8ab5-990b-4031-8b8e-16cec0f201a7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-82381960-t9wgm\" (UID: \"f35f8ab5-990b-4031-8b8e-16cec0f201a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82381960-t9wgm" Oct 30 16:41:28.220641 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:28.220612 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgqnx\" (UniqueName: \"kubernetes.io/projected/f35f8ab5-990b-4031-8b8e-16cec0f201a7-kube-api-access-zgqnx\") pod \"maas-api-revoke-keys-e2e-mt-82381960-t9wgm\" (UID: \"f35f8ab5-990b-4031-8b8e-16cec0f201a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82381960-t9wgm" Oct 30 16:41:28.343862 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:28.342700 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82381960-t9wgm" Oct 30 16:41:28.774210 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:28.774183 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82381960-t9wgm"] Oct 30 16:41:28.774534 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:41:28.774188 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf35f8ab5_990b_4031_8b8e_16cec0f201a7.slice/crio-e993fceefeb1140e984685ce9d5a8b6d4dda3ec2d11204fc13a9f56b5a842bee WatchSource:0}: Error finding container e993fceefeb1140e984685ce9d5a8b6d4dda3ec2d11204fc13a9f56b5a842bee: Status 404 returned error can't find the container with id e993fceefeb1140e984685ce9d5a8b6d4dda3ec2d11204fc13a9f56b5a842bee Oct 30 16:41:29.744238 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:29.744209 4493 generic.go:356] "Generic (PLEG): container finished" podID="f35f8ab5-990b-4031-8b8e-16cec0f201a7" containerID="eab562a57122e92a51f90c1da6a0a8eb2100f243cad1c3ef2917b90a74a7bf9b" exitCode=0 Oct 30 16:41:29.744377 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:29.744290 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82381960-t9wgm" event={"ID":"f35f8ab5-990b-4031-8b8e-16cec0f201a7","Type":"ContainerDied","Data":"eab562a57122e92a51f90c1da6a0a8eb2100f243cad1c3ef2917b90a74a7bf9b"} Oct 30 16:41:29.744377 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:29.744353 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82381960-t9wgm" event={"ID":"f35f8ab5-990b-4031-8b8e-16cec0f201a7","Type":"ContainerStarted","Data":"e993fceefeb1140e984685ce9d5a8b6d4dda3ec2d11204fc13a9f56b5a842bee"} Oct 30 16:41:30.342404 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:41:30.342356 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 16:41:30.909875 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:30.909837 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82381960-t9wgm" Oct 30 16:41:30.963993 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:30.963962 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f35f8ab5-990b-4031-8b8e-16cec0f201a7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f35f8ab5-990b-4031-8b8e-16cec0f201a7-maas-api-openshift-ca\") pod \"f35f8ab5-990b-4031-8b8e-16cec0f201a7\" (UID: \"f35f8ab5-990b-4031-8b8e-16cec0f201a7\") " Oct 30 16:41:30.964196 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:30.964065 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f35f8ab5-990b-4031-8b8e-16cec0f201a7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f35f8ab5-990b-4031-8b8e-16cec0f201a7-maas-api-xks-ca\") pod \"f35f8ab5-990b-4031-8b8e-16cec0f201a7\" (UID: \"f35f8ab5-990b-4031-8b8e-16cec0f201a7\") " Oct 30 16:41:30.964196 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:30.964158 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f35f8ab5-990b-4031-8b8e-16cec0f201a7-kube-api-access-zgqnx\" (UniqueName: \"kubernetes.io/projected/f35f8ab5-990b-4031-8b8e-16cec0f201a7-kube-api-access-zgqnx\") pod \"f35f8ab5-990b-4031-8b8e-16cec0f201a7\" (UID: \"f35f8ab5-990b-4031-8b8e-16cec0f201a7\") " Oct 30 16:41:30.964404 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:30.964374 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f35f8ab5-990b-4031-8b8e-16cec0f201a7-maas-api-openshift-ca" pod "f35f8ab5-990b-4031-8b8e-16cec0f201a7" (UID: "f35f8ab5-990b-4031-8b8e-16cec0f201a7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:41:30.964766 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:30.964738 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f35f8ab5-990b-4031-8b8e-16cec0f201a7-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:41:30.967526 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:30.967487 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f35f8ab5-990b-4031-8b8e-16cec0f201a7-maas-api-xks-ca" pod "f35f8ab5-990b-4031-8b8e-16cec0f201a7" (UID: "f35f8ab5-990b-4031-8b8e-16cec0f201a7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:41:30.967526 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:30.967510 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f35f8ab5-990b-4031-8b8e-16cec0f201a7-kube-api-access-zgqnx" pod "f35f8ab5-990b-4031-8b8e-16cec0f201a7" (UID: "f35f8ab5-990b-4031-8b8e-16cec0f201a7"). InnerVolumeSpecName "kube-api-access-zgqnx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:41:31.066353 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:31.066284 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f35f8ab5-990b-4031-8b8e-16cec0f201a7-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:41:31.066353 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:31.066311 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgqnx\" (UniqueName: \"kubernetes.io/projected/f35f8ab5-990b-4031-8b8e-16cec0f201a7-kube-api-access-zgqnx\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:41:31.762513 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:31.762481 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82381960-t9wgm" Oct 30 16:41:31.762939 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:31.762519 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82381960-t9wgm" event={"ID":"f35f8ab5-990b-4031-8b8e-16cec0f201a7","Type":"ContainerDied","Data":"e993fceefeb1140e984685ce9d5a8b6d4dda3ec2d11204fc13a9f56b5a842bee"} Oct 30 16:41:31.762939 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:31.762564 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e993fceefeb1140e984685ce9d5a8b6d4dda3ec2d11204fc13a9f56b5a842bee" Oct 30 16:41:34.117906 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:34.117872 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8295b2-dhz5d"] Oct 30 16:41:34.127875 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:34.127842 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8295b2-dhz5d"] Oct 30 16:41:34.943808 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:34.942634 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5dfac39d-8c50-469d-a0e0-38fdc16889d5" path="/var/lib/kubelet/pods/5dfac39d-8c50-469d-a0e0-38fdc16889d5/volumes" Oct 30 16:41:37.034198 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:37.034165 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b09f1-48lhp"] Oct 30 16:41:37.045336 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:37.045312 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0b09f1-48lhp"] Oct 30 16:41:38.935536 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:38.935497 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9abe55b2-ad36-4d9c-8b40-001d1c3b5b48" path="/var/lib/kubelet/pods/9abe55b2-ad36-4d9c-8b40-001d1c3b5b48/volumes" Oct 30 16:41:40.857670 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:40.857633 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8077da2e-xt2z9"] Oct 30 16:41:40.858278 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:40.858263 4493 cpu_manager.go:414] "containerMap: removing container" podUID="f35f8ab5-990b-4031-8b8e-16cec0f201a7" containerName="revoke-keys" Oct 30 16:41:40.858326 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:40.858279 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f35f8ab5-990b-4031-8b8e-16cec0f201a7" containerName="revoke-keys" Oct 30 16:41:40.858384 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:40.858375 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="f35f8ab5-990b-4031-8b8e-16cec0f201a7" containerName="revoke-keys" Oct 30 16:41:40.861425 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:40.861410 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8077da2e-xt2z9" Oct 30 16:41:40.889008 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:40.888972 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:41:40.889194 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:40.889079 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:41:40.889194 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:40.889177 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:41:40.889431 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:40.889392 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 16:41:40.895056 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:40.895031 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8077da2e-xt2z9"] Oct 30 16:41:41.054662 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:41.054625 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3a489ce-69b9-44fb-8f42-db7c25762d3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8077da2e-xt2z9\" (UID: \"a3a489ce-69b9-44fb-8f42-db7c25762d3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8077da2e-xt2z9" Oct 30 16:41:41.054896 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:41.054865 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgddm\" (UniqueName: \"kubernetes.io/projected/a3a489ce-69b9-44fb-8f42-db7c25762d3c-kube-api-access-pgddm\") pod \"maas-api-revoke-sub-8077da2e-xt2z9\" (UID: \"a3a489ce-69b9-44fb-8f42-db7c25762d3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8077da2e-xt2z9" Oct 30 16:41:41.054995 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:41.054979 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3a489ce-69b9-44fb-8f42-db7c25762d3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8077da2e-xt2z9\" (UID: \"a3a489ce-69b9-44fb-8f42-db7c25762d3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8077da2e-xt2z9" Oct 30 16:41:41.156424 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:41.156385 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgddm\" (UniqueName: \"kubernetes.io/projected/a3a489ce-69b9-44fb-8f42-db7c25762d3c-kube-api-access-pgddm\") pod \"maas-api-revoke-sub-8077da2e-xt2z9\" (UID: \"a3a489ce-69b9-44fb-8f42-db7c25762d3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8077da2e-xt2z9" Oct 30 16:41:41.156589 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:41.156493 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3a489ce-69b9-44fb-8f42-db7c25762d3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8077da2e-xt2z9\" (UID: \"a3a489ce-69b9-44fb-8f42-db7c25762d3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8077da2e-xt2z9" Oct 30 16:41:41.156662 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:41.156635 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3a489ce-69b9-44fb-8f42-db7c25762d3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8077da2e-xt2z9\" (UID: \"a3a489ce-69b9-44fb-8f42-db7c25762d3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8077da2e-xt2z9" Oct 30 16:41:41.157317 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:41.157290 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3a489ce-69b9-44fb-8f42-db7c25762d3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8077da2e-xt2z9\" (UID: \"a3a489ce-69b9-44fb-8f42-db7c25762d3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8077da2e-xt2z9" Oct 30 16:41:41.159220 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:41.159195 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3a489ce-69b9-44fb-8f42-db7c25762d3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8077da2e-xt2z9\" (UID: \"a3a489ce-69b9-44fb-8f42-db7c25762d3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8077da2e-xt2z9" Oct 30 16:41:41.179618 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:41.179584 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgddm\" (UniqueName: \"kubernetes.io/projected/a3a489ce-69b9-44fb-8f42-db7c25762d3c-kube-api-access-pgddm\") pod \"maas-api-revoke-sub-8077da2e-xt2z9\" (UID: \"a3a489ce-69b9-44fb-8f42-db7c25762d3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8077da2e-xt2z9" Oct 30 16:41:41.475852 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:41.475761 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8077da2e-xt2z9" Oct 30 16:41:41.826448 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:41.826412 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8077da2e-xt2z9"] Oct 30 16:41:41.828524 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:41:41.828494 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3a489ce_69b9_44fb_8f42_db7c25762d3c.slice/crio-9db92a44d868b97cbc5b7c61263bbd0401ad467f79610e5b1e83718bd66878cd WatchSource:0}: Error finding container 9db92a44d868b97cbc5b7c61263bbd0401ad467f79610e5b1e83718bd66878cd: Status 404 returned error can't find the container with id 9db92a44d868b97cbc5b7c61263bbd0401ad467f79610e5b1e83718bd66878cd Oct 30 16:41:41.867263 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:41.867225 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8077da2e-xt2z9" event={"ID":"a3a489ce-69b9-44fb-8f42-db7c25762d3c","Type":"ContainerStarted","Data":"9db92a44d868b97cbc5b7c61263bbd0401ad467f79610e5b1e83718bd66878cd"} Oct 30 16:41:42.112634 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:42.112533 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-90df54-qtjq7"] Oct 30 16:41:42.119190 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:42.119167 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-90df54-qtjq7" Oct 30 16:41:42.137741 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:42.137713 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-90df54-qtjq7"] Oct 30 16:41:42.176167 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:42.176132 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gn86\" (UniqueName: \"kubernetes.io/projected/a8a25853-2409-46b4-a516-525f8b8043d2-kube-api-access-4gn86\") pod \"maas-api-revoke-keys-e2e-api-b-90df54-qtjq7\" (UID: \"a8a25853-2409-46b4-a516-525f8b8043d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-90df54-qtjq7" Oct 30 16:41:42.176342 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:42.176177 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8a25853-2409-46b4-a516-525f8b8043d2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-90df54-qtjq7\" (UID: \"a8a25853-2409-46b4-a516-525f8b8043d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-90df54-qtjq7" Oct 30 16:41:42.176419 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:42.176369 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8a25853-2409-46b4-a516-525f8b8043d2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-90df54-qtjq7\" (UID: \"a8a25853-2409-46b4-a516-525f8b8043d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-90df54-qtjq7" Oct 30 16:41:42.277363 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:42.277331 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gn86\" (UniqueName: \"kubernetes.io/projected/a8a25853-2409-46b4-a516-525f8b8043d2-kube-api-access-4gn86\") pod \"maas-api-revoke-keys-e2e-api-b-90df54-qtjq7\" (UID: \"a8a25853-2409-46b4-a516-525f8b8043d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-90df54-qtjq7" Oct 30 16:41:42.277541 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:42.277374 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8a25853-2409-46b4-a516-525f8b8043d2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-90df54-qtjq7\" (UID: \"a8a25853-2409-46b4-a516-525f8b8043d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-90df54-qtjq7" Oct 30 16:41:42.277541 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:42.277459 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8a25853-2409-46b4-a516-525f8b8043d2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-90df54-qtjq7\" (UID: \"a8a25853-2409-46b4-a516-525f8b8043d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-90df54-qtjq7" Oct 30 16:41:42.278214 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:42.278194 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8a25853-2409-46b4-a516-525f8b8043d2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-90df54-qtjq7\" (UID: \"a8a25853-2409-46b4-a516-525f8b8043d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-90df54-qtjq7" Oct 30 16:41:42.279954 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:42.279936 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8a25853-2409-46b4-a516-525f8b8043d2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-90df54-qtjq7\" (UID: \"a8a25853-2409-46b4-a516-525f8b8043d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-90df54-qtjq7" Oct 30 16:41:42.300247 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:42.300224 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gn86\" (UniqueName: \"kubernetes.io/projected/a8a25853-2409-46b4-a516-525f8b8043d2-kube-api-access-4gn86\") pod \"maas-api-revoke-keys-e2e-api-b-90df54-qtjq7\" (UID: \"a8a25853-2409-46b4-a516-525f8b8043d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-90df54-qtjq7" Oct 30 16:41:42.433931 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:42.433899 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-90df54-qtjq7" Oct 30 16:41:42.749001 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:42.748974 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-90df54-qtjq7"] Oct 30 16:41:42.783707 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:41:42.783672 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8a25853_2409_46b4_a516_525f8b8043d2.slice/crio-94cecf71abfd49cf86bbd87c88f14042f69d15b5fdfaeba933572ad746e64299 WatchSource:0}: Error finding container 94cecf71abfd49cf86bbd87c88f14042f69d15b5fdfaeba933572ad746e64299: Status 404 returned error can't find the container with id 94cecf71abfd49cf86bbd87c88f14042f69d15b5fdfaeba933572ad746e64299 Oct 30 16:41:42.877330 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:42.877298 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-90df54-qtjq7" event={"ID":"a8a25853-2409-46b4-a516-525f8b8043d2","Type":"ContainerStarted","Data":"94cecf71abfd49cf86bbd87c88f14042f69d15b5fdfaeba933572ad746e64299"} Oct 30 16:41:42.879083 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:42.879054 4493 generic.go:356] "Generic (PLEG): container finished" podID="a3a489ce-69b9-44fb-8f42-db7c25762d3c" containerID="887dda815dd0e14d0c496d00c6038fb3b25facbf2e286af1238f99acbbabc6d7" exitCode=0 Oct 30 16:41:42.879206 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:42.879120 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8077da2e-xt2z9" event={"ID":"a3a489ce-69b9-44fb-8f42-db7c25762d3c","Type":"ContainerDied","Data":"887dda815dd0e14d0c496d00c6038fb3b25facbf2e286af1238f99acbbabc6d7"} Oct 30 16:41:43.888580 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:43.888549 4493 generic.go:356] "Generic (PLEG): container finished" podID="a8a25853-2409-46b4-a516-525f8b8043d2" containerID="8ad5bc622b1639235757cb8e7e0848634750b8e103bfa77bcaae2439c2efd7ad" exitCode=0 Oct 30 16:41:43.888970 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:43.888691 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-90df54-qtjq7" event={"ID":"a8a25853-2409-46b4-a516-525f8b8043d2","Type":"ContainerDied","Data":"8ad5bc622b1639235757cb8e7e0848634750b8e103bfa77bcaae2439c2efd7ad"} Oct 30 16:41:44.067446 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:44.067420 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8077da2e-xt2z9" Oct 30 16:41:44.099423 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:44.099397 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3a489ce-69b9-44fb-8f42-db7c25762d3c-kube-api-access-pgddm\" (UniqueName: \"kubernetes.io/projected/a3a489ce-69b9-44fb-8f42-db7c25762d3c-kube-api-access-pgddm\") pod \"a3a489ce-69b9-44fb-8f42-db7c25762d3c\" (UID: \"a3a489ce-69b9-44fb-8f42-db7c25762d3c\") " Oct 30 16:41:44.099679 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:44.099658 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3a489ce-69b9-44fb-8f42-db7c25762d3c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3a489ce-69b9-44fb-8f42-db7c25762d3c-maas-api-xks-ca\") pod \"a3a489ce-69b9-44fb-8f42-db7c25762d3c\" (UID: \"a3a489ce-69b9-44fb-8f42-db7c25762d3c\") " Oct 30 16:41:44.099722 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:44.099710 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3a489ce-69b9-44fb-8f42-db7c25762d3c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3a489ce-69b9-44fb-8f42-db7c25762d3c-maas-api-openshift-ca\") pod \"a3a489ce-69b9-44fb-8f42-db7c25762d3c\" (UID: \"a3a489ce-69b9-44fb-8f42-db7c25762d3c\") " Oct 30 16:41:44.100117 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:44.100090 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3a489ce-69b9-44fb-8f42-db7c25762d3c-maas-api-openshift-ca" pod "a3a489ce-69b9-44fb-8f42-db7c25762d3c" (UID: "a3a489ce-69b9-44fb-8f42-db7c25762d3c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:41:44.100295 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:44.100281 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3a489ce-69b9-44fb-8f42-db7c25762d3c-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:41:44.101611 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:44.101590 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3a489ce-69b9-44fb-8f42-db7c25762d3c-kube-api-access-pgddm" pod "a3a489ce-69b9-44fb-8f42-db7c25762d3c" (UID: "a3a489ce-69b9-44fb-8f42-db7c25762d3c"). InnerVolumeSpecName "kube-api-access-pgddm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:41:44.101858 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:44.101825 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3a489ce-69b9-44fb-8f42-db7c25762d3c-maas-api-xks-ca" pod "a3a489ce-69b9-44fb-8f42-db7c25762d3c" (UID: "a3a489ce-69b9-44fb-8f42-db7c25762d3c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:41:44.201510 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:44.201477 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgddm\" (UniqueName: \"kubernetes.io/projected/a3a489ce-69b9-44fb-8f42-db7c25762d3c-kube-api-access-pgddm\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:41:44.201510 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:44.201508 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3a489ce-69b9-44fb-8f42-db7c25762d3c-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:41:44.903948 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:44.903917 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8077da2e-xt2z9" Oct 30 16:41:44.907234 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:44.907201 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8077da2e-xt2z9" event={"ID":"a3a489ce-69b9-44fb-8f42-db7c25762d3c","Type":"ContainerDied","Data":"9db92a44d868b97cbc5b7c61263bbd0401ad467f79610e5b1e83718bd66878cd"} Oct 30 16:41:44.907352 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:44.907263 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9db92a44d868b97cbc5b7c61263bbd0401ad467f79610e5b1e83718bd66878cd" Oct 30 16:41:45.080195 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:45.080166 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-90df54-qtjq7" Oct 30 16:41:45.119456 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:45.119431 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8a25853-2409-46b4-a516-525f8b8043d2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8a25853-2409-46b4-a516-525f8b8043d2-maas-api-xks-ca\") pod \"a8a25853-2409-46b4-a516-525f8b8043d2\" (UID: \"a8a25853-2409-46b4-a516-525f8b8043d2\") " Oct 30 16:41:45.119604 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:45.119486 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8a25853-2409-46b4-a516-525f8b8043d2-kube-api-access-4gn86\" (UniqueName: \"kubernetes.io/projected/a8a25853-2409-46b4-a516-525f8b8043d2-kube-api-access-4gn86\") pod \"a8a25853-2409-46b4-a516-525f8b8043d2\" (UID: \"a8a25853-2409-46b4-a516-525f8b8043d2\") " Oct 30 16:41:45.119604 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:45.119564 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8a25853-2409-46b4-a516-525f8b8043d2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8a25853-2409-46b4-a516-525f8b8043d2-maas-api-openshift-ca\") pod \"a8a25853-2409-46b4-a516-525f8b8043d2\" (UID: \"a8a25853-2409-46b4-a516-525f8b8043d2\") " Oct 30 16:41:45.119986 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:45.119955 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8a25853-2409-46b4-a516-525f8b8043d2-maas-api-openshift-ca" pod "a8a25853-2409-46b4-a516-525f8b8043d2" (UID: "a8a25853-2409-46b4-a516-525f8b8043d2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:41:45.120467 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:45.120442 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8a25853-2409-46b4-a516-525f8b8043d2-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:41:45.121917 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:45.121889 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8a25853-2409-46b4-a516-525f8b8043d2-kube-api-access-4gn86" pod "a8a25853-2409-46b4-a516-525f8b8043d2" (UID: "a8a25853-2409-46b4-a516-525f8b8043d2"). InnerVolumeSpecName "kube-api-access-4gn86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:41:45.122061 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:45.122028 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8a25853-2409-46b4-a516-525f8b8043d2-maas-api-xks-ca" pod "a8a25853-2409-46b4-a516-525f8b8043d2" (UID: "a8a25853-2409-46b4-a516-525f8b8043d2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:41:45.229881 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:45.229756 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8a25853-2409-46b4-a516-525f8b8043d2-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:41:45.230099 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:45.230081 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gn86\" (UniqueName: \"kubernetes.io/projected/a8a25853-2409-46b4-a516-525f8b8043d2-kube-api-access-4gn86\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:41:45.254690 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:45.254659 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-83d885-rphbq"] Oct 30 16:41:45.255310 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:45.255294 4493 cpu_manager.go:414] "containerMap: removing container" podUID="a8a25853-2409-46b4-a516-525f8b8043d2" containerName="revoke-keys" Oct 30 16:41:45.255363 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:45.255311 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8a25853-2409-46b4-a516-525f8b8043d2" containerName="revoke-keys" Oct 30 16:41:45.255363 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:45.255347 4493 cpu_manager.go:414] "containerMap: removing container" podUID="a3a489ce-69b9-44fb-8f42-db7c25762d3c" containerName="revoke-keys" Oct 30 16:41:45.255363 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:45.255352 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3a489ce-69b9-44fb-8f42-db7c25762d3c" containerName="revoke-keys" Oct 30 16:41:45.255464 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:45.255451 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8a25853-2409-46b4-a516-525f8b8043d2" containerName="revoke-keys" Oct 30 16:41:45.255464 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:45.255461 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3a489ce-69b9-44fb-8f42-db7c25762d3c" containerName="revoke-keys" Oct 30 16:41:45.259163 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:45.259145 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-83d885-rphbq" Oct 30 16:41:45.279992 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:45.279966 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-83d885-rphbq"] Oct 30 16:41:45.331151 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:45.331122 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3905e43-1ec2-402f-a874-aaf7b84cb4ce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-83d885-rphbq\" (UID: \"f3905e43-1ec2-402f-a874-aaf7b84cb4ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-83d885-rphbq" Oct 30 16:41:45.331322 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:45.331192 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s82lx\" (UniqueName: \"kubernetes.io/projected/f3905e43-1ec2-402f-a874-aaf7b84cb4ce-kube-api-access-s82lx\") pod \"maas-api-revoke-keys-e2e-api-b-83d885-rphbq\" (UID: \"f3905e43-1ec2-402f-a874-aaf7b84cb4ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-83d885-rphbq" Oct 30 16:41:45.331365 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:45.331315 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3905e43-1ec2-402f-a874-aaf7b84cb4ce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-83d885-rphbq\" (UID: \"f3905e43-1ec2-402f-a874-aaf7b84cb4ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-83d885-rphbq" Oct 30 16:41:45.434542 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:45.434500 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s82lx\" (UniqueName: \"kubernetes.io/projected/f3905e43-1ec2-402f-a874-aaf7b84cb4ce-kube-api-access-s82lx\") pod \"maas-api-revoke-keys-e2e-api-b-83d885-rphbq\" (UID: \"f3905e43-1ec2-402f-a874-aaf7b84cb4ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-83d885-rphbq" Oct 30 16:41:45.434749 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:45.434601 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3905e43-1ec2-402f-a874-aaf7b84cb4ce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-83d885-rphbq\" (UID: \"f3905e43-1ec2-402f-a874-aaf7b84cb4ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-83d885-rphbq" Oct 30 16:41:45.434977 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:45.434949 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3905e43-1ec2-402f-a874-aaf7b84cb4ce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-83d885-rphbq\" (UID: \"f3905e43-1ec2-402f-a874-aaf7b84cb4ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-83d885-rphbq" Oct 30 16:41:45.435712 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:45.435686 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3905e43-1ec2-402f-a874-aaf7b84cb4ce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-83d885-rphbq\" (UID: \"f3905e43-1ec2-402f-a874-aaf7b84cb4ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-83d885-rphbq" Oct 30 16:41:45.437992 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:45.437967 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3905e43-1ec2-402f-a874-aaf7b84cb4ce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-83d885-rphbq\" (UID: \"f3905e43-1ec2-402f-a874-aaf7b84cb4ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-83d885-rphbq" Oct 30 16:41:45.461941 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:45.461913 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s82lx\" (UniqueName: \"kubernetes.io/projected/f3905e43-1ec2-402f-a874-aaf7b84cb4ce-kube-api-access-s82lx\") pod \"maas-api-revoke-keys-e2e-api-b-83d885-rphbq\" (UID: \"f3905e43-1ec2-402f-a874-aaf7b84cb4ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-83d885-rphbq" Oct 30 16:41:45.573633 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:45.573561 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-83d885-rphbq" Oct 30 16:41:45.889208 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:41:45.889173 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3905e43_1ec2_402f_a874_aaf7b84cb4ce.slice/crio-c4ac42c28cb6ab8d622143ecca28e223ee9c36752bceb8a8e8d95ab51c924772 WatchSource:0}: Error finding container c4ac42c28cb6ab8d622143ecca28e223ee9c36752bceb8a8e8d95ab51c924772: Status 404 returned error can't find the container with id c4ac42c28cb6ab8d622143ecca28e223ee9c36752bceb8a8e8d95ab51c924772 Oct 30 16:41:45.890885 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:45.890855 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-83d885-rphbq"] Oct 30 16:41:45.917920 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:45.917891 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-83d885-rphbq" event={"ID":"f3905e43-1ec2-402f-a874-aaf7b84cb4ce","Type":"ContainerStarted","Data":"c4ac42c28cb6ab8d622143ecca28e223ee9c36752bceb8a8e8d95ab51c924772"} Oct 30 16:41:45.920502 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:45.919693 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-90df54-qtjq7" Oct 30 16:41:45.920502 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:45.919725 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-90df54-qtjq7" event={"ID":"a8a25853-2409-46b4-a516-525f8b8043d2","Type":"ContainerDied","Data":"94cecf71abfd49cf86bbd87c88f14042f69d15b5fdfaeba933572ad746e64299"} Oct 30 16:41:45.920502 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:45.919773 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94cecf71abfd49cf86bbd87c88f14042f69d15b5fdfaeba933572ad746e64299" Oct 30 16:41:46.940818 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:46.940738 4493 generic.go:356] "Generic (PLEG): container finished" podID="f3905e43-1ec2-402f-a874-aaf7b84cb4ce" containerID="44aa3f71f557e86ab87a9047b80e37faf78e6154ad9a6f82fbd3ca159c75bfa1" exitCode=0 Oct 30 16:41:46.941417 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:46.941389 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-83d885-rphbq" event={"ID":"f3905e43-1ec2-402f-a874-aaf7b84cb4ce","Type":"ContainerDied","Data":"44aa3f71f557e86ab87a9047b80e37faf78e6154ad9a6f82fbd3ca159c75bfa1"} Oct 30 16:41:48.116113 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:48.116087 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-83d885-rphbq" Oct 30 16:41:48.175702 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:48.175667 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3905e43-1ec2-402f-a874-aaf7b84cb4ce-kube-api-access-s82lx\" (UniqueName: \"kubernetes.io/projected/f3905e43-1ec2-402f-a874-aaf7b84cb4ce-kube-api-access-s82lx\") pod \"f3905e43-1ec2-402f-a874-aaf7b84cb4ce\" (UID: \"f3905e43-1ec2-402f-a874-aaf7b84cb4ce\") " Oct 30 16:41:48.175899 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:48.175755 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3905e43-1ec2-402f-a874-aaf7b84cb4ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3905e43-1ec2-402f-a874-aaf7b84cb4ce-maas-api-openshift-ca\") pod \"f3905e43-1ec2-402f-a874-aaf7b84cb4ce\" (UID: \"f3905e43-1ec2-402f-a874-aaf7b84cb4ce\") " Oct 30 16:41:48.175971 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:48.175911 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3905e43-1ec2-402f-a874-aaf7b84cb4ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3905e43-1ec2-402f-a874-aaf7b84cb4ce-maas-api-xks-ca\") pod \"f3905e43-1ec2-402f-a874-aaf7b84cb4ce\" (UID: \"f3905e43-1ec2-402f-a874-aaf7b84cb4ce\") " Oct 30 16:41:48.176121 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:48.176096 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3905e43-1ec2-402f-a874-aaf7b84cb4ce-maas-api-openshift-ca" pod "f3905e43-1ec2-402f-a874-aaf7b84cb4ce" (UID: "f3905e43-1ec2-402f-a874-aaf7b84cb4ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:41:48.176517 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:48.176440 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3905e43-1ec2-402f-a874-aaf7b84cb4ce-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:41:48.178069 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:48.178046 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3905e43-1ec2-402f-a874-aaf7b84cb4ce-kube-api-access-s82lx" pod "f3905e43-1ec2-402f-a874-aaf7b84cb4ce" (UID: "f3905e43-1ec2-402f-a874-aaf7b84cb4ce"). InnerVolumeSpecName "kube-api-access-s82lx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:41:48.178470 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:48.178449 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3905e43-1ec2-402f-a874-aaf7b84cb4ce-maas-api-xks-ca" pod "f3905e43-1ec2-402f-a874-aaf7b84cb4ce" (UID: "f3905e43-1ec2-402f-a874-aaf7b84cb4ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:41:48.278188 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:48.278108 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3905e43-1ec2-402f-a874-aaf7b84cb4ce-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:41:48.278188 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:48.278135 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s82lx\" (UniqueName: \"kubernetes.io/projected/f3905e43-1ec2-402f-a874-aaf7b84cb4ce-kube-api-access-s82lx\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:41:48.518608 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:48.518569 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 16:41:48.971188 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:48.971152 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-83d885-rphbq" event={"ID":"f3905e43-1ec2-402f-a874-aaf7b84cb4ce","Type":"ContainerDied","Data":"c4ac42c28cb6ab8d622143ecca28e223ee9c36752bceb8a8e8d95ab51c924772"} Oct 30 16:41:48.971188 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:48.971176 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-83d885-rphbq" Oct 30 16:41:48.971371 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:48.971240 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4ac42c28cb6ab8d622143ecca28e223ee9c36752bceb8a8e8d95ab51c924772" Oct 30 16:41:53.384812 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:53.383045 4493 scope.go:122] "RemoveContainer" containerID="627984d734054ea6bf006e5a196d7fa6bb0df24a8dab8de33af980ee578c421c" Oct 30 16:41:53.434024 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:53.433993 4493 scope.go:122] "RemoveContainer" containerID="fc18f23133a0447c7cff008ffaf82fabd47600cb400ab0cdbc0ab134cd631f82" Oct 30 16:41:53.459731 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:53.459700 4493 scope.go:122] "RemoveContainer" containerID="484bc224d48b1f63815c3d0a06cf09c1ab3e30dd251ebfcbab34c7911eafa91e" Oct 30 16:41:53.473633 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:53.473598 4493 scope.go:122] "RemoveContainer" containerID="74a246e88ffc9741d9f0371caab8edfcedae2c9c80d93d63ca3c91a1dda7ded0" Oct 30 16:41:53.485139 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:53.485108 4493 scope.go:122] "RemoveContainer" containerID="3490bd6648a1e9d6c90087151a9eda007f84fa620eb7a7120bbf7131fd3b4325" Oct 30 16:41:53.495469 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:53.495443 4493 scope.go:122] "RemoveContainer" containerID="5cf4f5028b4a353d38a9d8c0a7a3e79816f6e9aad7811844621d213526734126" Oct 30 16:41:53.505869 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:53.505851 4493 scope.go:122] "RemoveContainer" containerID="b014b6db563656f1959b1e034e7c5433f2dbce62b1650d57521666a0035d5611" Oct 30 16:41:59.373022 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:59.372984 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82381960-t9wgm"] Oct 30 16:41:59.386709 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:41:59.386649 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82381960-t9wgm"] Oct 30 16:42:00.343179 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:42:00.343139 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 16:42:00.934223 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:00.934188 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f35f8ab5-990b-4031-8b8e-16cec0f201a7" path="/var/lib/kubelet/pods/f35f8ab5-990b-4031-8b8e-16cec0f201a7/volumes" Oct 30 16:42:01.034217 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:01.034185 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c0de514-tcr8z"] Oct 30 16:42:01.046795 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:01.046753 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c0de514-tcr8z"] Oct 30 16:42:02.935316 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:02.935278 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="811cd79d-91a9-4d1f-8cdc-5864ac4ca98f" path="/var/lib/kubelet/pods/811cd79d-91a9-4d1f-8cdc-5864ac4ca98f/volumes" Oct 30 16:42:08.032982 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:08.032946 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7410c7-ssbvm"] Oct 30 16:42:08.033532 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:08.033517 4493 cpu_manager.go:414] "containerMap: removing container" podUID="f3905e43-1ec2-402f-a874-aaf7b84cb4ce" containerName="revoke-keys" Oct 30 16:42:08.033532 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:08.033531 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3905e43-1ec2-402f-a874-aaf7b84cb4ce" containerName="revoke-keys" Oct 30 16:42:08.033659 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:08.033646 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3905e43-1ec2-402f-a874-aaf7b84cb4ce" containerName="revoke-keys" Oct 30 16:42:08.039371 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:08.039337 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7410c7-ssbvm" Oct 30 16:42:08.053494 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:08.053473 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:42:08.065368 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:08.065344 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 16:42:08.065502 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:08.065469 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:42:08.065644 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:08.065628 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:42:08.071151 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:08.071121 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7410c7-ssbvm"] Oct 30 16:42:08.112596 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:08.112572 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b29fb352-2ea5-4b64-b598-3a7b6e3d7e0b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7410c7-ssbvm\" (UID: \"b29fb352-2ea5-4b64-b598-3a7b6e3d7e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7410c7-ssbvm" Oct 30 16:42:08.112728 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:08.112614 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b29fb352-2ea5-4b64-b598-3a7b6e3d7e0b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7410c7-ssbvm\" (UID: \"b29fb352-2ea5-4b64-b598-3a7b6e3d7e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7410c7-ssbvm" Oct 30 16:42:08.112841 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:08.112750 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpfj6\" (UniqueName: \"kubernetes.io/projected/b29fb352-2ea5-4b64-b598-3a7b6e3d7e0b-kube-api-access-mpfj6\") pod \"maas-api-revoke-keys-e2e-api-a-7410c7-ssbvm\" (UID: \"b29fb352-2ea5-4b64-b598-3a7b6e3d7e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7410c7-ssbvm" Oct 30 16:42:08.215860 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:08.215076 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b29fb352-2ea5-4b64-b598-3a7b6e3d7e0b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7410c7-ssbvm\" (UID: \"b29fb352-2ea5-4b64-b598-3a7b6e3d7e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7410c7-ssbvm" Oct 30 16:42:08.215860 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:08.215137 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b29fb352-2ea5-4b64-b598-3a7b6e3d7e0b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7410c7-ssbvm\" (UID: \"b29fb352-2ea5-4b64-b598-3a7b6e3d7e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7410c7-ssbvm" Oct 30 16:42:08.215860 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:08.215224 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpfj6\" (UniqueName: \"kubernetes.io/projected/b29fb352-2ea5-4b64-b598-3a7b6e3d7e0b-kube-api-access-mpfj6\") pod \"maas-api-revoke-keys-e2e-api-a-7410c7-ssbvm\" (UID: \"b29fb352-2ea5-4b64-b598-3a7b6e3d7e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7410c7-ssbvm" Oct 30 16:42:08.218012 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:08.217975 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b29fb352-2ea5-4b64-b598-3a7b6e3d7e0b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7410c7-ssbvm\" (UID: \"b29fb352-2ea5-4b64-b598-3a7b6e3d7e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7410c7-ssbvm" Oct 30 16:42:08.223137 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:08.223077 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b29fb352-2ea5-4b64-b598-3a7b6e3d7e0b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7410c7-ssbvm\" (UID: \"b29fb352-2ea5-4b64-b598-3a7b6e3d7e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7410c7-ssbvm" Oct 30 16:42:08.240531 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:08.240501 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpfj6\" (UniqueName: \"kubernetes.io/projected/b29fb352-2ea5-4b64-b598-3a7b6e3d7e0b-kube-api-access-mpfj6\") pod \"maas-api-revoke-keys-e2e-api-a-7410c7-ssbvm\" (UID: \"b29fb352-2ea5-4b64-b598-3a7b6e3d7e0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7410c7-ssbvm" Oct 30 16:42:08.355685 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:08.355600 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7410c7-ssbvm" Oct 30 16:42:08.672715 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:08.672687 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7410c7-ssbvm"] Oct 30 16:42:08.672869 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:42:08.672815 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb29fb352_2ea5_4b64_b598_3a7b6e3d7e0b.slice/crio-76cb2a7d74aa54ee1e7111990bebea64cb6131141806d4bfc80c756f210a3baf WatchSource:0}: Error finding container 76cb2a7d74aa54ee1e7111990bebea64cb6131141806d4bfc80c756f210a3baf: Status 404 returned error can't find the container with id 76cb2a7d74aa54ee1e7111990bebea64cb6131141806d4bfc80c756f210a3baf Oct 30 16:42:09.170582 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:09.170495 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7410c7-ssbvm" event={"ID":"b29fb352-2ea5-4b64-b598-3a7b6e3d7e0b","Type":"ContainerStarted","Data":"e9c58f902473dcda44002369ebdee2bb11b263038dcf0ed4f4d0bf01a01de474"} Oct 30 16:42:09.170582 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:09.170546 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7410c7-ssbvm" event={"ID":"b29fb352-2ea5-4b64-b598-3a7b6e3d7e0b","Type":"ContainerStarted","Data":"76cb2a7d74aa54ee1e7111990bebea64cb6131141806d4bfc80c756f210a3baf"} Oct 30 16:42:09.237303 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:09.237249 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7410c7-ssbvm" podStartSLOduration=1.237234949 podStartE2EDuration="1.237234949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:42:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:42:09.232009956 +0000 UTC m=+1156.459185057" watchObservedRunningTime="2026-10-30 16:42:09.237234949 +0000 UTC m=+1156.464410049" Oct 30 16:42:10.181237 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:10.181202 4493 generic.go:356] "Generic (PLEG): container finished" podID="b29fb352-2ea5-4b64-b598-3a7b6e3d7e0b" containerID="e9c58f902473dcda44002369ebdee2bb11b263038dcf0ed4f4d0bf01a01de474" exitCode=0 Oct 30 16:42:10.181638 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:10.181248 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7410c7-ssbvm" event={"ID":"b29fb352-2ea5-4b64-b598-3a7b6e3d7e0b","Type":"ContainerDied","Data":"e9c58f902473dcda44002369ebdee2bb11b263038dcf0ed4f4d0bf01a01de474"} Oct 30 16:42:11.370552 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:11.370525 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7410c7-ssbvm" Oct 30 16:42:11.465341 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:11.465261 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b29fb352-2ea5-4b64-b598-3a7b6e3d7e0b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b29fb352-2ea5-4b64-b598-3a7b6e3d7e0b-maas-api-openshift-ca\") pod \"b29fb352-2ea5-4b64-b598-3a7b6e3d7e0b\" (UID: \"b29fb352-2ea5-4b64-b598-3a7b6e3d7e0b\") " Oct 30 16:42:11.465508 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:11.465362 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b29fb352-2ea5-4b64-b598-3a7b6e3d7e0b-kube-api-access-mpfj6\" (UniqueName: \"kubernetes.io/projected/b29fb352-2ea5-4b64-b598-3a7b6e3d7e0b-kube-api-access-mpfj6\") pod \"b29fb352-2ea5-4b64-b598-3a7b6e3d7e0b\" (UID: \"b29fb352-2ea5-4b64-b598-3a7b6e3d7e0b\") " Oct 30 16:42:11.465508 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:11.465443 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b29fb352-2ea5-4b64-b598-3a7b6e3d7e0b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b29fb352-2ea5-4b64-b598-3a7b6e3d7e0b-maas-api-xks-ca\") pod \"b29fb352-2ea5-4b64-b598-3a7b6e3d7e0b\" (UID: \"b29fb352-2ea5-4b64-b598-3a7b6e3d7e0b\") " Oct 30 16:42:11.465685 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:11.465651 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b29fb352-2ea5-4b64-b598-3a7b6e3d7e0b-maas-api-openshift-ca" pod "b29fb352-2ea5-4b64-b598-3a7b6e3d7e0b" (UID: "b29fb352-2ea5-4b64-b598-3a7b6e3d7e0b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:42:11.466406 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:11.466370 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b29fb352-2ea5-4b64-b598-3a7b6e3d7e0b-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:42:11.467739 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:11.467717 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b29fb352-2ea5-4b64-b598-3a7b6e3d7e0b-maas-api-xks-ca" pod "b29fb352-2ea5-4b64-b598-3a7b6e3d7e0b" (UID: "b29fb352-2ea5-4b64-b598-3a7b6e3d7e0b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:42:11.467976 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:11.467946 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b29fb352-2ea5-4b64-b598-3a7b6e3d7e0b-kube-api-access-mpfj6" pod "b29fb352-2ea5-4b64-b598-3a7b6e3d7e0b" (UID: "b29fb352-2ea5-4b64-b598-3a7b6e3d7e0b"). InnerVolumeSpecName "kube-api-access-mpfj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:42:11.567999 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:11.567966 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpfj6\" (UniqueName: \"kubernetes.io/projected/b29fb352-2ea5-4b64-b598-3a7b6e3d7e0b-kube-api-access-mpfj6\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:42:11.567999 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:11.567995 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b29fb352-2ea5-4b64-b598-3a7b6e3d7e0b-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:42:12.207140 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:12.207090 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7410c7-ssbvm" Oct 30 16:42:12.207140 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:12.207113 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7410c7-ssbvm" event={"ID":"b29fb352-2ea5-4b64-b598-3a7b6e3d7e0b","Type":"ContainerDied","Data":"76cb2a7d74aa54ee1e7111990bebea64cb6131141806d4bfc80c756f210a3baf"} Oct 30 16:42:12.207512 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:12.207163 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76cb2a7d74aa54ee1e7111990bebea64cb6131141806d4bfc80c756f210a3baf" Oct 30 16:42:15.035901 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:15.035870 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe256302-h86vt"] Oct 30 16:42:15.047576 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:15.047545 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe256302-h86vt"] Oct 30 16:42:16.933473 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:16.933433 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="303b0588-e73c-4d56-abef-328644e8ebe2" path="/var/lib/kubelet/pods/303b0588-e73c-4d56-abef-328644e8ebe2/volumes" Oct 30 16:42:18.966132 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:18.966091 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-83d885-rphbq"] Oct 30 16:42:18.976681 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:18.976654 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-83d885-rphbq"] Oct 30 16:42:19.828662 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:19.828624 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-90df54-qtjq7"] Oct 30 16:42:19.835687 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:19.835649 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-90df54-qtjq7"] Oct 30 16:42:20.936386 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:20.936346 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8a25853-2409-46b4-a516-525f8b8043d2" path="/var/lib/kubelet/pods/a8a25853-2409-46b4-a516-525f8b8043d2/volumes" Oct 30 16:42:20.936986 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:20.936967 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3905e43-1ec2-402f-a874-aaf7b84cb4ce" path="/var/lib/kubelet/pods/f3905e43-1ec2-402f-a874-aaf7b84cb4ce/volumes" Oct 30 16:42:25.034606 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:25.034568 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ffa75a6-4l9c8"] Oct 30 16:42:25.045804 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:25.045748 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ffa75a6-4l9c8"] Oct 30 16:42:26.982777 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:26.982045 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bc3dce4-cd68-4d7c-b0a9-d19859ddfe10" path="/var/lib/kubelet/pods/9bc3dce4-cd68-4d7c-b0a9-d19859ddfe10/volumes" Oct 30 16:42:26.994606 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:26.994568 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 16:42:29.057687 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:29.057653 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ef18671-splkp"] Oct 30 16:42:29.104810 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:29.103926 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ef18671-splkp"] Oct 30 16:42:30.349743 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:42:30.349705 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 16:42:30.933588 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:30.933554 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15735aee-4782-46de-94ea-fb3db181e6cd" path="/var/lib/kubelet/pods/15735aee-4782-46de-94ea-fb3db181e6cd/volumes" Oct 30 16:42:37.035613 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:37.035566 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a7428e5-9zqd4"] Oct 30 16:42:37.046327 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:37.046298 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a7428e5-9zqd4"] Oct 30 16:42:38.935147 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:38.935114 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="052926f9-c4c0-4540-8927-64a40bb4ed40" path="/var/lib/kubelet/pods/052926f9-c4c0-4540-8927-64a40bb4ed40/volumes" Oct 30 16:42:42.036225 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:42.036194 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-600903dc-6wp5m"] Oct 30 16:42:42.047596 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:42.047568 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-600903dc-6wp5m"] Oct 30 16:42:42.368836 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:42.368738 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e22aa1c8-7bj5d"] Oct 30 16:42:42.369439 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:42.369419 4493 cpu_manager.go:414] "containerMap: removing container" podUID="b29fb352-2ea5-4b64-b598-3a7b6e3d7e0b" containerName="revoke-keys" Oct 30 16:42:42.369555 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:42.369440 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b29fb352-2ea5-4b64-b598-3a7b6e3d7e0b" containerName="revoke-keys" Oct 30 16:42:42.369655 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:42.369642 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="b29fb352-2ea5-4b64-b598-3a7b6e3d7e0b" containerName="revoke-keys" Oct 30 16:42:42.373013 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:42.372993 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e22aa1c8-7bj5d" Oct 30 16:42:42.392750 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:42.392717 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:42:42.392750 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:42.392721 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:42:42.404561 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:42.404533 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 16:42:42.404561 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:42.404533 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:42:42.412229 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:42.412203 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e22aa1c8-7bj5d"] Oct 30 16:42:42.516901 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:42.516864 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9szg\" (UniqueName: \"kubernetes.io/projected/4487028d-34c7-4bee-abd2-28cc0a3e6b51-kube-api-access-r9szg\") pod \"maas-api-revoke-sub-e22aa1c8-7bj5d\" (UID: \"4487028d-34c7-4bee-abd2-28cc0a3e6b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e22aa1c8-7bj5d" Oct 30 16:42:42.517078 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:42.517047 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4487028d-34c7-4bee-abd2-28cc0a3e6b51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e22aa1c8-7bj5d\" (UID: \"4487028d-34c7-4bee-abd2-28cc0a3e6b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e22aa1c8-7bj5d" Oct 30 16:42:42.517140 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:42.517089 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4487028d-34c7-4bee-abd2-28cc0a3e6b51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e22aa1c8-7bj5d\" (UID: \"4487028d-34c7-4bee-abd2-28cc0a3e6b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e22aa1c8-7bj5d" Oct 30 16:42:42.618207 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:42.618175 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9szg\" (UniqueName: \"kubernetes.io/projected/4487028d-34c7-4bee-abd2-28cc0a3e6b51-kube-api-access-r9szg\") pod \"maas-api-revoke-sub-e22aa1c8-7bj5d\" (UID: \"4487028d-34c7-4bee-abd2-28cc0a3e6b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e22aa1c8-7bj5d" Oct 30 16:42:42.618377 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:42.618343 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4487028d-34c7-4bee-abd2-28cc0a3e6b51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e22aa1c8-7bj5d\" (UID: \"4487028d-34c7-4bee-abd2-28cc0a3e6b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e22aa1c8-7bj5d" Oct 30 16:42:42.618427 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:42.618383 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4487028d-34c7-4bee-abd2-28cc0a3e6b51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e22aa1c8-7bj5d\" (UID: \"4487028d-34c7-4bee-abd2-28cc0a3e6b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e22aa1c8-7bj5d" Oct 30 16:42:42.619134 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:42.619077 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4487028d-34c7-4bee-abd2-28cc0a3e6b51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e22aa1c8-7bj5d\" (UID: \"4487028d-34c7-4bee-abd2-28cc0a3e6b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e22aa1c8-7bj5d" Oct 30 16:42:42.620946 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:42.620922 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4487028d-34c7-4bee-abd2-28cc0a3e6b51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e22aa1c8-7bj5d\" (UID: \"4487028d-34c7-4bee-abd2-28cc0a3e6b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e22aa1c8-7bj5d" Oct 30 16:42:42.642935 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:42.642912 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9szg\" (UniqueName: \"kubernetes.io/projected/4487028d-34c7-4bee-abd2-28cc0a3e6b51-kube-api-access-r9szg\") pod \"maas-api-revoke-sub-e22aa1c8-7bj5d\" (UID: \"4487028d-34c7-4bee-abd2-28cc0a3e6b51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e22aa1c8-7bj5d" Oct 30 16:42:42.687991 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:42.687961 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e22aa1c8-7bj5d" Oct 30 16:42:42.934602 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:42.934568 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51f92005-2411-4550-828b-667d303b557f" path="/var/lib/kubelet/pods/51f92005-2411-4550-828b-667d303b557f/volumes" Oct 30 16:42:43.002411 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:43.002385 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e22aa1c8-7bj5d"] Oct 30 16:42:43.003260 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:42:43.003236 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4487028d_34c7_4bee_abd2_28cc0a3e6b51.slice/crio-33988c0222742b64dcaf9a5c0ccf4a56ef7c4d0119f4387d79267c2d0a357472 WatchSource:0}: Error finding container 33988c0222742b64dcaf9a5c0ccf4a56ef7c4d0119f4387d79267c2d0a357472: Status 404 returned error can't find the container with id 33988c0222742b64dcaf9a5c0ccf4a56ef7c4d0119f4387d79267c2d0a357472 Oct 30 16:42:43.565340 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:43.565246 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e22aa1c8-7bj5d" event={"ID":"4487028d-34c7-4bee-abd2-28cc0a3e6b51","Type":"ContainerStarted","Data":"dbd3646e32a56cea6902a1c6ca53ecf70d3018d6e0d211ee3368b003f30358a2"} Oct 30 16:42:43.565340 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:43.565301 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e22aa1c8-7bj5d" event={"ID":"4487028d-34c7-4bee-abd2-28cc0a3e6b51","Type":"ContainerStarted","Data":"33988c0222742b64dcaf9a5c0ccf4a56ef7c4d0119f4387d79267c2d0a357472"} Oct 30 16:42:43.606375 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:43.606327 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e22aa1c8-7bj5d" podStartSLOduration=1.606312617 podStartE2EDuration="1.606312617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:42:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:42:43.605277694 +0000 UTC m=+1190.832452791" watchObservedRunningTime="2026-10-30 16:42:43.606312617 +0000 UTC m=+1190.833487718" Oct 30 16:42:44.574820 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:44.574768 4493 generic.go:356] "Generic (PLEG): container finished" podID="4487028d-34c7-4bee-abd2-28cc0a3e6b51" containerID="dbd3646e32a56cea6902a1c6ca53ecf70d3018d6e0d211ee3368b003f30358a2" exitCode=0 Oct 30 16:42:44.575276 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:44.574852 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e22aa1c8-7bj5d" event={"ID":"4487028d-34c7-4bee-abd2-28cc0a3e6b51","Type":"ContainerDied","Data":"dbd3646e32a56cea6902a1c6ca53ecf70d3018d6e0d211ee3368b003f30358a2"} Oct 30 16:42:45.747064 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:45.747035 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e22aa1c8-7bj5d" Oct 30 16:42:45.761667 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:45.761638 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4487028d-34c7-4bee-abd2-28cc0a3e6b51-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4487028d-34c7-4bee-abd2-28cc0a3e6b51-maas-api-openshift-ca\") pod \"4487028d-34c7-4bee-abd2-28cc0a3e6b51\" (UID: \"4487028d-34c7-4bee-abd2-28cc0a3e6b51\") " Oct 30 16:42:45.761871 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:45.761692 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4487028d-34c7-4bee-abd2-28cc0a3e6b51-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4487028d-34c7-4bee-abd2-28cc0a3e6b51-maas-api-xks-ca\") pod \"4487028d-34c7-4bee-abd2-28cc0a3e6b51\" (UID: \"4487028d-34c7-4bee-abd2-28cc0a3e6b51\") " Oct 30 16:42:45.761871 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:45.761823 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4487028d-34c7-4bee-abd2-28cc0a3e6b51-kube-api-access-r9szg\" (UniqueName: \"kubernetes.io/projected/4487028d-34c7-4bee-abd2-28cc0a3e6b51-kube-api-access-r9szg\") pod \"4487028d-34c7-4bee-abd2-28cc0a3e6b51\" (UID: \"4487028d-34c7-4bee-abd2-28cc0a3e6b51\") " Oct 30 16:42:45.762074 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:45.762050 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4487028d-34c7-4bee-abd2-28cc0a3e6b51-maas-api-openshift-ca" pod "4487028d-34c7-4bee-abd2-28cc0a3e6b51" (UID: "4487028d-34c7-4bee-abd2-28cc0a3e6b51"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:42:45.762364 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:45.762348 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4487028d-34c7-4bee-abd2-28cc0a3e6b51-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:42:45.764474 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:45.764446 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4487028d-34c7-4bee-abd2-28cc0a3e6b51-kube-api-access-r9szg" pod "4487028d-34c7-4bee-abd2-28cc0a3e6b51" (UID: "4487028d-34c7-4bee-abd2-28cc0a3e6b51"). InnerVolumeSpecName "kube-api-access-r9szg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:42:45.764554 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:45.764511 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4487028d-34c7-4bee-abd2-28cc0a3e6b51-maas-api-xks-ca" pod "4487028d-34c7-4bee-abd2-28cc0a3e6b51" (UID: "4487028d-34c7-4bee-abd2-28cc0a3e6b51"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:42:45.863318 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:45.863231 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4487028d-34c7-4bee-abd2-28cc0a3e6b51-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:42:45.863318 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:45.863264 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9szg\" (UniqueName: \"kubernetes.io/projected/4487028d-34c7-4bee-abd2-28cc0a3e6b51-kube-api-access-r9szg\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:42:46.598726 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:46.598678 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e22aa1c8-7bj5d" event={"ID":"4487028d-34c7-4bee-abd2-28cc0a3e6b51","Type":"ContainerDied","Data":"33988c0222742b64dcaf9a5c0ccf4a56ef7c4d0119f4387d79267c2d0a357472"} Oct 30 16:42:46.599036 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:46.599008 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33988c0222742b64dcaf9a5c0ccf4a56ef7c4d0119f4387d79267c2d0a357472" Oct 30 16:42:46.599184 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:46.599169 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e22aa1c8-7bj5d" Oct 30 16:42:47.284769 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:47.284739 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7410c7-ssbvm"] Oct 30 16:42:47.295357 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:47.295332 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7410c7-ssbvm"] Oct 30 16:42:48.033270 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:48.033234 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a3be62e-ktqk2"] Oct 30 16:42:48.046494 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:48.046465 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a3be62e-ktqk2"] Oct 30 16:42:48.759623 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:48.759592 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82381960-rc9ll"] Oct 30 16:42:48.760232 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:48.760218 4493 cpu_manager.go:414] "containerMap: removing container" podUID="4487028d-34c7-4bee-abd2-28cc0a3e6b51" containerName="revoke-keys" Oct 30 16:42:48.760287 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:48.760233 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4487028d-34c7-4bee-abd2-28cc0a3e6b51" containerName="revoke-keys" Oct 30 16:42:48.760334 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:48.760324 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="4487028d-34c7-4bee-abd2-28cc0a3e6b51" containerName="revoke-keys" Oct 30 16:42:48.763974 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:48.763959 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82381960-rc9ll" Oct 30 16:42:48.792972 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:48.792946 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:42:48.793173 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:48.792989 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:42:48.798623 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:48.798597 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82381960-rc9ll"] Oct 30 16:42:48.798978 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:48.798957 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:42:48.798978 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:48.798973 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 16:42:48.905599 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:48.905567 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d722102-2740-4fee-bb6a-eed9aef29b8d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-82381960-rc9ll\" (UID: \"7d722102-2740-4fee-bb6a-eed9aef29b8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82381960-rc9ll" Oct 30 16:42:48.905912 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:48.905886 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d722102-2740-4fee-bb6a-eed9aef29b8d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-82381960-rc9ll\" (UID: \"7d722102-2740-4fee-bb6a-eed9aef29b8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82381960-rc9ll" Oct 30 16:42:48.906002 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:48.905927 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jndmd\" (UniqueName: \"kubernetes.io/projected/7d722102-2740-4fee-bb6a-eed9aef29b8d-kube-api-access-jndmd\") pod \"maas-api-revoke-keys-e2e-mt-82381960-rc9ll\" (UID: \"7d722102-2740-4fee-bb6a-eed9aef29b8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82381960-rc9ll" Oct 30 16:42:48.933469 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:48.933439 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b29fb352-2ea5-4b64-b598-3a7b6e3d7e0b" path="/var/lib/kubelet/pods/b29fb352-2ea5-4b64-b598-3a7b6e3d7e0b/volumes" Oct 30 16:42:48.934026 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:48.934006 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea6938c8-3f32-4aa9-8a26-660f2aa4e20f" path="/var/lib/kubelet/pods/ea6938c8-3f32-4aa9-8a26-660f2aa4e20f/volumes" Oct 30 16:42:49.007024 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:49.006980 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d722102-2740-4fee-bb6a-eed9aef29b8d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-82381960-rc9ll\" (UID: \"7d722102-2740-4fee-bb6a-eed9aef29b8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82381960-rc9ll" Oct 30 16:42:49.007219 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:49.007049 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jndmd\" (UniqueName: \"kubernetes.io/projected/7d722102-2740-4fee-bb6a-eed9aef29b8d-kube-api-access-jndmd\") pod \"maas-api-revoke-keys-e2e-mt-82381960-rc9ll\" (UID: \"7d722102-2740-4fee-bb6a-eed9aef29b8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82381960-rc9ll" Oct 30 16:42:49.007219 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:49.007106 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d722102-2740-4fee-bb6a-eed9aef29b8d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-82381960-rc9ll\" (UID: \"7d722102-2740-4fee-bb6a-eed9aef29b8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82381960-rc9ll" Oct 30 16:42:49.007858 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:49.007832 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d722102-2740-4fee-bb6a-eed9aef29b8d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-82381960-rc9ll\" (UID: \"7d722102-2740-4fee-bb6a-eed9aef29b8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82381960-rc9ll" Oct 30 16:42:49.009526 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:49.009505 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d722102-2740-4fee-bb6a-eed9aef29b8d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-82381960-rc9ll\" (UID: \"7d722102-2740-4fee-bb6a-eed9aef29b8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82381960-rc9ll" Oct 30 16:42:49.029770 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:49.029715 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jndmd\" (UniqueName: \"kubernetes.io/projected/7d722102-2740-4fee-bb6a-eed9aef29b8d-kube-api-access-jndmd\") pod \"maas-api-revoke-keys-e2e-mt-82381960-rc9ll\" (UID: \"7d722102-2740-4fee-bb6a-eed9aef29b8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82381960-rc9ll" Oct 30 16:42:49.076882 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:49.076844 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82381960-rc9ll" Oct 30 16:42:49.398548 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:49.398479 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82381960-rc9ll"] Oct 30 16:42:49.399613 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:42:49.399436 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d722102_2740_4fee_bb6a_eed9aef29b8d.slice/crio-0cd376578c1b46bebdbaee918b9288aa61597a960c2b44855279ef83e51f9f4f WatchSource:0}: Error finding container 0cd376578c1b46bebdbaee918b9288aa61597a960c2b44855279ef83e51f9f4f: Status 404 returned error can't find the container with id 0cd376578c1b46bebdbaee918b9288aa61597a960c2b44855279ef83e51f9f4f Oct 30 16:42:49.645040 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:49.644963 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82381960-rc9ll" event={"ID":"7d722102-2740-4fee-bb6a-eed9aef29b8d","Type":"ContainerStarted","Data":"0cd376578c1b46bebdbaee918b9288aa61597a960c2b44855279ef83e51f9f4f"} Oct 30 16:42:50.657918 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:50.657886 4493 generic.go:356] "Generic (PLEG): container finished" podID="7d722102-2740-4fee-bb6a-eed9aef29b8d" containerID="b692ba76caaeba582441eb7f419abae2652ba0c53af07d8bef220dc04a90b04a" exitCode=0 Oct 30 16:42:50.658350 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:50.657986 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82381960-rc9ll" event={"ID":"7d722102-2740-4fee-bb6a-eed9aef29b8d","Type":"ContainerDied","Data":"b692ba76caaeba582441eb7f419abae2652ba0c53af07d8bef220dc04a90b04a"} Oct 30 16:42:51.831768 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:51.831743 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82381960-rc9ll" Oct 30 16:42:51.949676 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:51.949598 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d722102-2740-4fee-bb6a-eed9aef29b8d-kube-api-access-jndmd\" (UniqueName: \"kubernetes.io/projected/7d722102-2740-4fee-bb6a-eed9aef29b8d-kube-api-access-jndmd\") pod \"7d722102-2740-4fee-bb6a-eed9aef29b8d\" (UID: \"7d722102-2740-4fee-bb6a-eed9aef29b8d\") " Oct 30 16:42:51.949676 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:51.949652 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d722102-2740-4fee-bb6a-eed9aef29b8d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d722102-2740-4fee-bb6a-eed9aef29b8d-maas-api-xks-ca\") pod \"7d722102-2740-4fee-bb6a-eed9aef29b8d\" (UID: \"7d722102-2740-4fee-bb6a-eed9aef29b8d\") " Oct 30 16:42:51.949887 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:51.949842 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d722102-2740-4fee-bb6a-eed9aef29b8d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d722102-2740-4fee-bb6a-eed9aef29b8d-maas-api-openshift-ca\") pod \"7d722102-2740-4fee-bb6a-eed9aef29b8d\" (UID: \"7d722102-2740-4fee-bb6a-eed9aef29b8d\") " Oct 30 16:42:51.950252 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:51.950223 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d722102-2740-4fee-bb6a-eed9aef29b8d-maas-api-openshift-ca" pod "7d722102-2740-4fee-bb6a-eed9aef29b8d" (UID: "7d722102-2740-4fee-bb6a-eed9aef29b8d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:42:51.950450 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:51.950434 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d722102-2740-4fee-bb6a-eed9aef29b8d-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:42:51.952546 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:51.952522 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d722102-2740-4fee-bb6a-eed9aef29b8d-maas-api-xks-ca" pod "7d722102-2740-4fee-bb6a-eed9aef29b8d" (UID: "7d722102-2740-4fee-bb6a-eed9aef29b8d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:42:51.952624 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:51.952555 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d722102-2740-4fee-bb6a-eed9aef29b8d-kube-api-access-jndmd" pod "7d722102-2740-4fee-bb6a-eed9aef29b8d" (UID: "7d722102-2740-4fee-bb6a-eed9aef29b8d"). InnerVolumeSpecName "kube-api-access-jndmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:42:52.052243 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:52.052219 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jndmd\" (UniqueName: \"kubernetes.io/projected/7d722102-2740-4fee-bb6a-eed9aef29b8d-kube-api-access-jndmd\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:42:52.052243 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:52.052243 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d722102-2740-4fee-bb6a-eed9aef29b8d-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:42:52.675907 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:52.675878 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82381960-rc9ll" Oct 30 16:42:52.676093 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:52.675905 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82381960-rc9ll" event={"ID":"7d722102-2740-4fee-bb6a-eed9aef29b8d","Type":"ContainerDied","Data":"0cd376578c1b46bebdbaee918b9288aa61597a960c2b44855279ef83e51f9f4f"} Oct 30 16:42:52.676093 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:52.675950 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0cd376578c1b46bebdbaee918b9288aa61597a960c2b44855279ef83e51f9f4f" Oct 30 16:42:53.034375 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:53.034301 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b602696d-vqqhk"] Oct 30 16:42:53.038368 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:53.038346 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b602696d-vqqhk"] Oct 30 16:42:53.430476 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:53.430437 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-56v9s_d3a1c417-2698-4462-b7ff-1d2e66d7a10f/ovn-acl-logging/0.log" Oct 30 16:42:53.476220 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:53.476191 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-56v9s_d3a1c417-2698-4462-b7ff-1d2e66d7a10f/ovn-acl-logging/0.log" Oct 30 16:42:53.572065 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:53.572034 4493 scope.go:122] "RemoveContainer" containerID="8ccf2e70590438e94ae105d822644cc483dfe19aefd3c7793d31400caae51c09" Oct 30 16:42:54.817489 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:54.817449 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q"] Oct 30 16:42:54.819653 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:54.818806 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q" podUID="386d43ba-11f5-440a-ab13-d6b444bacaae" containerName="istio-proxy" containerID="cri-o://5bac4d79cfebef6bc5690e3bc9ecd8f76d365fda174b01c0750d948f63fad880" gracePeriod=30 Oct 30 16:42:54.954432 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:54.954381 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12a64168-7ea5-4014-a6ff-d7070ed00f84" path="/var/lib/kubelet/pods/12a64168-7ea5-4014-a6ff-d7070ed00f84/volumes" Oct 30 16:42:55.456271 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:55.456236 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1189d4-xjcgm"] Oct 30 16:42:55.457245 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:55.457222 4493 cpu_manager.go:414] "containerMap: removing container" podUID="7d722102-2740-4fee-bb6a-eed9aef29b8d" containerName="revoke-keys" Oct 30 16:42:55.457316 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:55.457250 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d722102-2740-4fee-bb6a-eed9aef29b8d" containerName="revoke-keys" Oct 30 16:42:55.457411 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:55.457398 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d722102-2740-4fee-bb6a-eed9aef29b8d" containerName="revoke-keys" Oct 30 16:42:55.463440 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:55.463420 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1189d4-xjcgm" Oct 30 16:42:55.478113 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:55.478079 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:42:55.478436 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:55.478419 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:42:55.480319 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:55.480297 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1189d4-xjcgm"] Oct 30 16:42:55.562990 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:55.562954 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:42:55.563239 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:55.563047 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 16:42:55.597558 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:55.597525 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 16:42:55.600889 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:55.600857 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf8af140-4280-40fb-9998-54f30d0d8d4f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1189d4-xjcgm\" (UID: \"bf8af140-4280-40fb-9998-54f30d0d8d4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1189d4-xjcgm" Oct 30 16:42:55.601000 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:55.600915 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp9k2\" (UniqueName: \"kubernetes.io/projected/bf8af140-4280-40fb-9998-54f30d0d8d4f-kube-api-access-wp9k2\") pod \"maas-api-revoke-keys-e2e-api-b-1189d4-xjcgm\" (UID: \"bf8af140-4280-40fb-9998-54f30d0d8d4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1189d4-xjcgm" Oct 30 16:42:55.601049 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:55.601022 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf8af140-4280-40fb-9998-54f30d0d8d4f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1189d4-xjcgm\" (UID: \"bf8af140-4280-40fb-9998-54f30d0d8d4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1189d4-xjcgm" Oct 30 16:42:55.702320 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:55.702286 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf8af140-4280-40fb-9998-54f30d0d8d4f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1189d4-xjcgm\" (UID: \"bf8af140-4280-40fb-9998-54f30d0d8d4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1189d4-xjcgm" Oct 30 16:42:55.702511 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:55.702337 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp9k2\" (UniqueName: \"kubernetes.io/projected/bf8af140-4280-40fb-9998-54f30d0d8d4f-kube-api-access-wp9k2\") pod \"maas-api-revoke-keys-e2e-api-b-1189d4-xjcgm\" (UID: \"bf8af140-4280-40fb-9998-54f30d0d8d4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1189d4-xjcgm" Oct 30 16:42:55.702511 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:55.702365 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf8af140-4280-40fb-9998-54f30d0d8d4f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1189d4-xjcgm\" (UID: \"bf8af140-4280-40fb-9998-54f30d0d8d4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1189d4-xjcgm" Oct 30 16:42:55.703159 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:55.703132 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf8af140-4280-40fb-9998-54f30d0d8d4f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1189d4-xjcgm\" (UID: \"bf8af140-4280-40fb-9998-54f30d0d8d4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1189d4-xjcgm" Oct 30 16:42:55.706018 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:55.705977 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf8af140-4280-40fb-9998-54f30d0d8d4f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1189d4-xjcgm\" (UID: \"bf8af140-4280-40fb-9998-54f30d0d8d4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1189d4-xjcgm" Oct 30 16:42:55.727602 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:55.727520 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp9k2\" (UniqueName: \"kubernetes.io/projected/bf8af140-4280-40fb-9998-54f30d0d8d4f-kube-api-access-wp9k2\") pod \"maas-api-revoke-keys-e2e-api-b-1189d4-xjcgm\" (UID: \"bf8af140-4280-40fb-9998-54f30d0d8d4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1189d4-xjcgm" Oct 30 16:42:55.779463 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:55.779421 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1189d4-xjcgm" Oct 30 16:42:56.100939 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:56.100904 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1189d4-xjcgm"] Oct 30 16:42:56.101412 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:42:56.101388 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf8af140_4280_40fb_9998_54f30d0d8d4f.slice/crio-75ccc273a6838b356877cdedc20de97e309046c213ed2e5976105ed33ab3c987 WatchSource:0}: Error finding container 75ccc273a6838b356877cdedc20de97e309046c213ed2e5976105ed33ab3c987: Status 404 returned error can't find the container with id 75ccc273a6838b356877cdedc20de97e309046c213ed2e5976105ed33ab3c987 Oct 30 16:42:56.728752 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:56.728676 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1189d4-xjcgm" event={"ID":"bf8af140-4280-40fb-9998-54f30d0d8d4f","Type":"ContainerStarted","Data":"b12fa3950870d86d128c1f730edfe23374c8412866ffecbe1a22d578c4617ced"} Oct 30 16:42:56.729046 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:56.729026 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1189d4-xjcgm" event={"ID":"bf8af140-4280-40fb-9998-54f30d0d8d4f","Type":"ContainerStarted","Data":"75ccc273a6838b356877cdedc20de97e309046c213ed2e5976105ed33ab3c987"} Oct 30 16:42:56.790170 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:56.789290 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1189d4-xjcgm" podStartSLOduration=1.789263732 podStartE2EDuration="1.789263732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:42:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:42:56.76992204 +0000 UTC m=+1203.997097141" watchObservedRunningTime="2026-10-30 16:42:56.789263732 +0000 UTC m=+1204.016438835" Oct 30 16:42:57.382135 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:57.382048 4493 patch_prober.go:29] interesting pod/e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 16:42:57.382135 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:57.382119 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q" podUID="386d43ba-11f5-440a-ab13-d6b444bacaae" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 16:42:57.786684 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:57.780377 4493 generic.go:356] "Generic (PLEG): container finished" podID="bf8af140-4280-40fb-9998-54f30d0d8d4f" containerID="b12fa3950870d86d128c1f730edfe23374c8412866ffecbe1a22d578c4617ced" exitCode=0 Oct 30 16:42:57.786684 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:57.785860 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1189d4-xjcgm" event={"ID":"bf8af140-4280-40fb-9998-54f30d0d8d4f","Type":"ContainerDied","Data":"b12fa3950870d86d128c1f730edfe23374c8412866ffecbe1a22d578c4617ced"} Oct 30 16:42:59.428014 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:59.427984 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1189d4-xjcgm" Oct 30 16:42:59.612857 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:59.612824 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf8af140-4280-40fb-9998-54f30d0d8d4f-kube-api-access-wp9k2\" (UniqueName: \"kubernetes.io/projected/bf8af140-4280-40fb-9998-54f30d0d8d4f-kube-api-access-wp9k2\") pod \"bf8af140-4280-40fb-9998-54f30d0d8d4f\" (UID: \"bf8af140-4280-40fb-9998-54f30d0d8d4f\") " Oct 30 16:42:59.613101 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:59.612924 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf8af140-4280-40fb-9998-54f30d0d8d4f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf8af140-4280-40fb-9998-54f30d0d8d4f-maas-api-xks-ca\") pod \"bf8af140-4280-40fb-9998-54f30d0d8d4f\" (UID: \"bf8af140-4280-40fb-9998-54f30d0d8d4f\") " Oct 30 16:42:59.613101 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:59.613098 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf8af140-4280-40fb-9998-54f30d0d8d4f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf8af140-4280-40fb-9998-54f30d0d8d4f-maas-api-openshift-ca\") pod \"bf8af140-4280-40fb-9998-54f30d0d8d4f\" (UID: \"bf8af140-4280-40fb-9998-54f30d0d8d4f\") " Oct 30 16:42:59.615864 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:59.615768 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf8af140-4280-40fb-9998-54f30d0d8d4f-maas-api-openshift-ca" pod "bf8af140-4280-40fb-9998-54f30d0d8d4f" (UID: "bf8af140-4280-40fb-9998-54f30d0d8d4f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:42:59.616027 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:59.615995 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf8af140-4280-40fb-9998-54f30d0d8d4f-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:42:59.618762 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:59.618660 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf8af140-4280-40fb-9998-54f30d0d8d4f-kube-api-access-wp9k2" pod "bf8af140-4280-40fb-9998-54f30d0d8d4f" (UID: "bf8af140-4280-40fb-9998-54f30d0d8d4f"). InnerVolumeSpecName "kube-api-access-wp9k2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:42:59.624235 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:59.624204 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf8af140-4280-40fb-9998-54f30d0d8d4f-maas-api-xks-ca" pod "bf8af140-4280-40fb-9998-54f30d0d8d4f" (UID: "bf8af140-4280-40fb-9998-54f30d0d8d4f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:42:59.718021 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:59.717983 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wp9k2\" (UniqueName: \"kubernetes.io/projected/bf8af140-4280-40fb-9998-54f30d0d8d4f-kube-api-access-wp9k2\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:42:59.718021 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:59.718023 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf8af140-4280-40fb-9998-54f30d0d8d4f-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:42:59.813136 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:59.813104 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1189d4-xjcgm" event={"ID":"bf8af140-4280-40fb-9998-54f30d0d8d4f","Type":"ContainerDied","Data":"75ccc273a6838b356877cdedc20de97e309046c213ed2e5976105ed33ab3c987"} Oct 30 16:42:59.813293 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:59.813158 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75ccc273a6838b356877cdedc20de97e309046c213ed2e5976105ed33ab3c987" Oct 30 16:42:59.813293 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:42:59.813209 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1189d4-xjcgm" Oct 30 16:43:00.204091 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:00.204006 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q" Oct 30 16:43:00.252311 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:00.252275 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/386d43ba-11f5-440a-ab13-d6b444bacaae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/386d43ba-11f5-440a-ab13-d6b444bacaae-credential-socket\") pod \"386d43ba-11f5-440a-ab13-d6b444bacaae\" (UID: \"386d43ba-11f5-440a-ab13-d6b444bacaae\") " Oct 30 16:43:00.252523 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:00.252400 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/386d43ba-11f5-440a-ab13-d6b444bacaae-istio-token\" (UniqueName: \"kubernetes.io/projected/386d43ba-11f5-440a-ab13-d6b444bacaae-istio-token\") pod \"386d43ba-11f5-440a-ab13-d6b444bacaae\" (UID: \"386d43ba-11f5-440a-ab13-d6b444bacaae\") " Oct 30 16:43:00.252523 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:00.252437 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/386d43ba-11f5-440a-ab13-d6b444bacaae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/386d43ba-11f5-440a-ab13-d6b444bacaae-istio-podinfo\") pod \"386d43ba-11f5-440a-ab13-d6b444bacaae\" (UID: \"386d43ba-11f5-440a-ab13-d6b444bacaae\") " Oct 30 16:43:00.252523 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:00.252473 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/386d43ba-11f5-440a-ab13-d6b444bacaae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/386d43ba-11f5-440a-ab13-d6b444bacaae-istio-data\") pod \"386d43ba-11f5-440a-ab13-d6b444bacaae\" (UID: \"386d43ba-11f5-440a-ab13-d6b444bacaae\") " Oct 30 16:43:00.252523 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:00.252522 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/386d43ba-11f5-440a-ab13-d6b444bacaae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/386d43ba-11f5-440a-ab13-d6b444bacaae-istiod-ca-cert\") pod \"386d43ba-11f5-440a-ab13-d6b444bacaae\" (UID: \"386d43ba-11f5-440a-ab13-d6b444bacaae\") " Oct 30 16:43:00.252763 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:00.252586 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/386d43ba-11f5-440a-ab13-d6b444bacaae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/386d43ba-11f5-440a-ab13-d6b444bacaae-workload-certs\") pod \"386d43ba-11f5-440a-ab13-d6b444bacaae\" (UID: \"386d43ba-11f5-440a-ab13-d6b444bacaae\") " Oct 30 16:43:00.252763 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:00.252620 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/386d43ba-11f5-440a-ab13-d6b444bacaae-kube-api-access-lxvhz\" (UniqueName: \"kubernetes.io/projected/386d43ba-11f5-440a-ab13-d6b444bacaae-kube-api-access-lxvhz\") pod \"386d43ba-11f5-440a-ab13-d6b444bacaae\" (UID: \"386d43ba-11f5-440a-ab13-d6b444bacaae\") " Oct 30 16:43:00.252763 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:00.252679 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/386d43ba-11f5-440a-ab13-d6b444bacaae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/386d43ba-11f5-440a-ab13-d6b444bacaae-workload-socket\") pod \"386d43ba-11f5-440a-ab13-d6b444bacaae\" (UID: \"386d43ba-11f5-440a-ab13-d6b444bacaae\") " Oct 30 16:43:00.252763 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:00.252705 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/386d43ba-11f5-440a-ab13-d6b444bacaae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/386d43ba-11f5-440a-ab13-d6b444bacaae-istio-envoy\") pod \"386d43ba-11f5-440a-ab13-d6b444bacaae\" (UID: \"386d43ba-11f5-440a-ab13-d6b444bacaae\") " Oct 30 16:43:00.253356 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:00.253327 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/386d43ba-11f5-440a-ab13-d6b444bacaae-istio-data" pod "386d43ba-11f5-440a-ab13-d6b444bacaae" (UID: "386d43ba-11f5-440a-ab13-d6b444bacaae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:43:00.254930 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:00.254892 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/386d43ba-11f5-440a-ab13-d6b444bacaae-workload-socket" pod "386d43ba-11f5-440a-ab13-d6b444bacaae" (UID: "386d43ba-11f5-440a-ab13-d6b444bacaae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:43:00.255920 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:00.255886 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/386d43ba-11f5-440a-ab13-d6b444bacaae-workload-certs" pod "386d43ba-11f5-440a-ab13-d6b444bacaae" (UID: "386d43ba-11f5-440a-ab13-d6b444bacaae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:43:00.256035 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:00.255958 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/386d43ba-11f5-440a-ab13-d6b444bacaae-istiod-ca-cert" pod "386d43ba-11f5-440a-ab13-d6b444bacaae" (UID: "386d43ba-11f5-440a-ab13-d6b444bacaae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:43:00.256035 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:00.256002 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/386d43ba-11f5-440a-ab13-d6b444bacaae-credential-socket" pod "386d43ba-11f5-440a-ab13-d6b444bacaae" (UID: "386d43ba-11f5-440a-ab13-d6b444bacaae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:43:00.260142 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:00.260122 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/386d43ba-11f5-440a-ab13-d6b444bacaae-istio-envoy" pod "386d43ba-11f5-440a-ab13-d6b444bacaae" (UID: "386d43ba-11f5-440a-ab13-d6b444bacaae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:43:00.270982 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:00.270952 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/386d43ba-11f5-440a-ab13-d6b444bacaae-kube-api-access-lxvhz" pod "386d43ba-11f5-440a-ab13-d6b444bacaae" (UID: "386d43ba-11f5-440a-ab13-d6b444bacaae"). InnerVolumeSpecName "kube-api-access-lxvhz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:43:00.271099 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:00.271075 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/386d43ba-11f5-440a-ab13-d6b444bacaae-istio-podinfo" pod "386d43ba-11f5-440a-ab13-d6b444bacaae" (UID: "386d43ba-11f5-440a-ab13-d6b444bacaae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:43:00.287076 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:00.287020 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/386d43ba-11f5-440a-ab13-d6b444bacaae-istio-token" pod "386d43ba-11f5-440a-ab13-d6b444bacaae" (UID: "386d43ba-11f5-440a-ab13-d6b444bacaae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:43:00.358813 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:00.357874 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/386d43ba-11f5-440a-ab13-d6b444bacaae-istio-token\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:43:00.358813 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:00.357909 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/386d43ba-11f5-440a-ab13-d6b444bacaae-istio-podinfo\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:43:00.358813 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:00.357927 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/386d43ba-11f5-440a-ab13-d6b444bacaae-istio-data\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:43:00.358813 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:00.357939 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/386d43ba-11f5-440a-ab13-d6b444bacaae-istiod-ca-cert\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:43:00.358813 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:00.357952 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/386d43ba-11f5-440a-ab13-d6b444bacaae-workload-certs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:43:00.358813 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:00.357974 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxvhz\" (UniqueName: \"kubernetes.io/projected/386d43ba-11f5-440a-ab13-d6b444bacaae-kube-api-access-lxvhz\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:43:00.358813 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:00.357987 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/386d43ba-11f5-440a-ab13-d6b444bacaae-workload-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:43:00.358813 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:00.358003 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/386d43ba-11f5-440a-ab13-d6b444bacaae-istio-envoy\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:43:00.358813 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:00.358015 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/386d43ba-11f5-440a-ab13-d6b444bacaae-credential-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:43:00.377609 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:43:00.377565 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 16:43:00.827437 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:00.827397 4493 generic.go:356] "Generic (PLEG): container finished" podID="386d43ba-11f5-440a-ab13-d6b444bacaae" containerID="5bac4d79cfebef6bc5690e3bc9ecd8f76d365fda174b01c0750d948f63fad880" exitCode=0 Oct 30 16:43:00.827935 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:00.827443 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q" Oct 30 16:43:00.827935 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:00.827480 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q" event={"ID":"386d43ba-11f5-440a-ab13-d6b444bacaae","Type":"ContainerDied","Data":"5bac4d79cfebef6bc5690e3bc9ecd8f76d365fda174b01c0750d948f63fad880"} Oct 30 16:43:00.827935 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:00.827541 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q" event={"ID":"386d43ba-11f5-440a-ab13-d6b444bacaae","Type":"ContainerDied","Data":"5102848f6b5ba6a3fba3aa0eccf4fa37f9066158c70cef98110a8245d1954056"} Oct 30 16:43:00.827935 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:00.827570 4493 scope.go:122] "RemoveContainer" containerID="5bac4d79cfebef6bc5690e3bc9ecd8f76d365fda174b01c0750d948f63fad880" Oct 30 16:43:00.839867 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:00.839843 4493 scope.go:122] "RemoveContainer" containerID="9239688fe498dfdcd8b715a6e0d1b6caaffe02cad2c44bf435e781bec69579de" Oct 30 16:43:00.853533 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:00.853501 4493 scope.go:122] "RemoveContainer" containerID="5bac4d79cfebef6bc5690e3bc9ecd8f76d365fda174b01c0750d948f63fad880" Oct 30 16:43:00.854051 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:43:00.853977 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bac4d79cfebef6bc5690e3bc9ecd8f76d365fda174b01c0750d948f63fad880\": container with ID starting with 5bac4d79cfebef6bc5690e3bc9ecd8f76d365fda174b01c0750d948f63fad880 not found: ID does not exist" containerID="5bac4d79cfebef6bc5690e3bc9ecd8f76d365fda174b01c0750d948f63fad880" Oct 30 16:43:00.854176 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:00.854048 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bac4d79cfebef6bc5690e3bc9ecd8f76d365fda174b01c0750d948f63fad880"} err="failed to get container status \"5bac4d79cfebef6bc5690e3bc9ecd8f76d365fda174b01c0750d948f63fad880\": rpc error: code = NotFound desc = could not find container \"5bac4d79cfebef6bc5690e3bc9ecd8f76d365fda174b01c0750d948f63fad880\": container with ID starting with 5bac4d79cfebef6bc5690e3bc9ecd8f76d365fda174b01c0750d948f63fad880 not found: ID does not exist" Oct 30 16:43:00.854176 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:00.854086 4493 scope.go:122] "RemoveContainer" containerID="9239688fe498dfdcd8b715a6e0d1b6caaffe02cad2c44bf435e781bec69579de" Oct 30 16:43:00.854441 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:43:00.854402 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9239688fe498dfdcd8b715a6e0d1b6caaffe02cad2c44bf435e781bec69579de\": container with ID starting with 9239688fe498dfdcd8b715a6e0d1b6caaffe02cad2c44bf435e781bec69579de not found: ID does not exist" containerID="9239688fe498dfdcd8b715a6e0d1b6caaffe02cad2c44bf435e781bec69579de" Oct 30 16:43:00.854516 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:00.854456 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9239688fe498dfdcd8b715a6e0d1b6caaffe02cad2c44bf435e781bec69579de"} err="failed to get container status \"9239688fe498dfdcd8b715a6e0d1b6caaffe02cad2c44bf435e781bec69579de\": rpc error: code = NotFound desc = could not find container \"9239688fe498dfdcd8b715a6e0d1b6caaffe02cad2c44bf435e781bec69579de\": container with ID starting with 9239688fe498dfdcd8b715a6e0d1b6caaffe02cad2c44bf435e781bec69579de not found: ID does not exist" Oct 30 16:43:00.884215 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:00.884179 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q"] Oct 30 16:43:00.896391 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:00.896354 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-7410c7-openshift-default-6f5d55d6bf-wgm8q"] Oct 30 16:43:00.934197 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:00.934153 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="386d43ba-11f5-440a-ab13-d6b444bacaae" path="/var/lib/kubelet/pods/386d43ba-11f5-440a-ab13-d6b444bacaae/volumes" Oct 30 16:43:03.044847 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:03.044815 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1da9e70e-hnq6n"] Oct 30 16:43:03.056175 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:03.056149 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1da9e70e-hnq6n"] Oct 30 16:43:04.934468 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:04.934376 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18d34af8-18c4-453c-bfd3-7a4a8dffc7ff" path="/var/lib/kubelet/pods/18d34af8-18c4-453c-bfd3-7a4a8dffc7ff/volumes" Oct 30 16:43:06.046406 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:06.046369 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82381960-rc9ll"] Oct 30 16:43:06.057865 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:06.057840 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-82381960-rc9ll"] Oct 30 16:43:06.939183 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:06.939152 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d722102-2740-4fee-bb6a-eed9aef29b8d" path="/var/lib/kubelet/pods/7d722102-2740-4fee-bb6a-eed9aef29b8d/volumes" Oct 30 16:43:07.044001 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:07.043965 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-660c8fee-trm9h"] Oct 30 16:43:07.060634 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:07.060597 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-660c8fee-trm9h"] Oct 30 16:43:08.937277 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:08.937237 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bad8db2-5b23-43c5-a568-8e66d2dc59b5" path="/var/lib/kubelet/pods/1bad8db2-5b23-43c5-a568-8e66d2dc59b5/volumes" Oct 30 16:43:13.033428 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:13.033395 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22135e92-mzp7l"] Oct 30 16:43:13.045181 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:13.045152 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22135e92-mzp7l"] Oct 30 16:43:13.048996 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:13.048978 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e7b30f3-qtsf9"] Oct 30 16:43:13.052182 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:13.052159 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e7b30f3-qtsf9"] Oct 30 16:43:14.933698 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:14.933665 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d2a02f4-8325-41c6-a341-df1267e99563" path="/var/lib/kubelet/pods/8d2a02f4-8325-41c6-a341-df1267e99563/volumes" Oct 30 16:43:14.934198 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:14.934189 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e20ce11b-ff37-4c9a-b8be-503604a1bac4" path="/var/lib/kubelet/pods/e20ce11b-ff37-4c9a-b8be-503604a1bac4/volumes" Oct 30 16:43:17.054764 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:17.054729 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4ebd091-cj9n9"] Oct 30 16:43:17.066774 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:17.066749 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4ebd091-cj9n9"] Oct 30 16:43:18.935618 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:18.935581 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7379b269-37fa-48c8-bba5-8c3e74a3294c" path="/var/lib/kubelet/pods/7379b269-37fa-48c8-bba5-8c3e74a3294c/volumes" Oct 30 16:43:21.079690 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:21.079651 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 16:43:25.360298 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:25.360268 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1189d4-xjcgm"] Oct 30 16:43:25.371405 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:25.371380 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1189d4-xjcgm"] Oct 30 16:43:26.933951 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:26.933909 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf8af140-4280-40fb-9998-54f30d0d8d4f" path="/var/lib/kubelet/pods/bf8af140-4280-40fb-9998-54f30d0d8d4f/volumes" Oct 30 16:43:30.345031 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:43:30.344984 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 16:43:33.057464 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:33.057425 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c87577a2-b4vtg"] Oct 30 16:43:33.063580 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:33.063540 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c87577a2-b4vtg"] Oct 30 16:43:34.037922 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:34.037885 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ecbd336-sw9fp"] Oct 30 16:43:34.049252 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:34.049217 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ecbd336-sw9fp"] Oct 30 16:43:34.935709 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:34.935667 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68135857-e1be-4c00-bfd2-aa3efcd8ce22" path="/var/lib/kubelet/pods/68135857-e1be-4c00-bfd2-aa3efcd8ce22/volumes" Oct 30 16:43:34.936392 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:34.936368 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f88a9011-5708-4dbf-a1dd-24d44c6bde2c" path="/var/lib/kubelet/pods/f88a9011-5708-4dbf-a1dd-24d44c6bde2c/volumes" Oct 30 16:43:36.042753 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:36.042721 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd5d978f-rwghq"] Oct 30 16:43:36.056284 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:36.056253 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd5d978f-rwghq"] Oct 30 16:43:36.936459 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:36.936420 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38b708d1-f21e-4197-88d1-63d3e85f9475" path="/var/lib/kubelet/pods/38b708d1-f21e-4197-88d1-63d3e85f9475/volumes" Oct 30 16:43:47.035254 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:47.035215 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc09e051-k8p8h"] Oct 30 16:43:47.047003 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:47.046970 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc09e051-k8p8h"] Oct 30 16:43:48.934003 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:48.933967 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="457364de-ff92-4a03-a3f8-ae3a3268c487" path="/var/lib/kubelet/pods/457364de-ff92-4a03-a3f8-ae3a3268c487/volumes" Oct 30 16:43:53.606754 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:53.606716 4493 scope.go:122] "RemoveContainer" containerID="a2da5326a49b6564d4f278c2ad0720516043455fc6e39710d482284d53f07248" Oct 30 16:43:53.623962 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:53.623935 4493 scope.go:122] "RemoveContainer" containerID="1276b7c3e01101bdbec125da6f37173034d17fd0a5c490a2ae1f398bf81ffdcc" Oct 30 16:43:53.643733 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:53.643706 4493 scope.go:122] "RemoveContainer" containerID="457a1a65ce23b0fb987df44f19c512eebcacefb605eac7d90a44f55ae922dfad" Oct 30 16:43:53.654983 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:53.654954 4493 scope.go:122] "RemoveContainer" containerID="2b8e8d16c924705aacaaf346634f0763b46c19f06e000e09852709a14d294e79" Oct 30 16:43:53.664458 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:53.664443 4493 scope.go:122] "RemoveContainer" containerID="f4a6fcca5ed927bf9fa1a3cc71e14a6388a5c53ad77d2051bd58acec6332c1ee" Oct 30 16:43:53.674127 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:53.674109 4493 scope.go:122] "RemoveContainer" containerID="ead2ff9af8f361dc0b90a380dab78fa126dd350c733d7f422ea41e180364b49d" Oct 30 16:43:53.683572 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:53.683552 4493 scope.go:122] "RemoveContainer" containerID="ccd3c80420e82492596790378c2d3d2ff86cbcddb35579c58b826841fa0b19bf" Oct 30 16:43:53.695143 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:53.695125 4493 scope.go:122] "RemoveContainer" containerID="6a5e91035c45373e435a70623b35501767d83fdd007d727f7cd57f05f5732af9" Oct 30 16:43:54.035059 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:54.035021 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb8efadc-cpd4t"] Oct 30 16:43:54.046550 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:54.046513 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb8efadc-cpd4t"] Oct 30 16:43:54.942807 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:54.938854 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b9353ef-0d40-4b4e-b398-2c97692001bf" path="/var/lib/kubelet/pods/1b9353ef-0d40-4b4e-b398-2c97692001bf/volumes" Oct 30 16:43:55.023661 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:55.023628 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 16:43:55.034468 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:55.034438 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-696f6503-n2chf"] Oct 30 16:43:55.038048 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:55.038023 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-696f6503-n2chf"] Oct 30 16:43:56.934162 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:43:56.934130 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b379edd5-f6eb-4c61-9b59-1f9a36f0959d" path="/var/lib/kubelet/pods/b379edd5-f6eb-4c61-9b59-1f9a36f0959d/volumes" Oct 30 16:44:00.352471 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:44:00.352437 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 16:44:00.517651 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:00.517617 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm"] Oct 30 16:44:00.517962 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:00.517927 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm" podUID="0a035fea-221e-4b14-aa1f-68bd02b90794" containerName="istio-proxy" containerID="cri-o://cccb47c9e56127d3f2ca388ef3894f129f5e89c739da639105f252f4fc631778" gracePeriod=30 Oct 30 16:44:05.719509 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:05.719470 4493 generic.go:356] "Generic (PLEG): container finished" podID="0a035fea-221e-4b14-aa1f-68bd02b90794" containerID="cccb47c9e56127d3f2ca388ef3894f129f5e89c739da639105f252f4fc631778" exitCode=0 Oct 30 16:44:05.719990 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:05.719626 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm" event={"ID":"0a035fea-221e-4b14-aa1f-68bd02b90794","Type":"ContainerDied","Data":"cccb47c9e56127d3f2ca388ef3894f129f5e89c739da639105f252f4fc631778"} Oct 30 16:44:05.797204 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:05.797148 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm" Oct 30 16:44:05.861071 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:05.861041 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a035fea-221e-4b14-aa1f-68bd02b90794-kube-api-access-jwmjs\" (UniqueName: \"kubernetes.io/projected/0a035fea-221e-4b14-aa1f-68bd02b90794-kube-api-access-jwmjs\") pod \"0a035fea-221e-4b14-aa1f-68bd02b90794\" (UID: \"0a035fea-221e-4b14-aa1f-68bd02b90794\") " Oct 30 16:44:05.861234 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:05.861111 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a035fea-221e-4b14-aa1f-68bd02b90794-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a035fea-221e-4b14-aa1f-68bd02b90794-credential-socket\") pod \"0a035fea-221e-4b14-aa1f-68bd02b90794\" (UID: \"0a035fea-221e-4b14-aa1f-68bd02b90794\") " Oct 30 16:44:05.861234 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:05.861136 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0a035fea-221e-4b14-aa1f-68bd02b90794-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a035fea-221e-4b14-aa1f-68bd02b90794-istio-podinfo\") pod \"0a035fea-221e-4b14-aa1f-68bd02b90794\" (UID: \"0a035fea-221e-4b14-aa1f-68bd02b90794\") " Oct 30 16:44:05.861234 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:05.861181 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a035fea-221e-4b14-aa1f-68bd02b90794-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a035fea-221e-4b14-aa1f-68bd02b90794-workload-certs\") pod \"0a035fea-221e-4b14-aa1f-68bd02b90794\" (UID: \"0a035fea-221e-4b14-aa1f-68bd02b90794\") " Oct 30 16:44:05.861234 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:05.861211 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a035fea-221e-4b14-aa1f-68bd02b90794-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a035fea-221e-4b14-aa1f-68bd02b90794-istiod-ca-cert\") pod \"0a035fea-221e-4b14-aa1f-68bd02b90794\" (UID: \"0a035fea-221e-4b14-aa1f-68bd02b90794\") " Oct 30 16:44:05.861467 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:05.861280 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a035fea-221e-4b14-aa1f-68bd02b90794-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a035fea-221e-4b14-aa1f-68bd02b90794-workload-socket\") pod \"0a035fea-221e-4b14-aa1f-68bd02b90794\" (UID: \"0a035fea-221e-4b14-aa1f-68bd02b90794\") " Oct 30 16:44:05.861467 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:05.861353 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a035fea-221e-4b14-aa1f-68bd02b90794-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a035fea-221e-4b14-aa1f-68bd02b90794-istio-envoy\") pod \"0a035fea-221e-4b14-aa1f-68bd02b90794\" (UID: \"0a035fea-221e-4b14-aa1f-68bd02b90794\") " Oct 30 16:44:05.861467 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:05.861383 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a035fea-221e-4b14-aa1f-68bd02b90794-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a035fea-221e-4b14-aa1f-68bd02b90794-istio-data\") pod \"0a035fea-221e-4b14-aa1f-68bd02b90794\" (UID: \"0a035fea-221e-4b14-aa1f-68bd02b90794\") " Oct 30 16:44:05.861467 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:05.861451 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a035fea-221e-4b14-aa1f-68bd02b90794-istio-token\" (UniqueName: \"kubernetes.io/projected/0a035fea-221e-4b14-aa1f-68bd02b90794-istio-token\") pod \"0a035fea-221e-4b14-aa1f-68bd02b90794\" (UID: \"0a035fea-221e-4b14-aa1f-68bd02b90794\") " Oct 30 16:44:05.861467 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:05.861451 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a035fea-221e-4b14-aa1f-68bd02b90794-credential-socket" pod "0a035fea-221e-4b14-aa1f-68bd02b90794" (UID: "0a035fea-221e-4b14-aa1f-68bd02b90794"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:44:05.861727 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:05.861692 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a035fea-221e-4b14-aa1f-68bd02b90794-workload-certs" pod "0a035fea-221e-4b14-aa1f-68bd02b90794" (UID: "0a035fea-221e-4b14-aa1f-68bd02b90794"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:44:05.861985 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:05.861861 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a035fea-221e-4b14-aa1f-68bd02b90794-istiod-ca-cert" pod "0a035fea-221e-4b14-aa1f-68bd02b90794" (UID: "0a035fea-221e-4b14-aa1f-68bd02b90794"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:44:05.862776 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:05.862282 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a035fea-221e-4b14-aa1f-68bd02b90794-credential-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:44:05.862776 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:05.862309 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a035fea-221e-4b14-aa1f-68bd02b90794-workload-certs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:44:05.862776 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:05.862324 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a035fea-221e-4b14-aa1f-68bd02b90794-istiod-ca-cert\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:44:05.862776 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:05.862732 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a035fea-221e-4b14-aa1f-68bd02b90794-workload-socket" pod "0a035fea-221e-4b14-aa1f-68bd02b90794" (UID: "0a035fea-221e-4b14-aa1f-68bd02b90794"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:44:05.863454 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:05.863404 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a035fea-221e-4b14-aa1f-68bd02b90794-istio-data" pod "0a035fea-221e-4b14-aa1f-68bd02b90794" (UID: "0a035fea-221e-4b14-aa1f-68bd02b90794"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:44:05.863569 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:05.863532 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a035fea-221e-4b14-aa1f-68bd02b90794-istio-podinfo" pod "0a035fea-221e-4b14-aa1f-68bd02b90794" (UID: "0a035fea-221e-4b14-aa1f-68bd02b90794"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:44:05.864495 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:05.864472 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a035fea-221e-4b14-aa1f-68bd02b90794-kube-api-access-jwmjs" pod "0a035fea-221e-4b14-aa1f-68bd02b90794" (UID: "0a035fea-221e-4b14-aa1f-68bd02b90794"). InnerVolumeSpecName "kube-api-access-jwmjs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:44:05.864628 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:05.864601 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a035fea-221e-4b14-aa1f-68bd02b90794-istio-token" pod "0a035fea-221e-4b14-aa1f-68bd02b90794" (UID: "0a035fea-221e-4b14-aa1f-68bd02b90794"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:44:05.864757 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:05.864656 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a035fea-221e-4b14-aa1f-68bd02b90794-istio-envoy" pod "0a035fea-221e-4b14-aa1f-68bd02b90794" (UID: "0a035fea-221e-4b14-aa1f-68bd02b90794"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:44:05.963470 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:05.963441 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a035fea-221e-4b14-aa1f-68bd02b90794-istio-envoy\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:44:05.963470 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:05.963465 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a035fea-221e-4b14-aa1f-68bd02b90794-istio-data\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:44:05.963470 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:05.963474 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a035fea-221e-4b14-aa1f-68bd02b90794-istio-token\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:44:05.963828 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:05.963484 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwmjs\" (UniqueName: \"kubernetes.io/projected/0a035fea-221e-4b14-aa1f-68bd02b90794-kube-api-access-jwmjs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:44:05.963828 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:05.963492 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a035fea-221e-4b14-aa1f-68bd02b90794-istio-podinfo\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:44:05.963828 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:05.963500 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a035fea-221e-4b14-aa1f-68bd02b90794-workload-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:44:06.734010 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:06.733974 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm" Oct 30 16:44:06.734010 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:06.733987 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm" event={"ID":"0a035fea-221e-4b14-aa1f-68bd02b90794","Type":"ContainerDied","Data":"d2bf16d8615b56b93df0af0a6a725ac4aae8eb58a3baf7cab37b4be30fe2c80d"} Oct 30 16:44:06.734550 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:06.734051 4493 scope.go:122] "RemoveContainer" containerID="cccb47c9e56127d3f2ca388ef3894f129f5e89c739da639105f252f4fc631778" Oct 30 16:44:06.794502 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:06.794469 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm"] Oct 30 16:44:06.803709 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:06.803680 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-9lkxm"] Oct 30 16:44:07.025720 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:07.025643 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a035fea-221e-4b14-aa1f-68bd02b90794" path="/var/lib/kubelet/pods/0a035fea-221e-4b14-aa1f-68bd02b90794/volumes" Oct 30 16:44:11.029738 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:11.029697 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa13dec8-dvsd4"] Oct 30 16:44:11.039137 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:11.039111 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa13dec8-dvsd4"] Oct 30 16:44:12.945488 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:12.945423 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fd39b8a-e318-4412-b153-ee3c8dfae71b" path="/var/lib/kubelet/pods/3fd39b8a-e318-4412-b153-ee3c8dfae71b/volumes" Oct 30 16:44:18.383985 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:18.383945 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 16:44:29.416516 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:29.416479 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0a98d1-6b5d79cbb4-crqf4"] Oct 30 16:44:29.417093 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:29.417077 4493 cpu_manager.go:414] "containerMap: removing container" podUID="386d43ba-11f5-440a-ab13-d6b444bacaae" containerName="istio-proxy" Oct 30 16:44:29.417164 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:29.417095 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="386d43ba-11f5-440a-ab13-d6b444bacaae" containerName="istio-proxy" Oct 30 16:44:29.417164 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:29.417118 4493 cpu_manager.go:414] "containerMap: removing container" podUID="0a035fea-221e-4b14-aa1f-68bd02b90794" containerName="istio-proxy" Oct 30 16:44:29.417164 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:29.417123 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a035fea-221e-4b14-aa1f-68bd02b90794" containerName="istio-proxy" Oct 30 16:44:29.417164 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:29.417129 4493 cpu_manager.go:414] "containerMap: removing container" podUID="bf8af140-4280-40fb-9998-54f30d0d8d4f" containerName="revoke-keys" Oct 30 16:44:29.417164 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:29.417136 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf8af140-4280-40fb-9998-54f30d0d8d4f" containerName="revoke-keys" Oct 30 16:44:29.417383 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:29.417241 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="386d43ba-11f5-440a-ab13-d6b444bacaae" containerName="istio-proxy" Oct 30 16:44:29.417383 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:29.417252 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a035fea-221e-4b14-aa1f-68bd02b90794" containerName="istio-proxy" Oct 30 16:44:29.417383 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:29.417258 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf8af140-4280-40fb-9998-54f30d0d8d4f" containerName="revoke-keys" Oct 30 16:44:29.417383 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:29.417266 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="386d43ba-11f5-440a-ab13-d6b444bacaae" containerName="istio-proxy" Oct 30 16:44:29.421099 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:29.421075 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0a98d1-6b5d79cbb4-crqf4" Oct 30 16:44:29.444900 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:29.444869 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-0a98d1\"" Oct 30 16:44:29.445057 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:29.444874 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-0a98d1-dockercfg-xqdbd\"" Oct 30 16:44:29.450710 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:29.450683 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0a98d1-6b5d79cbb4-crqf4"] Oct 30 16:44:29.527231 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:29.527198 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc863628-c06a-4415-961d-0179b11b3a63-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0a98d1-6b5d79cbb4-crqf4\" (UID: \"dc863628-c06a-4415-961d-0179b11b3a63\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0a98d1-6b5d79cbb4-crqf4" Oct 30 16:44:29.527443 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:29.527360 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28z77\" (UniqueName: \"kubernetes.io/projected/dc863628-c06a-4415-961d-0179b11b3a63-kube-api-access-28z77\") pod \"payload-processing-e2e-ipp-a-0a98d1-6b5d79cbb4-crqf4\" (UID: \"dc863628-c06a-4415-961d-0179b11b3a63\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0a98d1-6b5d79cbb4-crqf4" Oct 30 16:44:29.628521 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:29.628484 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28z77\" (UniqueName: \"kubernetes.io/projected/dc863628-c06a-4415-961d-0179b11b3a63-kube-api-access-28z77\") pod \"payload-processing-e2e-ipp-a-0a98d1-6b5d79cbb4-crqf4\" (UID: \"dc863628-c06a-4415-961d-0179b11b3a63\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0a98d1-6b5d79cbb4-crqf4" Oct 30 16:44:29.628697 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:29.628607 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc863628-c06a-4415-961d-0179b11b3a63-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0a98d1-6b5d79cbb4-crqf4\" (UID: \"dc863628-c06a-4415-961d-0179b11b3a63\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0a98d1-6b5d79cbb4-crqf4" Oct 30 16:44:29.629389 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:29.629368 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc863628-c06a-4415-961d-0179b11b3a63-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0a98d1-6b5d79cbb4-crqf4\" (UID: \"dc863628-c06a-4415-961d-0179b11b3a63\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0a98d1-6b5d79cbb4-crqf4" Oct 30 16:44:29.650260 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:29.650232 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28z77\" (UniqueName: \"kubernetes.io/projected/dc863628-c06a-4415-961d-0179b11b3a63-kube-api-access-28z77\") pod \"payload-processing-e2e-ipp-a-0a98d1-6b5d79cbb4-crqf4\" (UID: \"dc863628-c06a-4415-961d-0179b11b3a63\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0a98d1-6b5d79cbb4-crqf4" Oct 30 16:44:29.739683 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:29.739605 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0a98d1-6b5d79cbb4-crqf4" Oct 30 16:44:30.064513 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:30.064422 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0a98d1-6b5d79cbb4-crqf4"] Oct 30 16:44:30.067936 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:44:30.067897 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc863628_c06a_4415_961d_0179b11b3a63.slice/crio-6043fab49c2a72e47a5893f55759dcf951c170bcfbad008a9a5518389dfb6131 WatchSource:0}: Error finding container 6043fab49c2a72e47a5893f55759dcf951c170bcfbad008a9a5518389dfb6131: Status 404 returned error can't find the container with id 6043fab49c2a72e47a5893f55759dcf951c170bcfbad008a9a5518389dfb6131 Oct 30 16:44:30.356339 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:44:30.356251 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 16:44:30.950849 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:30.950813 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0a98d1-6b5d79cbb4-crqf4" event={"ID":"dc863628-c06a-4415-961d-0179b11b3a63","Type":"ContainerStarted","Data":"6043fab49c2a72e47a5893f55759dcf951c170bcfbad008a9a5518389dfb6131"} Oct 30 16:44:32.286868 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:32.285980 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-843ab3-84f5645776-wbn9k"] Oct 30 16:44:32.286868 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:32.286656 4493 cpu_manager.go:414] "containerMap: removing container" podUID="386d43ba-11f5-440a-ab13-d6b444bacaae" containerName="istio-proxy" Oct 30 16:44:32.286868 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:32.286675 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="386d43ba-11f5-440a-ab13-d6b444bacaae" containerName="istio-proxy" Oct 30 16:44:32.292152 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:32.292127 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-843ab3-84f5645776-wbn9k" Oct 30 16:44:32.301049 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:32.300769 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-843ab3-84f5645776-wbn9k"] Oct 30 16:44:32.320663 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:32.320170 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-843ab3-dockercfg-5jd66\"" Oct 30 16:44:32.325861 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:32.323020 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-843ab3\"" Oct 30 16:44:32.387477 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:32.387437 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47194c28-9025-4eb5-98e6-9482dc97d190-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-843ab3-84f5645776-wbn9k\" (UID: \"47194c28-9025-4eb5-98e6-9482dc97d190\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-843ab3-84f5645776-wbn9k" Oct 30 16:44:32.387625 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:32.387561 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knz88\" (UniqueName: \"kubernetes.io/projected/47194c28-9025-4eb5-98e6-9482dc97d190-kube-api-access-knz88\") pod \"payload-processing-e2e-ipp-a-843ab3-84f5645776-wbn9k\" (UID: \"47194c28-9025-4eb5-98e6-9482dc97d190\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-843ab3-84f5645776-wbn9k" Oct 30 16:44:32.488758 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:32.488722 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47194c28-9025-4eb5-98e6-9482dc97d190-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-843ab3-84f5645776-wbn9k\" (UID: \"47194c28-9025-4eb5-98e6-9482dc97d190\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-843ab3-84f5645776-wbn9k" Oct 30 16:44:32.488963 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:32.488881 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knz88\" (UniqueName: \"kubernetes.io/projected/47194c28-9025-4eb5-98e6-9482dc97d190-kube-api-access-knz88\") pod \"payload-processing-e2e-ipp-a-843ab3-84f5645776-wbn9k\" (UID: \"47194c28-9025-4eb5-98e6-9482dc97d190\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-843ab3-84f5645776-wbn9k" Oct 30 16:44:32.489436 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:32.489415 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47194c28-9025-4eb5-98e6-9482dc97d190-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-843ab3-84f5645776-wbn9k\" (UID: \"47194c28-9025-4eb5-98e6-9482dc97d190\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-843ab3-84f5645776-wbn9k" Oct 30 16:44:32.511188 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:32.511160 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knz88\" (UniqueName: \"kubernetes.io/projected/47194c28-9025-4eb5-98e6-9482dc97d190-kube-api-access-knz88\") pod \"payload-processing-e2e-ipp-a-843ab3-84f5645776-wbn9k\" (UID: \"47194c28-9025-4eb5-98e6-9482dc97d190\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-843ab3-84f5645776-wbn9k" Oct 30 16:44:32.646321 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:32.646289 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-843ab3-84f5645776-wbn9k" Oct 30 16:44:32.957648 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:32.957616 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-843ab3-84f5645776-wbn9k"] Oct 30 16:44:32.959672 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:44:32.959645 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47194c28_9025_4eb5_98e6_9482dc97d190.slice/crio-3c01a86901465ec7f5129b5623230357e27129dace685594306fd942b7dfafe9 WatchSource:0}: Error finding container 3c01a86901465ec7f5129b5623230357e27129dace685594306fd942b7dfafe9: Status 404 returned error can't find the container with id 3c01a86901465ec7f5129b5623230357e27129dace685594306fd942b7dfafe9 Oct 30 16:44:32.983726 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:32.983692 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0a98d1-6b5d79cbb4-crqf4" event={"ID":"dc863628-c06a-4415-961d-0179b11b3a63","Type":"ContainerStarted","Data":"73e35e9b5cb1794693cb1a2fc2a90fc2ade0b9774029dc3e18bc15eaaf026069"} Oct 30 16:44:32.984069 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:32.983828 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-0a98d1-6b5d79cbb4-crqf4" Oct 30 16:44:32.985553 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:32.985525 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-843ab3-84f5645776-wbn9k" event={"ID":"47194c28-9025-4eb5-98e6-9482dc97d190","Type":"ContainerStarted","Data":"3c01a86901465ec7f5129b5623230357e27129dace685594306fd942b7dfafe9"} Oct 30 16:44:33.023616 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:33.023572 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-0a98d1-6b5d79cbb4-crqf4" podStartSLOduration=2.097589803 podStartE2EDuration="4.023558329s" totalImagesPullingTime="1.925968526s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:44:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:44:33.022999921 +0000 UTC m=+1300.250175021" watchObservedRunningTime="2026-10-30 16:44:33.023558329 +0000 UTC m=+1300.250733458" Oct 30 16:44:33.995867 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:33.995830 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-843ab3-84f5645776-wbn9k" event={"ID":"47194c28-9025-4eb5-98e6-9482dc97d190","Type":"ContainerStarted","Data":"3778e82bac83f982139b0da99bf0886e6fd926490253cdb8d9f42942ea4cbb7e"} Oct 30 16:44:33.996275 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:33.995900 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-843ab3-84f5645776-wbn9k" Oct 30 16:44:34.035949 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:34.035906 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-843ab3-84f5645776-wbn9k" podStartSLOduration=2.035892287 podStartE2EDuration="2.035892287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:44:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:44:34.032821073 +0000 UTC m=+1301.259996173" watchObservedRunningTime="2026-10-30 16:44:34.035892287 +0000 UTC m=+1301.263067386" Oct 30 16:44:43.998874 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:43.998843 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-0a98d1-6b5d79cbb4-crqf4" Oct 30 16:44:45.009809 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:45.009759 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-843ab3-84f5645776-wbn9k" Oct 30 16:44:53.882808 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:53.871926 4493 scope.go:122] "RemoveContainer" containerID="14ef523e96acac85ecf93a7e53fdade8930f2230fe5774d4f6cacb767e86f78b" Oct 30 16:44:53.929535 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:53.929506 4493 scope.go:122] "RemoveContainer" containerID="b4a0668b314d34526b46e7c7c8fd2a90709862187e7a2576275ed8977e8de0db" Oct 30 16:44:53.950798 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:53.950760 4493 scope.go:122] "RemoveContainer" containerID="7421cb9f73f21cb023685a964d32ecc058ea7382123b473817912dbb5ca38a13" Oct 30 16:44:53.960950 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:53.960927 4493 scope.go:122] "RemoveContainer" containerID="cd3c54eee9c1aec1c308fc1afd9f11de148d725d3a7d2cc352b5d27568e612d6" Oct 30 16:44:53.971117 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:53.971098 4493 scope.go:122] "RemoveContainer" containerID="b6eeeec2fd4bdf5fd568dd2b82208b2391e66387e60935c0109a0c3cad709f07" Oct 30 16:44:53.980703 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:53.980681 4493 scope.go:122] "RemoveContainer" containerID="4705851b329f640def4ff8216f9f1ba1f7125fd9e6ddcebd77f72ad5437b3caa" Oct 30 16:44:53.989712 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:53.989694 4493 scope.go:122] "RemoveContainer" containerID="c380ce61c044d1c138352a4b070f6fc63caf9693031cc6380221910983280958" Oct 30 16:44:54.000458 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:54.000442 4493 scope.go:122] "RemoveContainer" containerID="4497fbd4cbf4941d63f0a1d26c0bccebcb22b9e07fa2df5213ba8f13fe17ad8e" Oct 30 16:44:54.008981 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:54.008966 4493 scope.go:122] "RemoveContainer" containerID="e983a58cedcd385a4dd4de4b69aba1caca3afd90c3b18bf30a6979567b3bec39" Oct 30 16:44:54.017725 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:54.017708 4493 scope.go:122] "RemoveContainer" containerID="87f6939a0a65da3d7e966fc97e24f5813ac63836965edbeb6e1aabda8263903e" Oct 30 16:44:54.026850 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:44:54.026832 4493 scope.go:122] "RemoveContainer" containerID="55ceeb47a0dc74d42a6a41c072201a12ba10be9f7bbf954d3cbad0e69d0fa47c" Oct 30 16:45:00.151728 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:00.146020 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889645-6xjd5"] Oct 30 16:45:00.159324 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:00.159289 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889645-tcj9f"] Oct 30 16:45:00.159488 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:00.159437 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889645-6xjd5" Oct 30 16:45:00.165750 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:00.165720 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889645-ldjdk"] Oct 30 16:45:00.165927 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:00.165890 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889645-tcj9f" Oct 30 16:45:00.170614 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:00.170131 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889645-kstp4"] Oct 30 16:45:00.170614 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:00.170254 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889645-ldjdk" Oct 30 16:45:00.180291 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:00.180262 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:45:00.180549 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:00.180529 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:45:00.180767 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:00.180743 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 16:45:00.183170 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:00.182852 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889645-zj2fw"] Oct 30 16:45:00.184747 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:00.183386 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889645-kstp4" Oct 30 16:45:00.192999 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:00.192904 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889645-6xjd5"] Oct 30 16:45:00.192999 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:00.192951 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-08fcc8-29889645-fr2qh"] Oct 30 16:45:00.196205 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:00.195898 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889645-zj2fw" Oct 30 16:45:00.198887 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:00.198861 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-504442-29889645-d8ph4"] Oct 30 16:45:00.199519 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:00.199360 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-08fcc8-29889645-fr2qh" Oct 30 16:45:00.229370 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:00.207960 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889645-ldjdk"] Oct 30 16:45:00.229370 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:00.208000 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889645-kstp4"] Oct 30 16:45:00.229370 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:00.208082 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-504442-29889645-d8ph4" Oct 30 16:45:00.241725 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:00.241694 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-504442-29889645-d8ph4"] Oct 30 16:45:00.244512 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:00.244401 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-08fcc8-29889645-fr2qh"] Oct 30 16:45:00.283814 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:00.276931 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889645-zj2fw"] Oct 30 16:45:00.283814 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:00.282095 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e8ec889-c0ea-4342-93f2-4f5b56639037-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-420ff3-29889645-6xjd5\" (UID: \"1e8ec889-c0ea-4342-93f2-4f5b56639037\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889645-6xjd5" Oct 30 16:45:00.283814 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:00.282206 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7j2k\" (UniqueName: \"kubernetes.io/projected/70a7611f-e41f-4bca-ba61-24ca0d0a3464-kube-api-access-c7j2k\") pod \"maas-api-key-cleanup-e2e-ipp-a-504442-29889645-d8ph4\" (UID: \"70a7611f-e41f-4bca-ba61-24ca0d0a3464\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-504442-29889645-d8ph4" Oct 30 16:45:00.283814 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:00.282249 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8430536-01ed-4fab-833a-176ac3939e9c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-a77b48-29889645-zj2fw\" (UID: \"e8430536-01ed-4fab-833a-176ac3939e9c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889645-zj2fw" Oct 30 16:45:00.283814 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:00.282284 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ed89856-cce2-41c5-b13b-348bf55b3825-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-08fcc8-29889645-fr2qh\" (UID: \"0ed89856-cce2-41c5-b13b-348bf55b3825\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-08fcc8-29889645-fr2qh" Oct 30 16:45:00.283814 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:00.282353 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hqbh\" (UniqueName: \"kubernetes.io/projected/ed948051-f611-435b-82b4-3a30d53d99f1-kube-api-access-8hqbh\") pod \"maas-api-key-cleanup-e2e-worker-w0-f3503b-29889645-kstp4\" (UID: \"ed948051-f611-435b-82b4-3a30d53d99f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889645-kstp4" Oct 30 16:45:00.283814 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:00.282395 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj9g7\" (UniqueName: \"kubernetes.io/projected/e8430536-01ed-4fab-833a-176ac3939e9c-kube-api-access-vj9g7\") pod \"maas-api-key-cleanup-e2e-worker-w1-a77b48-29889645-zj2fw\" (UID: \"e8430536-01ed-4fab-833a-176ac3939e9c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889645-zj2fw" Oct 30 16:45:00.283814 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:00.282427 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70a7611f-e41f-4bca-ba61-24ca0d0a3464-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-504442-29889645-d8ph4\" (UID: \"70a7611f-e41f-4bca-ba61-24ca0d0a3464\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-504442-29889645-d8ph4" Oct 30 16:45:00.283814 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:00.282489 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvkgg\" (UniqueName: \"kubernetes.io/projected/09d7b91f-5649-41b5-921c-80b1d25ddcb9-kube-api-access-tvkgg\") pod \"maas-api-key-cleanup-e2e-worker-w3-2d365c-29889645-tcj9f\" (UID: \"09d7b91f-5649-41b5-921c-80b1d25ddcb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889645-tcj9f" Oct 30 16:45:00.283814 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:00.282613 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5tpc\" (UniqueName: \"kubernetes.io/projected/0ed89856-cce2-41c5-b13b-348bf55b3825-kube-api-access-p5tpc\") pod \"maas-api-key-cleanup-e2e-api-a-08fcc8-29889645-fr2qh\" (UID: \"0ed89856-cce2-41c5-b13b-348bf55b3825\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-08fcc8-29889645-fr2qh" Oct 30 16:45:00.283814 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:00.282672 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e170303e-4547-4a45-8807-94e735e4ea91-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-18519c-29889645-ldjdk\" (UID: \"e170303e-4547-4a45-8807-94e735e4ea91\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889645-ldjdk" Oct 30 16:45:00.283814 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:00.282705 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9vd7\" (UniqueName: \"kubernetes.io/projected/e170303e-4547-4a45-8807-94e735e4ea91-kube-api-access-n9vd7\") pod \"maas-api-key-cleanup-e2e-worker-w2-18519c-29889645-ldjdk\" (UID: \"e170303e-4547-4a45-8807-94e735e4ea91\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889645-ldjdk" Oct 30 16:45:00.283814 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:00.282752 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed948051-f611-435b-82b4-3a30d53d99f1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f3503b-29889645-kstp4\" (UID: \"ed948051-f611-435b-82b4-3a30d53d99f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889645-kstp4" Oct 30 16:45:00.283814 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:00.282934 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5wbq\" (UniqueName: \"kubernetes.io/projected/1e8ec889-c0ea-4342-93f2-4f5b56639037-kube-api-access-z5wbq\") pod \"maas-api-key-cleanup-e2e-worker-w4-420ff3-29889645-6xjd5\" (UID: \"1e8ec889-c0ea-4342-93f2-4f5b56639037\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889645-6xjd5" Oct 30 16:45:00.283814 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:00.283041 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09d7b91f-5649-41b5-921c-80b1d25ddcb9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-2d365c-29889645-tcj9f\" (UID: \"09d7b91f-5649-41b5-921c-80b1d25ddcb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889645-tcj9f" Oct 30 16:45:00.292516 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:00.292488 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889645-tcj9f"] Oct 30 16:45:00.359687 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:45:00.359634 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 16:45:00.384908 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:00.384874 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7j2k\" (UniqueName: \"kubernetes.io/projected/70a7611f-e41f-4bca-ba61-24ca0d0a3464-kube-api-access-c7j2k\") pod \"maas-api-key-cleanup-e2e-ipp-a-504442-29889645-d8ph4\" (UID: \"70a7611f-e41f-4bca-ba61-24ca0d0a3464\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-504442-29889645-d8ph4" Oct 30 16:45:00.385069 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:00.384931 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8430536-01ed-4fab-833a-176ac3939e9c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-a77b48-29889645-zj2fw\" (UID: \"e8430536-01ed-4fab-833a-176ac3939e9c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889645-zj2fw" Oct 30 16:45:00.385069 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:00.384959 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ed89856-cce2-41c5-b13b-348bf55b3825-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-08fcc8-29889645-fr2qh\" (UID: \"0ed89856-cce2-41c5-b13b-348bf55b3825\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-08fcc8-29889645-fr2qh" Oct 30 16:45:00.385069 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:00.385030 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hqbh\" (UniqueName: \"kubernetes.io/projected/ed948051-f611-435b-82b4-3a30d53d99f1-kube-api-access-8hqbh\") pod \"maas-api-key-cleanup-e2e-worker-w0-f3503b-29889645-kstp4\" (UID: \"ed948051-f611-435b-82b4-3a30d53d99f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889645-kstp4" Oct 30 16:45:00.385069 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:00.385062 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj9g7\" (UniqueName: \"kubernetes.io/projected/e8430536-01ed-4fab-833a-176ac3939e9c-kube-api-access-vj9g7\") pod \"maas-api-key-cleanup-e2e-worker-w1-a77b48-29889645-zj2fw\" (UID: \"e8430536-01ed-4fab-833a-176ac3939e9c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889645-zj2fw" Oct 30 16:45:00.385274 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:00.385089 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70a7611f-e41f-4bca-ba61-24ca0d0a3464-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-504442-29889645-d8ph4\" (UID: \"70a7611f-e41f-4bca-ba61-24ca0d0a3464\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-504442-29889645-d8ph4" Oct 30 16:45:00.385274 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:00.385147 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvkgg\" (UniqueName: \"kubernetes.io/projected/09d7b91f-5649-41b5-921c-80b1d25ddcb9-kube-api-access-tvkgg\") pod \"maas-api-key-cleanup-e2e-worker-w3-2d365c-29889645-tcj9f\" (UID: \"09d7b91f-5649-41b5-921c-80b1d25ddcb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889645-tcj9f" Oct 30 16:45:00.385388 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:00.385279 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5tpc\" (UniqueName: \"kubernetes.io/projected/0ed89856-cce2-41c5-b13b-348bf55b3825-kube-api-access-p5tpc\") pod \"maas-api-key-cleanup-e2e-api-a-08fcc8-29889645-fr2qh\" (UID: \"0ed89856-cce2-41c5-b13b-348bf55b3825\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-08fcc8-29889645-fr2qh" Oct 30 16:45:00.385388 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:00.385334 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e170303e-4547-4a45-8807-94e735e4ea91-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-18519c-29889645-ldjdk\" (UID: \"e170303e-4547-4a45-8807-94e735e4ea91\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889645-ldjdk" Oct 30 16:45:00.385388 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:00.385359 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9vd7\" (UniqueName: \"kubernetes.io/projected/e170303e-4547-4a45-8807-94e735e4ea91-kube-api-access-n9vd7\") pod \"maas-api-key-cleanup-e2e-worker-w2-18519c-29889645-ldjdk\" (UID: \"e170303e-4547-4a45-8807-94e735e4ea91\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889645-ldjdk" Oct 30 16:45:00.385543 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:00.385399 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed948051-f611-435b-82b4-3a30d53d99f1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f3503b-29889645-kstp4\" (UID: \"ed948051-f611-435b-82b4-3a30d53d99f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889645-kstp4" Oct 30 16:45:00.385543 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:00.385427 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5wbq\" (UniqueName: \"kubernetes.io/projected/1e8ec889-c0ea-4342-93f2-4f5b56639037-kube-api-access-z5wbq\") pod \"maas-api-key-cleanup-e2e-worker-w4-420ff3-29889645-6xjd5\" (UID: \"1e8ec889-c0ea-4342-93f2-4f5b56639037\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889645-6xjd5" Oct 30 16:45:00.385543 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:00.385457 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09d7b91f-5649-41b5-921c-80b1d25ddcb9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-2d365c-29889645-tcj9f\" (UID: \"09d7b91f-5649-41b5-921c-80b1d25ddcb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889645-tcj9f" Oct 30 16:45:00.385543 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:00.385512 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e8ec889-c0ea-4342-93f2-4f5b56639037-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-420ff3-29889645-6xjd5\" (UID: \"1e8ec889-c0ea-4342-93f2-4f5b56639037\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889645-6xjd5" Oct 30 16:45:00.385880 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:00.385850 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ed89856-cce2-41c5-b13b-348bf55b3825-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-08fcc8-29889645-fr2qh\" (UID: \"0ed89856-cce2-41c5-b13b-348bf55b3825\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-08fcc8-29889645-fr2qh" Oct 30 16:45:00.386003 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:00.385913 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8430536-01ed-4fab-833a-176ac3939e9c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-a77b48-29889645-zj2fw\" (UID: \"e8430536-01ed-4fab-833a-176ac3939e9c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889645-zj2fw" Oct 30 16:45:00.386003 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:00.385937 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70a7611f-e41f-4bca-ba61-24ca0d0a3464-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-504442-29889645-d8ph4\" (UID: \"70a7611f-e41f-4bca-ba61-24ca0d0a3464\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-504442-29889645-d8ph4" Oct 30 16:45:00.386243 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:00.386208 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e170303e-4547-4a45-8807-94e735e4ea91-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-18519c-29889645-ldjdk\" (UID: \"e170303e-4547-4a45-8807-94e735e4ea91\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889645-ldjdk" Oct 30 16:45:00.386413 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:00.386385 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e8ec889-c0ea-4342-93f2-4f5b56639037-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-420ff3-29889645-6xjd5\" (UID: \"1e8ec889-c0ea-4342-93f2-4f5b56639037\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889645-6xjd5" Oct 30 16:45:00.386535 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:00.386513 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed948051-f611-435b-82b4-3a30d53d99f1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f3503b-29889645-kstp4\" (UID: \"ed948051-f611-435b-82b4-3a30d53d99f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889645-kstp4" Oct 30 16:45:00.386753 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:00.386737 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09d7b91f-5649-41b5-921c-80b1d25ddcb9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-2d365c-29889645-tcj9f\" (UID: \"09d7b91f-5649-41b5-921c-80b1d25ddcb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889645-tcj9f" Oct 30 16:45:00.408677 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:00.407251 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7j2k\" (UniqueName: \"kubernetes.io/projected/70a7611f-e41f-4bca-ba61-24ca0d0a3464-kube-api-access-c7j2k\") pod \"maas-api-key-cleanup-e2e-ipp-a-504442-29889645-d8ph4\" (UID: \"70a7611f-e41f-4bca-ba61-24ca0d0a3464\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-504442-29889645-d8ph4" Oct 30 16:45:00.411866 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:00.409696 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9vd7\" (UniqueName: \"kubernetes.io/projected/e170303e-4547-4a45-8807-94e735e4ea91-kube-api-access-n9vd7\") pod \"maas-api-key-cleanup-e2e-worker-w2-18519c-29889645-ldjdk\" (UID: \"e170303e-4547-4a45-8807-94e735e4ea91\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889645-ldjdk" Oct 30 16:45:00.411866 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:00.409916 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5tpc\" (UniqueName: \"kubernetes.io/projected/0ed89856-cce2-41c5-b13b-348bf55b3825-kube-api-access-p5tpc\") pod \"maas-api-key-cleanup-e2e-api-a-08fcc8-29889645-fr2qh\" (UID: \"0ed89856-cce2-41c5-b13b-348bf55b3825\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-08fcc8-29889645-fr2qh" Oct 30 16:45:00.411866 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:00.410337 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj9g7\" (UniqueName: \"kubernetes.io/projected/e8430536-01ed-4fab-833a-176ac3939e9c-kube-api-access-vj9g7\") pod \"maas-api-key-cleanup-e2e-worker-w1-a77b48-29889645-zj2fw\" (UID: \"e8430536-01ed-4fab-833a-176ac3939e9c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889645-zj2fw" Oct 30 16:45:00.411866 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:00.410904 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hqbh\" (UniqueName: \"kubernetes.io/projected/ed948051-f611-435b-82b4-3a30d53d99f1-kube-api-access-8hqbh\") pod \"maas-api-key-cleanup-e2e-worker-w0-f3503b-29889645-kstp4\" (UID: \"ed948051-f611-435b-82b4-3a30d53d99f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889645-kstp4" Oct 30 16:45:00.411866 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:00.411207 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5wbq\" (UniqueName: \"kubernetes.io/projected/1e8ec889-c0ea-4342-93f2-4f5b56639037-kube-api-access-z5wbq\") pod \"maas-api-key-cleanup-e2e-worker-w4-420ff3-29889645-6xjd5\" (UID: \"1e8ec889-c0ea-4342-93f2-4f5b56639037\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889645-6xjd5" Oct 30 16:45:00.411866 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:00.411418 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvkgg\" (UniqueName: \"kubernetes.io/projected/09d7b91f-5649-41b5-921c-80b1d25ddcb9-kube-api-access-tvkgg\") pod \"maas-api-key-cleanup-e2e-worker-w3-2d365c-29889645-tcj9f\" (UID: \"09d7b91f-5649-41b5-921c-80b1d25ddcb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889645-tcj9f" Oct 30 16:45:00.498061 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:00.498020 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889645-6xjd5" Oct 30 16:45:00.533773 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:00.533523 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889645-tcj9f" Oct 30 16:45:00.547931 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:00.547675 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889645-kstp4" Oct 30 16:45:00.557707 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:00.557642 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889645-ldjdk" Oct 30 16:45:00.581805 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:00.581158 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889645-zj2fw" Oct 30 16:45:00.593661 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:00.591657 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-08fcc8-29889645-fr2qh" Oct 30 16:45:00.611061 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:00.611034 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-504442-29889645-d8ph4" Oct 30 16:45:00.864600 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:00.864569 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889645-6xjd5"] Oct 30 16:45:00.892973 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:00.892710 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889645-tcj9f"] Oct 30 16:45:00.896661 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:45:00.895867 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09d7b91f_5649_41b5_921c_80b1d25ddcb9.slice/crio-77b7a50b0d4ed783154063d19479757e78284d50e59ff10231ffdce639e5dce4 WatchSource:0}: Error finding container 77b7a50b0d4ed783154063d19479757e78284d50e59ff10231ffdce639e5dce4: Status 404 returned error can't find the container with id 77b7a50b0d4ed783154063d19479757e78284d50e59ff10231ffdce639e5dce4 Oct 30 16:45:01.044836 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:01.044806 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889645-zj2fw"] Oct 30 16:45:01.095201 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:01.095170 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-08fcc8-29889645-fr2qh"] Oct 30 16:45:01.121829 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:01.121798 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889645-kstp4"] Oct 30 16:45:01.217686 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:01.217661 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889645-ldjdk"] Oct 30 16:45:01.265439 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:01.265393 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-504442-29889645-d8ph4"] Oct 30 16:45:01.316872 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:01.314605 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889645-kstp4" event={"ID":"ed948051-f611-435b-82b4-3a30d53d99f1","Type":"ContainerStarted","Data":"5eb7d835224c165dbd8d05cd1da613535668ab1b5c440551db29a9d4d5c6b6c5"} Oct 30 16:45:01.319889 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:01.319832 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-08fcc8-29889645-fr2qh" event={"ID":"0ed89856-cce2-41c5-b13b-348bf55b3825","Type":"ContainerStarted","Data":"1e43f502a0f3cf7ab3426d1a13e8ce3eab92323c637daeb51469a8751882ef48"} Oct 30 16:45:01.322666 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:01.322615 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889645-tcj9f" event={"ID":"09d7b91f-5649-41b5-921c-80b1d25ddcb9","Type":"ContainerStarted","Data":"77b7a50b0d4ed783154063d19479757e78284d50e59ff10231ffdce639e5dce4"} Oct 30 16:45:01.327690 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:01.326830 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889645-zj2fw" event={"ID":"e8430536-01ed-4fab-833a-176ac3939e9c","Type":"ContainerStarted","Data":"1c4082763fabbe7ba7603022bedfe4a12bd69349f04c249a44b26eef3f91895b"} Oct 30 16:45:01.327690 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:01.326887 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889645-6xjd5" event={"ID":"1e8ec889-c0ea-4342-93f2-4f5b56639037","Type":"ContainerStarted","Data":"0b1a54fe552af5b812d0d7b5068003c8edc81db2d8b0ebe6a4d88196a61886a4"} Oct 30 16:45:01.407177 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:45:01.407133 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70a7611f_e41f_4bca_ba61_24ca0d0a3464.slice/crio-e46e1e6bd596430928472fea86fb69ab84d3cf07cabf629d3a94fca11758a5a8 WatchSource:0}: Error finding container e46e1e6bd596430928472fea86fb69ab84d3cf07cabf629d3a94fca11758a5a8: Status 404 returned error can't find the container with id e46e1e6bd596430928472fea86fb69ab84d3cf07cabf629d3a94fca11758a5a8 Oct 30 16:45:02.029321 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:02.029288 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 16:45:02.339627 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:02.339575 4493 generic.go:356] "Generic (PLEG): container finished" podID="ed948051-f611-435b-82b4-3a30d53d99f1" containerID="62c7b1ac9927b7ae9081d644e480b09956e9122ca2953f0f0ff955ffff5f742f" exitCode=60 Oct 30 16:45:02.340165 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:02.339674 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889645-kstp4" event={"ID":"ed948051-f611-435b-82b4-3a30d53d99f1","Type":"ContainerDied","Data":"62c7b1ac9927b7ae9081d644e480b09956e9122ca2953f0f0ff955ffff5f742f"} Oct 30 16:45:02.340165 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:02.340127 4493 scope.go:122] "RemoveContainer" containerID="62c7b1ac9927b7ae9081d644e480b09956e9122ca2953f0f0ff955ffff5f742f" Oct 30 16:45:02.343268 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:02.343235 4493 generic.go:356] "Generic (PLEG): container finished" podID="0ed89856-cce2-41c5-b13b-348bf55b3825" containerID="167f51a135d8fe35f6bf87057a19e5ece1a74e31f1631bc355f563e58e9ccb6b" exitCode=60 Oct 30 16:45:02.343407 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:02.343321 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-08fcc8-29889645-fr2qh" event={"ID":"0ed89856-cce2-41c5-b13b-348bf55b3825","Type":"ContainerDied","Data":"167f51a135d8fe35f6bf87057a19e5ece1a74e31f1631bc355f563e58e9ccb6b"} Oct 30 16:45:02.343600 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:02.343579 4493 scope.go:122] "RemoveContainer" containerID="167f51a135d8fe35f6bf87057a19e5ece1a74e31f1631bc355f563e58e9ccb6b" Oct 30 16:45:02.350914 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:02.347619 4493 generic.go:356] "Generic (PLEG): container finished" podID="09d7b91f-5649-41b5-921c-80b1d25ddcb9" containerID="4909d9ee5e4555d68e8ce38ab6c30cc9e5ba47952dde625fe5c874bae46affb0" exitCode=60 Oct 30 16:45:02.350914 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:02.347696 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889645-tcj9f" event={"ID":"09d7b91f-5649-41b5-921c-80b1d25ddcb9","Type":"ContainerDied","Data":"4909d9ee5e4555d68e8ce38ab6c30cc9e5ba47952dde625fe5c874bae46affb0"} Oct 30 16:45:02.350914 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:02.348060 4493 scope.go:122] "RemoveContainer" containerID="4909d9ee5e4555d68e8ce38ab6c30cc9e5ba47952dde625fe5c874bae46affb0" Oct 30 16:45:02.375984 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:02.375909 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-504442-29889645-d8ph4" event={"ID":"70a7611f-e41f-4bca-ba61-24ca0d0a3464","Type":"ContainerStarted","Data":"b9e76e08a9975369270b575678883a5c789390ff6105d85b47893a433b654b5c"} Oct 30 16:45:02.375984 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:02.375977 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-504442-29889645-d8ph4" event={"ID":"70a7611f-e41f-4bca-ba61-24ca0d0a3464","Type":"ContainerStarted","Data":"e46e1e6bd596430928472fea86fb69ab84d3cf07cabf629d3a94fca11758a5a8"} Oct 30 16:45:02.386513 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:02.386245 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889645-ldjdk" event={"ID":"e170303e-4547-4a45-8807-94e735e4ea91","Type":"ContainerStarted","Data":"01dec72041e6d96215b05d6842d46438d88462a80bb1820d0384c6b39f8e91b3"} Oct 30 16:45:02.387906 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:02.387880 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889645-ldjdk" event={"ID":"e170303e-4547-4a45-8807-94e735e4ea91","Type":"ContainerStarted","Data":"c27eb05e30371cf63988fd14a36d458893866521cf1c6b17152ebdeea602d761"} Oct 30 16:45:02.393438 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:02.393407 4493 generic.go:356] "Generic (PLEG): container finished" podID="e8430536-01ed-4fab-833a-176ac3939e9c" containerID="667008342b6d647678f13597e1b23418e9c8fd351ecc12e20247391894eea792" exitCode=60 Oct 30 16:45:02.393550 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:02.393489 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889645-zj2fw" event={"ID":"e8430536-01ed-4fab-833a-176ac3939e9c","Type":"ContainerDied","Data":"667008342b6d647678f13597e1b23418e9c8fd351ecc12e20247391894eea792"} Oct 30 16:45:02.393988 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:02.393956 4493 scope.go:122] "RemoveContainer" containerID="667008342b6d647678f13597e1b23418e9c8fd351ecc12e20247391894eea792" Oct 30 16:45:02.398908 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:02.397652 4493 generic.go:356] "Generic (PLEG): container finished" podID="1e8ec889-c0ea-4342-93f2-4f5b56639037" containerID="74541b1762466c15662e28171cb363946c64b332431afaddef1e0d8f53d31d3f" exitCode=60 Oct 30 16:45:02.398908 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:02.397679 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889645-6xjd5" event={"ID":"1e8ec889-c0ea-4342-93f2-4f5b56639037","Type":"ContainerDied","Data":"74541b1762466c15662e28171cb363946c64b332431afaddef1e0d8f53d31d3f"} Oct 30 16:45:02.398908 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:02.398002 4493 scope.go:122] "RemoveContainer" containerID="74541b1762466c15662e28171cb363946c64b332431afaddef1e0d8f53d31d3f" Oct 30 16:45:02.506202 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:02.506145 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-504442-29889645-d8ph4" podStartSLOduration=2.506127687 podStartE2EDuration="2.506127687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:45:02.503483872 +0000 UTC m=+1329.730658971" watchObservedRunningTime="2026-10-30 16:45:02.506127687 +0000 UTC m=+1329.733302788" Oct 30 16:45:02.543662 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:02.543608 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889645-ldjdk" podStartSLOduration=2.543592551 podStartE2EDuration="2.543592551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:45:02.539158069 +0000 UTC m=+1329.766333170" watchObservedRunningTime="2026-10-30 16:45:02.543592551 +0000 UTC m=+1329.770767659" Oct 30 16:45:03.414474 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:03.413649 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889645-kstp4" event={"ID":"ed948051-f611-435b-82b4-3a30d53d99f1","Type":"ContainerStarted","Data":"debe000b8c6b429c28c6b9fd8a6e19b84e4a45054f54476ff2164921677a5664"} Oct 30 16:45:03.416872 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:03.416224 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-08fcc8-29889645-fr2qh" event={"ID":"0ed89856-cce2-41c5-b13b-348bf55b3825","Type":"ContainerStarted","Data":"7a92ba61e0477adb319151b23c0482d71d25ca387fe76cdce58ce2b54f734a73"} Oct 30 16:45:03.419981 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:03.419955 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889645-tcj9f" event={"ID":"09d7b91f-5649-41b5-921c-80b1d25ddcb9","Type":"ContainerStarted","Data":"28e34dc2402193875efadf268b029c199896da18b3dbb420f1d0e997ec72a2ce"} Oct 30 16:45:03.434799 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:03.422558 4493 generic.go:356] "Generic (PLEG): container finished" podID="70a7611f-e41f-4bca-ba61-24ca0d0a3464" containerID="b9e76e08a9975369270b575678883a5c789390ff6105d85b47893a433b654b5c" exitCode=60 Oct 30 16:45:03.434799 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:03.422635 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-504442-29889645-d8ph4" event={"ID":"70a7611f-e41f-4bca-ba61-24ca0d0a3464","Type":"ContainerDied","Data":"b9e76e08a9975369270b575678883a5c789390ff6105d85b47893a433b654b5c"} Oct 30 16:45:03.434799 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:03.422925 4493 scope.go:122] "RemoveContainer" containerID="b9e76e08a9975369270b575678883a5c789390ff6105d85b47893a433b654b5c" Oct 30 16:45:03.434799 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:03.427299 4493 generic.go:356] "Generic (PLEG): container finished" podID="e170303e-4547-4a45-8807-94e735e4ea91" containerID="01dec72041e6d96215b05d6842d46438d88462a80bb1820d0384c6b39f8e91b3" exitCode=60 Oct 30 16:45:03.434799 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:03.427648 4493 scope.go:122] "RemoveContainer" containerID="01dec72041e6d96215b05d6842d46438d88462a80bb1820d0384c6b39f8e91b3" Oct 30 16:45:03.434799 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:03.427990 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889645-ldjdk" event={"ID":"e170303e-4547-4a45-8807-94e735e4ea91","Type":"ContainerDied","Data":"01dec72041e6d96215b05d6842d46438d88462a80bb1820d0384c6b39f8e91b3"} Oct 30 16:45:03.441151 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:03.438253 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889645-zj2fw" event={"ID":"e8430536-01ed-4fab-833a-176ac3939e9c","Type":"ContainerStarted","Data":"7a066210d837b4a4553e4e43d78d518597044a229f8a6a627bd9b36175d11e2c"} Oct 30 16:45:03.441151 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:03.439171 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889645-6xjd5" event={"ID":"1e8ec889-c0ea-4342-93f2-4f5b56639037","Type":"ContainerStarted","Data":"2c89e9bdc90fca985ce5616802ed6abd9867f8f9d91b33c024ed1b9d8d43c692"} Oct 30 16:45:03.459164 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:03.459115 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889645-kstp4" podStartSLOduration=3.459096305 podStartE2EDuration="3.459096305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:45:03.455110971 +0000 UTC m=+1330.682286073" watchObservedRunningTime="2026-10-30 16:45:03.459096305 +0000 UTC m=+1330.686271406" Oct 30 16:45:03.533380 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:03.533322 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889645-6xjd5" podStartSLOduration=3.53330734 podStartE2EDuration="3.53330734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:45:03.531544883 +0000 UTC m=+1330.758719983" watchObservedRunningTime="2026-10-30 16:45:03.53330734 +0000 UTC m=+1330.760482439" Oct 30 16:45:03.533812 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:03.533699 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889645-zj2fw" podStartSLOduration=3.533691453 podStartE2EDuration="3.533691453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:45:03.495728311 +0000 UTC m=+1330.722903411" watchObservedRunningTime="2026-10-30 16:45:03.533691453 +0000 UTC m=+1330.760866557" Oct 30 16:45:03.574278 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:03.574215 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-08fcc8-29889645-fr2qh" podStartSLOduration=3.574197698 podStartE2EDuration="3.574197698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:45:03.56745962 +0000 UTC m=+1330.794634719" watchObservedRunningTime="2026-10-30 16:45:03.574197698 +0000 UTC m=+1330.801372798" Oct 30 16:45:03.695495 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:03.695369 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889645-tcj9f" podStartSLOduration=3.6953480279999997 podStartE2EDuration="3.695348028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:45:03.691403003 +0000 UTC m=+1330.918578104" watchObservedRunningTime="2026-10-30 16:45:03.695348028 +0000 UTC m=+1330.922523129" Oct 30 16:45:04.449462 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:04.449429 4493 generic.go:356] "Generic (PLEG): container finished" podID="ed948051-f611-435b-82b4-3a30d53d99f1" containerID="debe000b8c6b429c28c6b9fd8a6e19b84e4a45054f54476ff2164921677a5664" exitCode=60 Oct 30 16:45:04.449868 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:04.449517 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889645-kstp4" event={"ID":"ed948051-f611-435b-82b4-3a30d53d99f1","Type":"ContainerDied","Data":"debe000b8c6b429c28c6b9fd8a6e19b84e4a45054f54476ff2164921677a5664"} Oct 30 16:45:04.449868 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:04.449631 4493 scope.go:122] "RemoveContainer" containerID="62c7b1ac9927b7ae9081d644e480b09956e9122ca2953f0f0ff955ffff5f742f" Oct 30 16:45:04.449868 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:04.449816 4493 scope.go:122] "RemoveContainer" containerID="debe000b8c6b429c28c6b9fd8a6e19b84e4a45054f54476ff2164921677a5664" Oct 30 16:45:04.450171 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:45:04.450145 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f3503b-29889645-kstp4_odh-ai-gateway-infra(ed948051-f611-435b-82b4-3a30d53d99f1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889645-kstp4" podUID="ed948051-f611-435b-82b4-3a30d53d99f1" Oct 30 16:45:04.452141 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:04.452114 4493 generic.go:356] "Generic (PLEG): container finished" podID="0ed89856-cce2-41c5-b13b-348bf55b3825" containerID="7a92ba61e0477adb319151b23c0482d71d25ca387fe76cdce58ce2b54f734a73" exitCode=60 Oct 30 16:45:04.452255 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:04.452210 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-08fcc8-29889645-fr2qh" event={"ID":"0ed89856-cce2-41c5-b13b-348bf55b3825","Type":"ContainerDied","Data":"7a92ba61e0477adb319151b23c0482d71d25ca387fe76cdce58ce2b54f734a73"} Oct 30 16:45:04.452430 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:04.452412 4493 scope.go:122] "RemoveContainer" containerID="7a92ba61e0477adb319151b23c0482d71d25ca387fe76cdce58ce2b54f734a73" Oct 30 16:45:04.452725 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:45:04.452705 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-08fcc8-29889645-fr2qh_odh-ai-gateway-infra(0ed89856-cce2-41c5-b13b-348bf55b3825)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-08fcc8-29889645-fr2qh" podUID="0ed89856-cce2-41c5-b13b-348bf55b3825" Oct 30 16:45:04.454642 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:04.454617 4493 generic.go:356] "Generic (PLEG): container finished" podID="09d7b91f-5649-41b5-921c-80b1d25ddcb9" containerID="28e34dc2402193875efadf268b029c199896da18b3dbb420f1d0e997ec72a2ce" exitCode=60 Oct 30 16:45:04.454762 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:04.454694 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889645-tcj9f" event={"ID":"09d7b91f-5649-41b5-921c-80b1d25ddcb9","Type":"ContainerDied","Data":"28e34dc2402193875efadf268b029c199896da18b3dbb420f1d0e997ec72a2ce"} Oct 30 16:45:04.454946 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:04.454928 4493 scope.go:122] "RemoveContainer" containerID="28e34dc2402193875efadf268b029c199896da18b3dbb420f1d0e997ec72a2ce" Oct 30 16:45:04.455218 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:45:04.455195 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-2d365c-29889645-tcj9f_odh-ai-gateway-infra(09d7b91f-5649-41b5-921c-80b1d25ddcb9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889645-tcj9f" podUID="09d7b91f-5649-41b5-921c-80b1d25ddcb9" Oct 30 16:45:04.457378 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:04.457356 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-504442-29889645-d8ph4" event={"ID":"70a7611f-e41f-4bca-ba61-24ca0d0a3464","Type":"ContainerStarted","Data":"201722b48f2b67b88a9aadf7c8d0feccac582c2031017fd51af49eb5eb4d6790"} Oct 30 16:45:04.459551 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:04.459515 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889645-ldjdk" event={"ID":"e170303e-4547-4a45-8807-94e735e4ea91","Type":"ContainerStarted","Data":"ebb11455fd4a1fb4430dc73b16b5d71e633f57c3296df952b9374bcf86e85f27"} Oct 30 16:45:04.461452 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:04.461432 4493 generic.go:356] "Generic (PLEG): container finished" podID="e8430536-01ed-4fab-833a-176ac3939e9c" containerID="7a066210d837b4a4553e4e43d78d518597044a229f8a6a627bd9b36175d11e2c" exitCode=60 Oct 30 16:45:04.461539 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:04.461494 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889645-zj2fw" event={"ID":"e8430536-01ed-4fab-833a-176ac3939e9c","Type":"ContainerDied","Data":"7a066210d837b4a4553e4e43d78d518597044a229f8a6a627bd9b36175d11e2c"} Oct 30 16:45:04.461689 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:04.461674 4493 scope.go:122] "RemoveContainer" containerID="7a066210d837b4a4553e4e43d78d518597044a229f8a6a627bd9b36175d11e2c" Oct 30 16:45:04.462019 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:45:04.461998 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-a77b48-29889645-zj2fw_odh-ai-gateway-infra(e8430536-01ed-4fab-833a-176ac3939e9c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889645-zj2fw" podUID="e8430536-01ed-4fab-833a-176ac3939e9c" Oct 30 16:45:04.463531 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:04.463511 4493 generic.go:356] "Generic (PLEG): container finished" podID="1e8ec889-c0ea-4342-93f2-4f5b56639037" containerID="2c89e9bdc90fca985ce5616802ed6abd9867f8f9d91b33c024ed1b9d8d43c692" exitCode=60 Oct 30 16:45:04.463610 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:04.463581 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889645-6xjd5" event={"ID":"1e8ec889-c0ea-4342-93f2-4f5b56639037","Type":"ContainerDied","Data":"2c89e9bdc90fca985ce5616802ed6abd9867f8f9d91b33c024ed1b9d8d43c692"} Oct 30 16:45:04.463853 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:04.463836 4493 scope.go:122] "RemoveContainer" containerID="2c89e9bdc90fca985ce5616802ed6abd9867f8f9d91b33c024ed1b9d8d43c692" Oct 30 16:45:04.464140 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:45:04.464116 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-420ff3-29889645-6xjd5_odh-ai-gateway-infra(1e8ec889-c0ea-4342-93f2-4f5b56639037)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889645-6xjd5" podUID="1e8ec889-c0ea-4342-93f2-4f5b56639037" Oct 30 16:45:04.464872 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:04.464858 4493 scope.go:122] "RemoveContainer" containerID="167f51a135d8fe35f6bf87057a19e5ece1a74e31f1631bc355f563e58e9ccb6b" Oct 30 16:45:04.478196 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:04.478177 4493 scope.go:122] "RemoveContainer" containerID="4909d9ee5e4555d68e8ce38ab6c30cc9e5ba47952dde625fe5c874bae46affb0" Oct 30 16:45:04.491503 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:04.491484 4493 scope.go:122] "RemoveContainer" containerID="667008342b6d647678f13597e1b23418e9c8fd351ecc12e20247391894eea792" Oct 30 16:45:04.505593 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:04.505569 4493 scope.go:122] "RemoveContainer" containerID="74541b1762466c15662e28171cb363946c64b332431afaddef1e0d8f53d31d3f" Oct 30 16:45:05.472905 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:05.472876 4493 scope.go:122] "RemoveContainer" containerID="2c89e9bdc90fca985ce5616802ed6abd9867f8f9d91b33c024ed1b9d8d43c692" Oct 30 16:45:05.473369 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:45:05.473244 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-420ff3-29889645-6xjd5_odh-ai-gateway-infra(1e8ec889-c0ea-4342-93f2-4f5b56639037)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889645-6xjd5" podUID="1e8ec889-c0ea-4342-93f2-4f5b56639037" Oct 30 16:45:05.474654 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:05.474632 4493 scope.go:122] "RemoveContainer" containerID="debe000b8c6b429c28c6b9fd8a6e19b84e4a45054f54476ff2164921677a5664" Oct 30 16:45:05.474897 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:45:05.474877 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f3503b-29889645-kstp4_odh-ai-gateway-infra(ed948051-f611-435b-82b4-3a30d53d99f1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889645-kstp4" podUID="ed948051-f611-435b-82b4-3a30d53d99f1" Oct 30 16:45:05.476123 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:05.476106 4493 scope.go:122] "RemoveContainer" containerID="7a92ba61e0477adb319151b23c0482d71d25ca387fe76cdce58ce2b54f734a73" Oct 30 16:45:05.476332 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:45:05.476307 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-08fcc8-29889645-fr2qh_odh-ai-gateway-infra(0ed89856-cce2-41c5-b13b-348bf55b3825)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-08fcc8-29889645-fr2qh" podUID="0ed89856-cce2-41c5-b13b-348bf55b3825" Oct 30 16:45:05.477663 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:05.477647 4493 scope.go:122] "RemoveContainer" containerID="28e34dc2402193875efadf268b029c199896da18b3dbb420f1d0e997ec72a2ce" Oct 30 16:45:05.477871 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:45:05.477850 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-2d365c-29889645-tcj9f_odh-ai-gateway-infra(09d7b91f-5649-41b5-921c-80b1d25ddcb9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889645-tcj9f" podUID="09d7b91f-5649-41b5-921c-80b1d25ddcb9" Oct 30 16:45:05.479301 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:05.479282 4493 generic.go:356] "Generic (PLEG): container finished" podID="70a7611f-e41f-4bca-ba61-24ca0d0a3464" containerID="201722b48f2b67b88a9aadf7c8d0feccac582c2031017fd51af49eb5eb4d6790" exitCode=60 Oct 30 16:45:05.479390 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:05.479367 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-504442-29889645-d8ph4" event={"ID":"70a7611f-e41f-4bca-ba61-24ca0d0a3464","Type":"ContainerDied","Data":"201722b48f2b67b88a9aadf7c8d0feccac582c2031017fd51af49eb5eb4d6790"} Oct 30 16:45:05.479445 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:05.479432 4493 scope.go:122] "RemoveContainer" containerID="b9e76e08a9975369270b575678883a5c789390ff6105d85b47893a433b654b5c" Oct 30 16:45:05.479541 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:05.479523 4493 scope.go:122] "RemoveContainer" containerID="201722b48f2b67b88a9aadf7c8d0feccac582c2031017fd51af49eb5eb4d6790" Oct 30 16:45:05.479749 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:45:05.479728 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-504442-29889645-d8ph4_odh-ai-gateway-infra(70a7611f-e41f-4bca-ba61-24ca0d0a3464)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-504442-29889645-d8ph4" podUID="70a7611f-e41f-4bca-ba61-24ca0d0a3464" Oct 30 16:45:05.481133 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:05.481113 4493 generic.go:356] "Generic (PLEG): container finished" podID="e170303e-4547-4a45-8807-94e735e4ea91" containerID="ebb11455fd4a1fb4430dc73b16b5d71e633f57c3296df952b9374bcf86e85f27" exitCode=60 Oct 30 16:45:05.481221 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:05.481186 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889645-ldjdk" event={"ID":"e170303e-4547-4a45-8807-94e735e4ea91","Type":"ContainerDied","Data":"ebb11455fd4a1fb4430dc73b16b5d71e633f57c3296df952b9374bcf86e85f27"} Oct 30 16:45:05.481348 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:05.481334 4493 scope.go:122] "RemoveContainer" containerID="ebb11455fd4a1fb4430dc73b16b5d71e633f57c3296df952b9374bcf86e85f27" Oct 30 16:45:05.481534 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:45:05.481519 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-18519c-29889645-ldjdk_odh-ai-gateway-infra(e170303e-4547-4a45-8807-94e735e4ea91)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889645-ldjdk" podUID="e170303e-4547-4a45-8807-94e735e4ea91" Oct 30 16:45:05.482928 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:05.482912 4493 scope.go:122] "RemoveContainer" containerID="7a066210d837b4a4553e4e43d78d518597044a229f8a6a627bd9b36175d11e2c" Oct 30 16:45:05.483091 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:45:05.483075 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-a77b48-29889645-zj2fw_odh-ai-gateway-infra(e8430536-01ed-4fab-833a-176ac3939e9c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889645-zj2fw" podUID="e8430536-01ed-4fab-833a-176ac3939e9c" Oct 30 16:45:05.492967 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:05.492950 4493 scope.go:122] "RemoveContainer" containerID="01dec72041e6d96215b05d6842d46438d88462a80bb1820d0384c6b39f8e91b3" Oct 30 16:45:06.492865 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:06.492836 4493 scope.go:122] "RemoveContainer" containerID="201722b48f2b67b88a9aadf7c8d0feccac582c2031017fd51af49eb5eb4d6790" Oct 30 16:45:06.493320 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:45:06.493164 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-504442-29889645-d8ph4_odh-ai-gateway-infra(70a7611f-e41f-4bca-ba61-24ca0d0a3464)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-504442-29889645-d8ph4" podUID="70a7611f-e41f-4bca-ba61-24ca0d0a3464" Oct 30 16:45:06.494524 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:06.494507 4493 scope.go:122] "RemoveContainer" containerID="ebb11455fd4a1fb4430dc73b16b5d71e633f57c3296df952b9374bcf86e85f27" Oct 30 16:45:06.494795 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:45:06.494765 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-18519c-29889645-ldjdk_odh-ai-gateway-infra(e170303e-4547-4a45-8807-94e735e4ea91)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889645-ldjdk" podUID="e170303e-4547-4a45-8807-94e735e4ea91" Oct 30 16:45:14.926559 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:14.926529 4493 scope.go:122] "RemoveContainer" containerID="debe000b8c6b429c28c6b9fd8a6e19b84e4a45054f54476ff2164921677a5664" Oct 30 16:45:14.927190 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:14.926645 4493 scope.go:122] "RemoveContainer" containerID="7a92ba61e0477adb319151b23c0482d71d25ca387fe76cdce58ce2b54f734a73" Oct 30 16:45:14.927190 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:14.926675 4493 scope.go:122] "RemoveContainer" containerID="28e34dc2402193875efadf268b029c199896da18b3dbb420f1d0e997ec72a2ce" Oct 30 16:45:14.927190 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:14.926907 4493 scope.go:122] "RemoveContainer" containerID="7a066210d837b4a4553e4e43d78d518597044a229f8a6a627bd9b36175d11e2c" Oct 30 16:45:15.596589 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:15.596558 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889645-kstp4" event={"ID":"ed948051-f611-435b-82b4-3a30d53d99f1","Type":"ContainerStarted","Data":"ff3328a272393895d66d7012c345e7944b35687ba6cb97630d8e33ff38b24fc1"} Oct 30 16:45:15.598585 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:15.598561 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-08fcc8-29889645-fr2qh" event={"ID":"0ed89856-cce2-41c5-b13b-348bf55b3825","Type":"ContainerStarted","Data":"4619cb6c6db3affdd785b74d5774fdf44fc2338212dce49d3178fc2715eac148"} Oct 30 16:45:15.600527 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:15.600507 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889645-tcj9f" event={"ID":"09d7b91f-5649-41b5-921c-80b1d25ddcb9","Type":"ContainerStarted","Data":"385ba56ef872cf42c085e27575ba12fe51dd68417779c3ed4f4ab17ff04996f0"} Oct 30 16:45:15.602399 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:15.602379 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889645-zj2fw" event={"ID":"e8430536-01ed-4fab-833a-176ac3939e9c","Type":"ContainerStarted","Data":"53b9c799e70b3b2963a6afe3c132f037203e327d480dc5eb32efe7cef7591de4"} Oct 30 16:45:15.730419 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:15.730375 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr"] Oct 30 16:45:15.730763 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:15.730740 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr" podUID="6b860be8-5304-4364-a0b6-e06906cf5f4f" containerName="istio-proxy" containerID="cri-o://8f5c470a5e709f6e2880deb6300e233b992e725dcc8a499b5ef07b7ced37b460" gracePeriod=30 Oct 30 16:45:15.925894 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:15.925869 4493 scope.go:122] "RemoveContainer" containerID="2c89e9bdc90fca985ce5616802ed6abd9867f8f9d91b33c024ed1b9d8d43c692" Oct 30 16:45:15.926169 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:15.926147 4493 scope.go:122] "RemoveContainer" containerID="ebb11455fd4a1fb4430dc73b16b5d71e633f57c3296df952b9374bcf86e85f27" Oct 30 16:45:16.615596 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:16.615496 4493 generic.go:356] "Generic (PLEG): container finished" podID="ed948051-f611-435b-82b4-3a30d53d99f1" containerID="ff3328a272393895d66d7012c345e7944b35687ba6cb97630d8e33ff38b24fc1" exitCode=60 Oct 30 16:45:16.615596 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:16.615562 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889645-kstp4" event={"ID":"ed948051-f611-435b-82b4-3a30d53d99f1","Type":"ContainerDied","Data":"ff3328a272393895d66d7012c345e7944b35687ba6cb97630d8e33ff38b24fc1"} Oct 30 16:45:16.616152 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:16.615637 4493 scope.go:122] "RemoveContainer" containerID="debe000b8c6b429c28c6b9fd8a6e19b84e4a45054f54476ff2164921677a5664" Oct 30 16:45:16.616152 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:16.616011 4493 scope.go:122] "RemoveContainer" containerID="ff3328a272393895d66d7012c345e7944b35687ba6cb97630d8e33ff38b24fc1" Oct 30 16:45:16.616315 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:45:16.616289 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f3503b-29889645-kstp4_odh-ai-gateway-infra(ed948051-f611-435b-82b4-3a30d53d99f1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889645-kstp4" podUID="ed948051-f611-435b-82b4-3a30d53d99f1" Oct 30 16:45:16.617835 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:16.617776 4493 generic.go:356] "Generic (PLEG): container finished" podID="0ed89856-cce2-41c5-b13b-348bf55b3825" containerID="4619cb6c6db3affdd785b74d5774fdf44fc2338212dce49d3178fc2715eac148" exitCode=60 Oct 30 16:45:16.617942 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:16.617815 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-08fcc8-29889645-fr2qh" event={"ID":"0ed89856-cce2-41c5-b13b-348bf55b3825","Type":"ContainerDied","Data":"4619cb6c6db3affdd785b74d5774fdf44fc2338212dce49d3178fc2715eac148"} Oct 30 16:45:16.620525 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:16.618171 4493 scope.go:122] "RemoveContainer" containerID="4619cb6c6db3affdd785b74d5774fdf44fc2338212dce49d3178fc2715eac148" Oct 30 16:45:16.620525 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:45:16.618389 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-08fcc8-29889645-fr2qh_odh-ai-gateway-infra(0ed89856-cce2-41c5-b13b-348bf55b3825)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-08fcc8-29889645-fr2qh" podUID="0ed89856-cce2-41c5-b13b-348bf55b3825" Oct 30 16:45:16.620525 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:16.620213 4493 generic.go:356] "Generic (PLEG): container finished" podID="09d7b91f-5649-41b5-921c-80b1d25ddcb9" containerID="385ba56ef872cf42c085e27575ba12fe51dd68417779c3ed4f4ab17ff04996f0" exitCode=60 Oct 30 16:45:16.620525 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:16.620277 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889645-tcj9f" event={"ID":"09d7b91f-5649-41b5-921c-80b1d25ddcb9","Type":"ContainerDied","Data":"385ba56ef872cf42c085e27575ba12fe51dd68417779c3ed4f4ab17ff04996f0"} Oct 30 16:45:16.620955 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:16.620589 4493 scope.go:122] "RemoveContainer" containerID="385ba56ef872cf42c085e27575ba12fe51dd68417779c3ed4f4ab17ff04996f0" Oct 30 16:45:16.620955 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:45:16.620834 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-2d365c-29889645-tcj9f_odh-ai-gateway-infra(09d7b91f-5649-41b5-921c-80b1d25ddcb9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889645-tcj9f" podUID="09d7b91f-5649-41b5-921c-80b1d25ddcb9" Oct 30 16:45:16.622591 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:16.622568 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889645-ldjdk" event={"ID":"e170303e-4547-4a45-8807-94e735e4ea91","Type":"ContainerStarted","Data":"2e3317012cbfcca83cecb29f285f13dce2ce47d7a5a55fbb5b5555dc07809244"} Oct 30 16:45:16.624544 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:16.624526 4493 generic.go:356] "Generic (PLEG): container finished" podID="e8430536-01ed-4fab-833a-176ac3939e9c" containerID="53b9c799e70b3b2963a6afe3c132f037203e327d480dc5eb32efe7cef7591de4" exitCode=60 Oct 30 16:45:16.624644 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:16.624588 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889645-zj2fw" event={"ID":"e8430536-01ed-4fab-833a-176ac3939e9c","Type":"ContainerDied","Data":"53b9c799e70b3b2963a6afe3c132f037203e327d480dc5eb32efe7cef7591de4"} Oct 30 16:45:16.625050 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:16.625033 4493 scope.go:122] "RemoveContainer" containerID="53b9c799e70b3b2963a6afe3c132f037203e327d480dc5eb32efe7cef7591de4" Oct 30 16:45:16.625272 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:45:16.625254 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-a77b48-29889645-zj2fw_odh-ai-gateway-infra(e8430536-01ed-4fab-833a-176ac3939e9c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889645-zj2fw" podUID="e8430536-01ed-4fab-833a-176ac3939e9c" Oct 30 16:45:16.628968 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:16.628942 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889645-6xjd5" event={"ID":"1e8ec889-c0ea-4342-93f2-4f5b56639037","Type":"ContainerStarted","Data":"de7bf24435922de649e981498bd748993ecc6fc96325b9d6b4ce285a9b402634"} Oct 30 16:45:16.630933 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:16.630908 4493 scope.go:122] "RemoveContainer" containerID="7a92ba61e0477adb319151b23c0482d71d25ca387fe76cdce58ce2b54f734a73" Oct 30 16:45:16.647094 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:16.647072 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889645-kstp4"] Oct 30 16:45:16.647590 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:16.647569 4493 scope.go:122] "RemoveContainer" containerID="28e34dc2402193875efadf268b029c199896da18b3dbb420f1d0e997ec72a2ce" Oct 30 16:45:16.661679 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:16.661658 4493 scope.go:122] "RemoveContainer" containerID="7a066210d837b4a4553e4e43d78d518597044a229f8a6a627bd9b36175d11e2c" Oct 30 16:45:16.682056 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:16.682029 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-08fcc8-29889645-fr2qh"] Oct 30 16:45:16.719501 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:16.719471 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889645-tcj9f"] Oct 30 16:45:16.753764 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:16.753732 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889645-zj2fw"] Oct 30 16:45:17.037895 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:17.037857 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-020b099b-wxsr9"] Oct 30 16:45:17.043777 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:17.043753 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-020b099b-wxsr9"] Oct 30 16:45:17.641687 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:17.641655 4493 generic.go:356] "Generic (PLEG): container finished" podID="e170303e-4547-4a45-8807-94e735e4ea91" containerID="2e3317012cbfcca83cecb29f285f13dce2ce47d7a5a55fbb5b5555dc07809244" exitCode=60 Oct 30 16:45:17.642295 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:17.641738 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889645-ldjdk" event={"ID":"e170303e-4547-4a45-8807-94e735e4ea91","Type":"ContainerDied","Data":"2e3317012cbfcca83cecb29f285f13dce2ce47d7a5a55fbb5b5555dc07809244"} Oct 30 16:45:17.642295 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:17.641840 4493 scope.go:122] "RemoveContainer" containerID="ebb11455fd4a1fb4430dc73b16b5d71e633f57c3296df952b9374bcf86e85f27" Oct 30 16:45:17.642460 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:17.642446 4493 scope.go:122] "RemoveContainer" containerID="2e3317012cbfcca83cecb29f285f13dce2ce47d7a5a55fbb5b5555dc07809244" Oct 30 16:45:17.642748 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:45:17.642722 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-18519c-29889645-ldjdk_odh-ai-gateway-infra(e170303e-4547-4a45-8807-94e735e4ea91)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889645-ldjdk" podUID="e170303e-4547-4a45-8807-94e735e4ea91" Oct 30 16:45:17.646538 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:17.646514 4493 generic.go:356] "Generic (PLEG): container finished" podID="1e8ec889-c0ea-4342-93f2-4f5b56639037" containerID="de7bf24435922de649e981498bd748993ecc6fc96325b9d6b4ce285a9b402634" exitCode=60 Oct 30 16:45:17.646649 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:17.646593 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889645-6xjd5" event={"ID":"1e8ec889-c0ea-4342-93f2-4f5b56639037","Type":"ContainerDied","Data":"de7bf24435922de649e981498bd748993ecc6fc96325b9d6b4ce285a9b402634"} Oct 30 16:45:17.647067 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:17.647045 4493 scope.go:122] "RemoveContainer" containerID="de7bf24435922de649e981498bd748993ecc6fc96325b9d6b4ce285a9b402634" Oct 30 16:45:17.647383 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:45:17.647356 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-420ff3-29889645-6xjd5_odh-ai-gateway-infra(1e8ec889-c0ea-4342-93f2-4f5b56639037)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889645-6xjd5" podUID="1e8ec889-c0ea-4342-93f2-4f5b56639037" Oct 30 16:45:17.659559 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:17.659534 4493 scope.go:122] "RemoveContainer" containerID="2c89e9bdc90fca985ce5616802ed6abd9867f8f9d91b33c024ed1b9d8d43c692" Oct 30 16:45:17.754712 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:17.754681 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889645-6xjd5"] Oct 30 16:45:17.850898 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:17.850859 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889645-ldjdk"] Oct 30 16:45:17.895915 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:17.892407 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889645-kstp4" Oct 30 16:45:17.927876 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:17.927426 4493 scope.go:122] "RemoveContainer" containerID="201722b48f2b67b88a9aadf7c8d0feccac582c2031017fd51af49eb5eb4d6790" Oct 30 16:45:17.951532 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:17.951495 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed948051-f611-435b-82b4-3a30d53d99f1-kube-api-access-8hqbh\" (UniqueName: \"kubernetes.io/projected/ed948051-f611-435b-82b4-3a30d53d99f1-kube-api-access-8hqbh\") pod \"ed948051-f611-435b-82b4-3a30d53d99f1\" (UID: \"ed948051-f611-435b-82b4-3a30d53d99f1\") " Oct 30 16:45:17.953340 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:17.953313 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed948051-f611-435b-82b4-3a30d53d99f1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed948051-f611-435b-82b4-3a30d53d99f1-maas-api-service-ca\") pod \"ed948051-f611-435b-82b4-3a30d53d99f1\" (UID: \"ed948051-f611-435b-82b4-3a30d53d99f1\") " Oct 30 16:45:17.955917 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:17.955889 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed948051-f611-435b-82b4-3a30d53d99f1-maas-api-service-ca" pod "ed948051-f611-435b-82b4-3a30d53d99f1" (UID: "ed948051-f611-435b-82b4-3a30d53d99f1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:45:17.991954 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:17.983930 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed948051-f611-435b-82b4-3a30d53d99f1-kube-api-access-8hqbh" pod "ed948051-f611-435b-82b4-3a30d53d99f1" (UID: "ed948051-f611-435b-82b4-3a30d53d99f1"). InnerVolumeSpecName "kube-api-access-8hqbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:45:18.055267 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:18.055235 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hqbh\" (UniqueName: \"kubernetes.io/projected/ed948051-f611-435b-82b4-3a30d53d99f1-kube-api-access-8hqbh\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:45:18.055267 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:18.055263 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed948051-f611-435b-82b4-3a30d53d99f1-maas-api-service-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:45:18.061530 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:18.061507 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889645-zj2fw" Oct 30 16:45:18.064231 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:18.064211 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-08fcc8-29889645-fr2qh" Oct 30 16:45:18.067986 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:18.067969 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889645-tcj9f" Oct 30 16:45:18.156595 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:18.156502 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ed89856-cce2-41c5-b13b-348bf55b3825-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ed89856-cce2-41c5-b13b-348bf55b3825-maas-api-service-ca\") pod \"0ed89856-cce2-41c5-b13b-348bf55b3825\" (UID: \"0ed89856-cce2-41c5-b13b-348bf55b3825\") " Oct 30 16:45:18.156595 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:18.156565 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ed89856-cce2-41c5-b13b-348bf55b3825-kube-api-access-p5tpc\" (UniqueName: \"kubernetes.io/projected/0ed89856-cce2-41c5-b13b-348bf55b3825-kube-api-access-p5tpc\") pod \"0ed89856-cce2-41c5-b13b-348bf55b3825\" (UID: \"0ed89856-cce2-41c5-b13b-348bf55b3825\") " Oct 30 16:45:18.156878 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:18.156666 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8430536-01ed-4fab-833a-176ac3939e9c-kube-api-access-vj9g7\" (UniqueName: \"kubernetes.io/projected/e8430536-01ed-4fab-833a-176ac3939e9c-kube-api-access-vj9g7\") pod \"e8430536-01ed-4fab-833a-176ac3939e9c\" (UID: \"e8430536-01ed-4fab-833a-176ac3939e9c\") " Oct 30 16:45:18.156878 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:18.156708 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09d7b91f-5649-41b5-921c-80b1d25ddcb9-kube-api-access-tvkgg\" (UniqueName: \"kubernetes.io/projected/09d7b91f-5649-41b5-921c-80b1d25ddcb9-kube-api-access-tvkgg\") pod \"09d7b91f-5649-41b5-921c-80b1d25ddcb9\" (UID: \"09d7b91f-5649-41b5-921c-80b1d25ddcb9\") " Oct 30 16:45:18.156878 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:18.156856 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09d7b91f-5649-41b5-921c-80b1d25ddcb9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09d7b91f-5649-41b5-921c-80b1d25ddcb9-maas-api-service-ca\") pod \"09d7b91f-5649-41b5-921c-80b1d25ddcb9\" (UID: \"09d7b91f-5649-41b5-921c-80b1d25ddcb9\") " Oct 30 16:45:18.157053 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:18.156907 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ed89856-cce2-41c5-b13b-348bf55b3825-maas-api-service-ca" pod "0ed89856-cce2-41c5-b13b-348bf55b3825" (UID: "0ed89856-cce2-41c5-b13b-348bf55b3825"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:45:18.157053 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:18.156920 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8430536-01ed-4fab-833a-176ac3939e9c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8430536-01ed-4fab-833a-176ac3939e9c-maas-api-service-ca\") pod \"e8430536-01ed-4fab-833a-176ac3939e9c\" (UID: \"e8430536-01ed-4fab-833a-176ac3939e9c\") " Oct 30 16:45:18.157363 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:18.157285 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09d7b91f-5649-41b5-921c-80b1d25ddcb9-maas-api-service-ca" pod "09d7b91f-5649-41b5-921c-80b1d25ddcb9" (UID: "09d7b91f-5649-41b5-921c-80b1d25ddcb9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:45:18.157460 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:18.157432 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8430536-01ed-4fab-833a-176ac3939e9c-maas-api-service-ca" pod "e8430536-01ed-4fab-833a-176ac3939e9c" (UID: "e8430536-01ed-4fab-833a-176ac3939e9c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:45:18.157606 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:18.157582 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09d7b91f-5649-41b5-921c-80b1d25ddcb9-maas-api-service-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:45:18.157743 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:18.157610 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8430536-01ed-4fab-833a-176ac3939e9c-maas-api-service-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:45:18.157743 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:18.157625 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ed89856-cce2-41c5-b13b-348bf55b3825-maas-api-service-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:45:18.160103 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:18.160075 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8430536-01ed-4fab-833a-176ac3939e9c-kube-api-access-vj9g7" pod "e8430536-01ed-4fab-833a-176ac3939e9c" (UID: "e8430536-01ed-4fab-833a-176ac3939e9c"). InnerVolumeSpecName "kube-api-access-vj9g7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:45:18.160640 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:18.160616 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09d7b91f-5649-41b5-921c-80b1d25ddcb9-kube-api-access-tvkgg" pod "09d7b91f-5649-41b5-921c-80b1d25ddcb9" (UID: "09d7b91f-5649-41b5-921c-80b1d25ddcb9"). InnerVolumeSpecName "kube-api-access-tvkgg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:45:18.162079 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:18.162046 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ed89856-cce2-41c5-b13b-348bf55b3825-kube-api-access-p5tpc" pod "0ed89856-cce2-41c5-b13b-348bf55b3825" (UID: "0ed89856-cce2-41c5-b13b-348bf55b3825"). InnerVolumeSpecName "kube-api-access-p5tpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:45:18.259314 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:18.259285 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5tpc\" (UniqueName: \"kubernetes.io/projected/0ed89856-cce2-41c5-b13b-348bf55b3825-kube-api-access-p5tpc\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:45:18.259314 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:18.259313 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vj9g7\" (UniqueName: \"kubernetes.io/projected/e8430536-01ed-4fab-833a-176ac3939e9c-kube-api-access-vj9g7\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:45:18.259314 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:18.259322 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvkgg\" (UniqueName: \"kubernetes.io/projected/09d7b91f-5649-41b5-921c-80b1d25ddcb9-kube-api-access-tvkgg\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:45:18.666671 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:18.666629 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889645-kstp4" Oct 30 16:45:18.666671 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:18.666643 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889645-kstp4" event={"ID":"ed948051-f611-435b-82b4-3a30d53d99f1","Type":"ContainerDied","Data":"5eb7d835224c165dbd8d05cd1da613535668ab1b5c440551db29a9d4d5c6b6c5"} Oct 30 16:45:18.667198 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:18.666705 4493 scope.go:122] "RemoveContainer" containerID="ff3328a272393895d66d7012c345e7944b35687ba6cb97630d8e33ff38b24fc1" Oct 30 16:45:18.668227 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:18.668166 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-08fcc8-29889645-fr2qh" Oct 30 16:45:18.668326 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:18.668163 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-08fcc8-29889645-fr2qh" event={"ID":"0ed89856-cce2-41c5-b13b-348bf55b3825","Type":"ContainerDied","Data":"1e43f502a0f3cf7ab3426d1a13e8ce3eab92323c637daeb51469a8751882ef48"} Oct 30 16:45:18.669549 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:18.669533 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889645-tcj9f" Oct 30 16:45:18.669641 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:18.669532 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889645-tcj9f" event={"ID":"09d7b91f-5649-41b5-921c-80b1d25ddcb9","Type":"ContainerDied","Data":"77b7a50b0d4ed783154063d19479757e78284d50e59ff10231ffdce639e5dce4"} Oct 30 16:45:18.671863 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:18.671828 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-504442-29889645-d8ph4" event={"ID":"70a7611f-e41f-4bca-ba61-24ca0d0a3464","Type":"ContainerStarted","Data":"080da269d00095fdc23465f462416f44e0560567982a0a56bd6d9504c0002789"} Oct 30 16:45:18.675936 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:18.675916 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889645-zj2fw" Oct 30 16:45:18.676031 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:18.675923 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889645-zj2fw" event={"ID":"e8430536-01ed-4fab-833a-176ac3939e9c","Type":"ContainerDied","Data":"1c4082763fabbe7ba7603022bedfe4a12bd69349f04c249a44b26eef3f91895b"} Oct 30 16:45:18.684940 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:18.684824 4493 scope.go:122] "RemoveContainer" containerID="4619cb6c6db3affdd785b74d5774fdf44fc2338212dce49d3178fc2715eac148" Oct 30 16:45:18.696146 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:18.696118 4493 scope.go:122] "RemoveContainer" containerID="385ba56ef872cf42c085e27575ba12fe51dd68417779c3ed4f4ab17ff04996f0" Oct 30 16:45:18.708276 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:18.708159 4493 scope.go:122] "RemoveContainer" containerID="53b9c799e70b3b2963a6afe3c132f037203e327d480dc5eb32efe7cef7591de4" Oct 30 16:45:19.012701 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:19.012672 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fcd7b01-59e6-43c3-a720-054782533d20" path="/var/lib/kubelet/pods/1fcd7b01-59e6-43c3-a720-054782533d20/volumes" Oct 30 16:45:19.013283 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:19.013249 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889645-zj2fw"] Oct 30 16:45:19.013504 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:19.013481 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889645-zj2fw"] Oct 30 16:45:19.025199 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:19.025169 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-08fcc8-29889645-fr2qh"] Oct 30 16:45:19.032237 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:19.032211 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-08fcc8-29889645-fr2qh"] Oct 30 16:45:19.064591 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:19.062895 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889645-kstp4"] Oct 30 16:45:19.086187 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:19.086140 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889645-kstp4"] Oct 30 16:45:19.093943 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:19.093917 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889645-ldjdk" Oct 30 16:45:19.100374 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:19.100349 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889645-6xjd5" Oct 30 16:45:19.133757 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:19.132885 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889645-tcj9f"] Oct 30 16:45:19.139300 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:19.139269 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889645-tcj9f"] Oct 30 16:45:19.235582 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:19.235487 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e8ec889-c0ea-4342-93f2-4f5b56639037-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e8ec889-c0ea-4342-93f2-4f5b56639037-maas-api-service-ca\") pod \"1e8ec889-c0ea-4342-93f2-4f5b56639037\" (UID: \"1e8ec889-c0ea-4342-93f2-4f5b56639037\") " Oct 30 16:45:19.235910 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:19.235626 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e170303e-4547-4a45-8807-94e735e4ea91-kube-api-access-n9vd7\" (UniqueName: \"kubernetes.io/projected/e170303e-4547-4a45-8807-94e735e4ea91-kube-api-access-n9vd7\") pod \"e170303e-4547-4a45-8807-94e735e4ea91\" (UID: \"e170303e-4547-4a45-8807-94e735e4ea91\") " Oct 30 16:45:19.235910 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:19.235675 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e8ec889-c0ea-4342-93f2-4f5b56639037-kube-api-access-z5wbq\" (UniqueName: \"kubernetes.io/projected/1e8ec889-c0ea-4342-93f2-4f5b56639037-kube-api-access-z5wbq\") pod \"1e8ec889-c0ea-4342-93f2-4f5b56639037\" (UID: \"1e8ec889-c0ea-4342-93f2-4f5b56639037\") " Oct 30 16:45:19.235910 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:19.235698 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e170303e-4547-4a45-8807-94e735e4ea91-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e170303e-4547-4a45-8807-94e735e4ea91-maas-api-service-ca\") pod \"e170303e-4547-4a45-8807-94e735e4ea91\" (UID: \"e170303e-4547-4a45-8807-94e735e4ea91\") " Oct 30 16:45:19.236634 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:19.236606 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e170303e-4547-4a45-8807-94e735e4ea91-maas-api-service-ca" pod "e170303e-4547-4a45-8807-94e735e4ea91" (UID: "e170303e-4547-4a45-8807-94e735e4ea91"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:45:19.237163 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:19.237141 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e8ec889-c0ea-4342-93f2-4f5b56639037-maas-api-service-ca" pod "1e8ec889-c0ea-4342-93f2-4f5b56639037" (UID: "1e8ec889-c0ea-4342-93f2-4f5b56639037"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:45:19.239677 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:19.239649 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e170303e-4547-4a45-8807-94e735e4ea91-kube-api-access-n9vd7" pod "e170303e-4547-4a45-8807-94e735e4ea91" (UID: "e170303e-4547-4a45-8807-94e735e4ea91"). InnerVolumeSpecName "kube-api-access-n9vd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:45:19.240743 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:19.240721 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e8ec889-c0ea-4342-93f2-4f5b56639037-kube-api-access-z5wbq" pod "1e8ec889-c0ea-4342-93f2-4f5b56639037" (UID: "1e8ec889-c0ea-4342-93f2-4f5b56639037"). InnerVolumeSpecName "kube-api-access-z5wbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:45:19.338014 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:19.337971 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e8ec889-c0ea-4342-93f2-4f5b56639037-maas-api-service-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:45:19.338014 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:19.338012 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9vd7\" (UniqueName: \"kubernetes.io/projected/e170303e-4547-4a45-8807-94e735e4ea91-kube-api-access-n9vd7\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:45:19.338298 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:19.338028 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5wbq\" (UniqueName: \"kubernetes.io/projected/1e8ec889-c0ea-4342-93f2-4f5b56639037-kube-api-access-z5wbq\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:45:19.338298 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:19.338043 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e170303e-4547-4a45-8807-94e735e4ea91-maas-api-service-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:45:19.693675 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:19.693643 4493 generic.go:356] "Generic (PLEG): container finished" podID="70a7611f-e41f-4bca-ba61-24ca0d0a3464" containerID="080da269d00095fdc23465f462416f44e0560567982a0a56bd6d9504c0002789" exitCode=60 Oct 30 16:45:19.694201 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:19.693715 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-504442-29889645-d8ph4" event={"ID":"70a7611f-e41f-4bca-ba61-24ca0d0a3464","Type":"ContainerDied","Data":"080da269d00095fdc23465f462416f44e0560567982a0a56bd6d9504c0002789"} Oct 30 16:45:19.694201 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:19.693834 4493 scope.go:122] "RemoveContainer" containerID="201722b48f2b67b88a9aadf7c8d0feccac582c2031017fd51af49eb5eb4d6790" Oct 30 16:45:19.694321 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:19.694226 4493 scope.go:122] "RemoveContainer" containerID="080da269d00095fdc23465f462416f44e0560567982a0a56bd6d9504c0002789" Oct 30 16:45:19.694589 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:45:19.694566 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-504442-29889645-d8ph4_odh-ai-gateway-infra(70a7611f-e41f-4bca-ba61-24ca0d0a3464)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-504442-29889645-d8ph4" podUID="70a7611f-e41f-4bca-ba61-24ca0d0a3464" Oct 30 16:45:19.695261 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:19.695240 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889645-ldjdk" event={"ID":"e170303e-4547-4a45-8807-94e735e4ea91","Type":"ContainerDied","Data":"c27eb05e30371cf63988fd14a36d458893866521cf1c6b17152ebdeea602d761"} Oct 30 16:45:19.695345 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:19.695245 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889645-ldjdk" Oct 30 16:45:19.697602 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:19.697583 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889645-6xjd5" Oct 30 16:45:19.697711 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:19.697611 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889645-6xjd5" event={"ID":"1e8ec889-c0ea-4342-93f2-4f5b56639037","Type":"ContainerDied","Data":"0b1a54fe552af5b812d0d7b5068003c8edc81db2d8b0ebe6a4d88196a61886a4"} Oct 30 16:45:19.710012 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:19.709994 4493 scope.go:122] "RemoveContainer" containerID="2e3317012cbfcca83cecb29f285f13dce2ce47d7a5a55fbb5b5555dc07809244" Oct 30 16:45:19.721327 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:19.721306 4493 scope.go:122] "RemoveContainer" containerID="de7bf24435922de649e981498bd748993ecc6fc96325b9d6b4ce285a9b402634" Oct 30 16:45:19.765087 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:19.765045 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889645-6xjd5"] Oct 30 16:45:19.769697 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:19.769672 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889645-6xjd5"] Oct 30 16:45:19.775331 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:19.775305 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-504442-29889645-d8ph4"] Oct 30 16:45:19.802093 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:19.802058 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889645-ldjdk"] Oct 30 16:45:19.808755 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:19.808719 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889645-ldjdk"] Oct 30 16:45:20.943835 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:20.942947 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09d7b91f-5649-41b5-921c-80b1d25ddcb9" path="/var/lib/kubelet/pods/09d7b91f-5649-41b5-921c-80b1d25ddcb9/volumes" Oct 30 16:45:20.943835 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:20.943404 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ed89856-cce2-41c5-b13b-348bf55b3825" path="/var/lib/kubelet/pods/0ed89856-cce2-41c5-b13b-348bf55b3825/volumes" Oct 30 16:45:20.943835 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:20.943755 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e8ec889-c0ea-4342-93f2-4f5b56639037" path="/var/lib/kubelet/pods/1e8ec889-c0ea-4342-93f2-4f5b56639037/volumes" Oct 30 16:45:20.944808 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:20.944766 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e170303e-4547-4a45-8807-94e735e4ea91" path="/var/lib/kubelet/pods/e170303e-4547-4a45-8807-94e735e4ea91/volumes" Oct 30 16:45:20.945270 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:20.945251 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8430536-01ed-4fab-833a-176ac3939e9c" path="/var/lib/kubelet/pods/e8430536-01ed-4fab-833a-176ac3939e9c/volumes" Oct 30 16:45:20.945687 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:20.945670 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed948051-f611-435b-82b4-3a30d53d99f1" path="/var/lib/kubelet/pods/ed948051-f611-435b-82b4-3a30d53d99f1/volumes" Oct 30 16:45:20.947687 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:20.947664 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-504442-29889645-d8ph4" Oct 30 16:45:21.077273 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:21.077251 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70a7611f-e41f-4bca-ba61-24ca0d0a3464-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70a7611f-e41f-4bca-ba61-24ca0d0a3464-maas-api-service-ca\") pod \"70a7611f-e41f-4bca-ba61-24ca0d0a3464\" (UID: \"70a7611f-e41f-4bca-ba61-24ca0d0a3464\") " Oct 30 16:45:21.077485 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:21.077463 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70a7611f-e41f-4bca-ba61-24ca0d0a3464-kube-api-access-c7j2k\" (UniqueName: \"kubernetes.io/projected/70a7611f-e41f-4bca-ba61-24ca0d0a3464-kube-api-access-c7j2k\") pod \"70a7611f-e41f-4bca-ba61-24ca0d0a3464\" (UID: \"70a7611f-e41f-4bca-ba61-24ca0d0a3464\") " Oct 30 16:45:21.077601 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:21.077578 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70a7611f-e41f-4bca-ba61-24ca0d0a3464-maas-api-service-ca" pod "70a7611f-e41f-4bca-ba61-24ca0d0a3464" (UID: "70a7611f-e41f-4bca-ba61-24ca0d0a3464"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:45:21.077979 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:21.077949 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70a7611f-e41f-4bca-ba61-24ca0d0a3464-maas-api-service-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:45:21.079871 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:21.079850 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70a7611f-e41f-4bca-ba61-24ca0d0a3464-kube-api-access-c7j2k" pod "70a7611f-e41f-4bca-ba61-24ca0d0a3464" (UID: "70a7611f-e41f-4bca-ba61-24ca0d0a3464"). InnerVolumeSpecName "kube-api-access-c7j2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:45:21.082732 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:21.082715 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr" Oct 30 16:45:21.179106 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:21.179070 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b860be8-5304-4364-a0b6-e06906cf5f4f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b860be8-5304-4364-a0b6-e06906cf5f4f-istiod-ca-cert\") pod \"6b860be8-5304-4364-a0b6-e06906cf5f4f\" (UID: \"6b860be8-5304-4364-a0b6-e06906cf5f4f\") " Oct 30 16:45:21.179259 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:21.179197 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b860be8-5304-4364-a0b6-e06906cf5f4f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b860be8-5304-4364-a0b6-e06906cf5f4f-workload-certs\") pod \"6b860be8-5304-4364-a0b6-e06906cf5f4f\" (UID: \"6b860be8-5304-4364-a0b6-e06906cf5f4f\") " Oct 30 16:45:21.179340 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:21.179275 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b860be8-5304-4364-a0b6-e06906cf5f4f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b860be8-5304-4364-a0b6-e06906cf5f4f-istio-data\") pod \"6b860be8-5304-4364-a0b6-e06906cf5f4f\" (UID: \"6b860be8-5304-4364-a0b6-e06906cf5f4f\") " Oct 30 16:45:21.179392 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:21.179347 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b860be8-5304-4364-a0b6-e06906cf5f4f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b860be8-5304-4364-a0b6-e06906cf5f4f-credential-socket\") pod \"6b860be8-5304-4364-a0b6-e06906cf5f4f\" (UID: \"6b860be8-5304-4364-a0b6-e06906cf5f4f\") " Oct 30 16:45:21.179453 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:21.179408 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b860be8-5304-4364-a0b6-e06906cf5f4f-istio-token\" (UniqueName: \"kubernetes.io/projected/6b860be8-5304-4364-a0b6-e06906cf5f4f-istio-token\") pod \"6b860be8-5304-4364-a0b6-e06906cf5f4f\" (UID: \"6b860be8-5304-4364-a0b6-e06906cf5f4f\") " Oct 30 16:45:21.179453 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:21.179416 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b860be8-5304-4364-a0b6-e06906cf5f4f-istiod-ca-cert" pod "6b860be8-5304-4364-a0b6-e06906cf5f4f" (UID: "6b860be8-5304-4364-a0b6-e06906cf5f4f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:45:21.179557 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:21.179453 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b860be8-5304-4364-a0b6-e06906cf5f4f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b860be8-5304-4364-a0b6-e06906cf5f4f-workload-socket\") pod \"6b860be8-5304-4364-a0b6-e06906cf5f4f\" (UID: \"6b860be8-5304-4364-a0b6-e06906cf5f4f\") " Oct 30 16:45:21.179557 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:21.179521 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b860be8-5304-4364-a0b6-e06906cf5f4f-kube-api-access-djkb7\" (UniqueName: \"kubernetes.io/projected/6b860be8-5304-4364-a0b6-e06906cf5f4f-kube-api-access-djkb7\") pod \"6b860be8-5304-4364-a0b6-e06906cf5f4f\" (UID: \"6b860be8-5304-4364-a0b6-e06906cf5f4f\") " Oct 30 16:45:21.179666 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:21.179565 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b860be8-5304-4364-a0b6-e06906cf5f4f-workload-certs" pod "6b860be8-5304-4364-a0b6-e06906cf5f4f" (UID: "6b860be8-5304-4364-a0b6-e06906cf5f4f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:45:21.179666 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:21.179615 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b860be8-5304-4364-a0b6-e06906cf5f4f-istio-data" pod "6b860be8-5304-4364-a0b6-e06906cf5f4f" (UID: "6b860be8-5304-4364-a0b6-e06906cf5f4f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:45:21.179666 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:21.179620 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6b860be8-5304-4364-a0b6-e06906cf5f4f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b860be8-5304-4364-a0b6-e06906cf5f4f-istio-podinfo\") pod \"6b860be8-5304-4364-a0b6-e06906cf5f4f\" (UID: \"6b860be8-5304-4364-a0b6-e06906cf5f4f\") " Oct 30 16:45:21.179666 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:21.179641 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b860be8-5304-4364-a0b6-e06906cf5f4f-credential-socket" pod "6b860be8-5304-4364-a0b6-e06906cf5f4f" (UID: "6b860be8-5304-4364-a0b6-e06906cf5f4f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:45:21.179892 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:21.179701 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b860be8-5304-4364-a0b6-e06906cf5f4f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b860be8-5304-4364-a0b6-e06906cf5f4f-istio-envoy\") pod \"6b860be8-5304-4364-a0b6-e06906cf5f4f\" (UID: \"6b860be8-5304-4364-a0b6-e06906cf5f4f\") " Oct 30 16:45:21.179892 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:21.179751 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b860be8-5304-4364-a0b6-e06906cf5f4f-workload-socket" pod "6b860be8-5304-4364-a0b6-e06906cf5f4f" (UID: "6b860be8-5304-4364-a0b6-e06906cf5f4f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:45:21.180226 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:21.180203 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b860be8-5304-4364-a0b6-e06906cf5f4f-istiod-ca-cert\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:45:21.180226 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:21.180225 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b860be8-5304-4364-a0b6-e06906cf5f4f-workload-certs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:45:21.180529 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:21.180233 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b860be8-5304-4364-a0b6-e06906cf5f4f-istio-data\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:45:21.180529 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:21.180241 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b860be8-5304-4364-a0b6-e06906cf5f4f-credential-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:45:21.180529 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:21.180249 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b860be8-5304-4364-a0b6-e06906cf5f4f-workload-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:45:21.180529 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:21.180258 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7j2k\" (UniqueName: \"kubernetes.io/projected/70a7611f-e41f-4bca-ba61-24ca0d0a3464-kube-api-access-c7j2k\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:45:21.182041 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:21.182021 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b860be8-5304-4364-a0b6-e06906cf5f4f-istio-podinfo" pod "6b860be8-5304-4364-a0b6-e06906cf5f4f" (UID: "6b860be8-5304-4364-a0b6-e06906cf5f4f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:45:21.182384 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:21.182353 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b860be8-5304-4364-a0b6-e06906cf5f4f-istio-token" pod "6b860be8-5304-4364-a0b6-e06906cf5f4f" (UID: "6b860be8-5304-4364-a0b6-e06906cf5f4f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:45:21.182384 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:21.182372 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b860be8-5304-4364-a0b6-e06906cf5f4f-kube-api-access-djkb7" pod "6b860be8-5304-4364-a0b6-e06906cf5f4f" (UID: "6b860be8-5304-4364-a0b6-e06906cf5f4f"). InnerVolumeSpecName "kube-api-access-djkb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:45:21.182578 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:21.182558 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b860be8-5304-4364-a0b6-e06906cf5f4f-istio-envoy" pod "6b860be8-5304-4364-a0b6-e06906cf5f4f" (UID: "6b860be8-5304-4364-a0b6-e06906cf5f4f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:45:21.281040 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:21.281004 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b860be8-5304-4364-a0b6-e06906cf5f4f-istio-token\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:45:21.281040 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:21.281034 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djkb7\" (UniqueName: \"kubernetes.io/projected/6b860be8-5304-4364-a0b6-e06906cf5f4f-kube-api-access-djkb7\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:45:21.281218 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:21.281048 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b860be8-5304-4364-a0b6-e06906cf5f4f-istio-podinfo\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:45:21.281218 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:21.281077 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b860be8-5304-4364-a0b6-e06906cf5f4f-istio-envoy\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:45:21.722673 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:21.722638 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-504442-29889645-d8ph4" Oct 30 16:45:21.722673 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:21.722657 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-504442-29889645-d8ph4" event={"ID":"70a7611f-e41f-4bca-ba61-24ca0d0a3464","Type":"ContainerDied","Data":"e46e1e6bd596430928472fea86fb69ab84d3cf07cabf629d3a94fca11758a5a8"} Oct 30 16:45:21.722947 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:21.722752 4493 scope.go:122] "RemoveContainer" containerID="080da269d00095fdc23465f462416f44e0560567982a0a56bd6d9504c0002789" Oct 30 16:45:21.724415 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:21.724391 4493 generic.go:356] "Generic (PLEG): container finished" podID="6b860be8-5304-4364-a0b6-e06906cf5f4f" containerID="8f5c470a5e709f6e2880deb6300e233b992e725dcc8a499b5ef07b7ced37b460" exitCode=0 Oct 30 16:45:21.724493 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:21.724458 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr" Oct 30 16:45:21.724544 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:21.724471 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr" event={"ID":"6b860be8-5304-4364-a0b6-e06906cf5f4f","Type":"ContainerDied","Data":"8f5c470a5e709f6e2880deb6300e233b992e725dcc8a499b5ef07b7ced37b460"} Oct 30 16:45:21.724544 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:21.724537 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr" event={"ID":"6b860be8-5304-4364-a0b6-e06906cf5f4f","Type":"ContainerDied","Data":"246d6c2c3f96a72e3322bddb3e4e241065231f4d9e5744700c0265e6f9d18a78"} Oct 30 16:45:21.739937 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:21.738941 4493 scope.go:122] "RemoveContainer" containerID="8f5c470a5e709f6e2880deb6300e233b992e725dcc8a499b5ef07b7ced37b460" Oct 30 16:45:21.752948 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:21.752930 4493 scope.go:122] "RemoveContainer" containerID="8f5c470a5e709f6e2880deb6300e233b992e725dcc8a499b5ef07b7ced37b460" Oct 30 16:45:21.753246 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:45:21.753227 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f5c470a5e709f6e2880deb6300e233b992e725dcc8a499b5ef07b7ced37b460\": container with ID starting with 8f5c470a5e709f6e2880deb6300e233b992e725dcc8a499b5ef07b7ced37b460 not found: ID does not exist" containerID="8f5c470a5e709f6e2880deb6300e233b992e725dcc8a499b5ef07b7ced37b460" Oct 30 16:45:21.753323 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:21.753259 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f5c470a5e709f6e2880deb6300e233b992e725dcc8a499b5ef07b7ced37b460"} err="failed to get container status \"8f5c470a5e709f6e2880deb6300e233b992e725dcc8a499b5ef07b7ced37b460\": rpc error: code = NotFound desc = could not find container \"8f5c470a5e709f6e2880deb6300e233b992e725dcc8a499b5ef07b7ced37b460\": container with ID starting with 8f5c470a5e709f6e2880deb6300e233b992e725dcc8a499b5ef07b7ced37b460 not found: ID does not exist" Oct 30 16:45:21.787790 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:21.787768 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr"] Oct 30 16:45:21.791140 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:21.791122 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-dxgsr"] Oct 30 16:45:21.828743 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:21.828722 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-504442-29889645-d8ph4"] Oct 30 16:45:21.835177 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:21.835157 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-504442-29889645-d8ph4"] Oct 30 16:45:22.935584 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:22.935550 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b860be8-5304-4364-a0b6-e06906cf5f4f" path="/var/lib/kubelet/pods/6b860be8-5304-4364-a0b6-e06906cf5f4f/volumes" Oct 30 16:45:22.936042 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:22.936026 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70a7611f-e41f-4bca-ba61-24ca0d0a3464" path="/var/lib/kubelet/pods/70a7611f-e41f-4bca-ba61-24ca0d0a3464/volumes" Oct 30 16:45:24.035768 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:24.035738 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae0011a0-2b8t5"] Oct 30 16:45:24.045707 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:24.045680 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae0011a0-2b8t5"] Oct 30 16:45:24.933358 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:24.933322 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e69b09a-3937-46d0-864d-929d6f53620b" path="/var/lib/kubelet/pods/1e69b09a-3937-46d0-864d-929d6f53620b/volumes" Oct 30 16:45:30.343208 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:45:30.343156 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 16:45:35.501261 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:35.501222 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 16:45:39.061198 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:39.061160 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4a4c68-54fd7c68d8-g96m2"] Oct 30 16:45:39.092092 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:39.092057 4493 cpu_manager.go:414] "containerMap: removing container" podUID="e8430536-01ed-4fab-833a-176ac3939e9c" containerName="cleanup" Oct 30 16:45:39.092403 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:39.092373 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8430536-01ed-4fab-833a-176ac3939e9c" containerName="cleanup" Oct 30 16:45:39.093021 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:39.093005 4493 cpu_manager.go:414] "containerMap: removing container" podUID="e8430536-01ed-4fab-833a-176ac3939e9c" containerName="cleanup" Oct 30 16:45:39.093142 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:39.093130 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8430536-01ed-4fab-833a-176ac3939e9c" containerName="cleanup" Oct 30 16:45:39.093233 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:39.093223 4493 cpu_manager.go:414] "containerMap: removing container" podUID="09d7b91f-5649-41b5-921c-80b1d25ddcb9" containerName="cleanup" Oct 30 16:45:39.093398 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:39.093385 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09d7b91f-5649-41b5-921c-80b1d25ddcb9" containerName="cleanup" Oct 30 16:45:39.094114 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:39.094098 4493 cpu_manager.go:414] "containerMap: removing container" podUID="70a7611f-e41f-4bca-ba61-24ca0d0a3464" containerName="cleanup" Oct 30 16:45:39.096082 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:39.096049 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70a7611f-e41f-4bca-ba61-24ca0d0a3464" containerName="cleanup" Oct 30 16:45:39.096202 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:39.096191 4493 cpu_manager.go:414] "containerMap: removing container" podUID="6b860be8-5304-4364-a0b6-e06906cf5f4f" containerName="istio-proxy" Oct 30 16:45:39.096329 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:39.096319 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b860be8-5304-4364-a0b6-e06906cf5f4f" containerName="istio-proxy" Oct 30 16:45:39.096829 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:39.096812 4493 cpu_manager.go:414] "containerMap: removing container" podUID="1e8ec889-c0ea-4342-93f2-4f5b56639037" containerName="cleanup" Oct 30 16:45:39.096936 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:39.096925 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e8ec889-c0ea-4342-93f2-4f5b56639037" containerName="cleanup" Oct 30 16:45:39.097012 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:39.097004 4493 cpu_manager.go:414] "containerMap: removing container" podUID="e170303e-4547-4a45-8807-94e735e4ea91" containerName="cleanup" Oct 30 16:45:39.097071 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:39.097062 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e170303e-4547-4a45-8807-94e735e4ea91" containerName="cleanup" Oct 30 16:45:39.097130 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:39.097122 4493 cpu_manager.go:414] "containerMap: removing container" podUID="1e8ec889-c0ea-4342-93f2-4f5b56639037" containerName="cleanup" Oct 30 16:45:39.097195 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:39.097187 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e8ec889-c0ea-4342-93f2-4f5b56639037" containerName="cleanup" Oct 30 16:45:39.097258 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:39.097250 4493 cpu_manager.go:414] "containerMap: removing container" podUID="1e8ec889-c0ea-4342-93f2-4f5b56639037" containerName="cleanup" Oct 30 16:45:39.097312 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:39.097305 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e8ec889-c0ea-4342-93f2-4f5b56639037" containerName="cleanup" Oct 30 16:45:39.097395 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:39.097385 4493 cpu_manager.go:414] "containerMap: removing container" podUID="09d7b91f-5649-41b5-921c-80b1d25ddcb9" containerName="cleanup" Oct 30 16:45:39.097512 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:39.097501 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09d7b91f-5649-41b5-921c-80b1d25ddcb9" containerName="cleanup" Oct 30 16:45:39.097581 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:39.097574 4493 cpu_manager.go:414] "containerMap: removing container" podUID="70a7611f-e41f-4bca-ba61-24ca0d0a3464" containerName="cleanup" Oct 30 16:45:39.097981 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:39.097637 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70a7611f-e41f-4bca-ba61-24ca0d0a3464" containerName="cleanup" Oct 30 16:45:39.098096 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:39.098083 4493 cpu_manager.go:414] "containerMap: removing container" podUID="70a7611f-e41f-4bca-ba61-24ca0d0a3464" containerName="cleanup" Oct 30 16:45:39.098163 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:39.098152 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70a7611f-e41f-4bca-ba61-24ca0d0a3464" containerName="cleanup" Oct 30 16:45:39.098254 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:39.098243 4493 cpu_manager.go:414] "containerMap: removing container" podUID="e170303e-4547-4a45-8807-94e735e4ea91" containerName="cleanup" Oct 30 16:45:39.098345 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:39.098335 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e170303e-4547-4a45-8807-94e735e4ea91" containerName="cleanup" Oct 30 16:45:39.098422 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:39.098413 4493 cpu_manager.go:414] "containerMap: removing container" podUID="0ed89856-cce2-41c5-b13b-348bf55b3825" containerName="cleanup" Oct 30 16:45:39.098497 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:39.098488 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ed89856-cce2-41c5-b13b-348bf55b3825" containerName="cleanup" Oct 30 16:45:39.098570 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:39.098562 4493 cpu_manager.go:414] "containerMap: removing container" podUID="e8430536-01ed-4fab-833a-176ac3939e9c" containerName="cleanup" Oct 30 16:45:39.098659 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:39.098636 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8430536-01ed-4fab-833a-176ac3939e9c" containerName="cleanup" Oct 30 16:45:39.098760 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:39.098750 4493 cpu_manager.go:414] "containerMap: removing container" podUID="0ed89856-cce2-41c5-b13b-348bf55b3825" containerName="cleanup" Oct 30 16:45:39.098849 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:39.098839 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ed89856-cce2-41c5-b13b-348bf55b3825" containerName="cleanup" Oct 30 16:45:39.101982 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:39.101964 4493 cpu_manager.go:414] "containerMap: removing container" podUID="ed948051-f611-435b-82b4-3a30d53d99f1" containerName="cleanup" Oct 30 16:45:39.102102 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:39.102089 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed948051-f611-435b-82b4-3a30d53d99f1" containerName="cleanup" Oct 30 16:45:39.102204 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:39.102195 4493 cpu_manager.go:414] "containerMap: removing container" podUID="ed948051-f611-435b-82b4-3a30d53d99f1" containerName="cleanup" Oct 30 16:45:39.102275 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:39.102265 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed948051-f611-435b-82b4-3a30d53d99f1" containerName="cleanup" Oct 30 16:45:39.102350 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:39.102341 4493 cpu_manager.go:414] "containerMap: removing container" podUID="ed948051-f611-435b-82b4-3a30d53d99f1" containerName="cleanup" Oct 30 16:45:39.102415 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:39.102406 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed948051-f611-435b-82b4-3a30d53d99f1" containerName="cleanup" Oct 30 16:45:39.102674 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:39.102662 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8430536-01ed-4fab-833a-176ac3939e9c" containerName="cleanup" Oct 30 16:45:39.102753 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:39.102744 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8430536-01ed-4fab-833a-176ac3939e9c" containerName="cleanup" Oct 30 16:45:39.102861 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:39.102850 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="09d7b91f-5649-41b5-921c-80b1d25ddcb9" containerName="cleanup" Oct 30 16:45:39.102938 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:39.102929 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="70a7611f-e41f-4bca-ba61-24ca0d0a3464" containerName="cleanup" Oct 30 16:45:39.103068 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:39.103057 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b860be8-5304-4364-a0b6-e06906cf5f4f" containerName="istio-proxy" Oct 30 16:45:39.103144 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:39.103135 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e8ec889-c0ea-4342-93f2-4f5b56639037" containerName="cleanup" Oct 30 16:45:39.103222 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:39.103213 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="e170303e-4547-4a45-8807-94e735e4ea91" containerName="cleanup" Oct 30 16:45:39.103292 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:39.103284 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e8ec889-c0ea-4342-93f2-4f5b56639037" containerName="cleanup" Oct 30 16:45:39.103366 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:39.103357 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e8ec889-c0ea-4342-93f2-4f5b56639037" containerName="cleanup" Oct 30 16:45:39.103439 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:39.103430 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="09d7b91f-5649-41b5-921c-80b1d25ddcb9" containerName="cleanup" Oct 30 16:45:39.103512 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:39.103503 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="70a7611f-e41f-4bca-ba61-24ca0d0a3464" containerName="cleanup" Oct 30 16:45:39.103590 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:39.103581 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="09d7b91f-5649-41b5-921c-80b1d25ddcb9" containerName="cleanup" Oct 30 16:45:39.103662 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:39.103653 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="70a7611f-e41f-4bca-ba61-24ca0d0a3464" containerName="cleanup" Oct 30 16:45:39.103732 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:39.103722 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="e170303e-4547-4a45-8807-94e735e4ea91" containerName="cleanup" Oct 30 16:45:39.103825 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:39.103815 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ed89856-cce2-41c5-b13b-348bf55b3825" containerName="cleanup" Oct 30 16:45:39.103938 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:39.103928 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ed89856-cce2-41c5-b13b-348bf55b3825" containerName="cleanup" Oct 30 16:45:39.104021 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:39.104012 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ed89856-cce2-41c5-b13b-348bf55b3825" containerName="cleanup" Oct 30 16:45:39.104087 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:39.104079 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed948051-f611-435b-82b4-3a30d53d99f1" containerName="cleanup" Oct 30 16:45:39.104162 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:39.104153 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed948051-f611-435b-82b4-3a30d53d99f1" containerName="cleanup" Oct 30 16:45:39.104232 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:39.104222 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="e170303e-4547-4a45-8807-94e735e4ea91" containerName="cleanup" Oct 30 16:45:39.104299 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:39.104290 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed948051-f611-435b-82b4-3a30d53d99f1" containerName="cleanup" Oct 30 16:45:39.118297 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:39.118212 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4a4c68-54fd7c68d8-g96m2"] Oct 30 16:45:39.118428 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:39.118329 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4a4c68-54fd7c68d8-g96m2" Oct 30 16:45:39.145749 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:39.145713 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-4a4c68-dockercfg-gq6cr\"" Oct 30 16:45:39.145896 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:39.145770 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-4a4c68\"" Oct 30 16:45:39.180730 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:39.180697 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dstxk\" (UniqueName: \"kubernetes.io/projected/8e2b6cc4-8cfe-4cf3-b221-fc948169dd31-kube-api-access-dstxk\") pod \"payload-processing-e2e-ipp-b-4a4c68-54fd7c68d8-g96m2\" (UID: \"8e2b6cc4-8cfe-4cf3-b221-fc948169dd31\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4a4c68-54fd7c68d8-g96m2" Oct 30 16:45:39.180922 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:39.180761 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e2b6cc4-8cfe-4cf3-b221-fc948169dd31-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4a4c68-54fd7c68d8-g96m2\" (UID: \"8e2b6cc4-8cfe-4cf3-b221-fc948169dd31\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4a4c68-54fd7c68d8-g96m2" Oct 30 16:45:39.282343 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:39.282313 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dstxk\" (UniqueName: \"kubernetes.io/projected/8e2b6cc4-8cfe-4cf3-b221-fc948169dd31-kube-api-access-dstxk\") pod \"payload-processing-e2e-ipp-b-4a4c68-54fd7c68d8-g96m2\" (UID: \"8e2b6cc4-8cfe-4cf3-b221-fc948169dd31\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4a4c68-54fd7c68d8-g96m2" Oct 30 16:45:39.282495 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:39.282361 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e2b6cc4-8cfe-4cf3-b221-fc948169dd31-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4a4c68-54fd7c68d8-g96m2\" (UID: \"8e2b6cc4-8cfe-4cf3-b221-fc948169dd31\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4a4c68-54fd7c68d8-g96m2" Oct 30 16:45:39.283046 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:39.283027 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e2b6cc4-8cfe-4cf3-b221-fc948169dd31-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-4a4c68-54fd7c68d8-g96m2\" (UID: \"8e2b6cc4-8cfe-4cf3-b221-fc948169dd31\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4a4c68-54fd7c68d8-g96m2" Oct 30 16:45:39.311907 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:39.311827 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dstxk\" (UniqueName: \"kubernetes.io/projected/8e2b6cc4-8cfe-4cf3-b221-fc948169dd31-kube-api-access-dstxk\") pod \"payload-processing-e2e-ipp-b-4a4c68-54fd7c68d8-g96m2\" (UID: \"8e2b6cc4-8cfe-4cf3-b221-fc948169dd31\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-4a4c68-54fd7c68d8-g96m2" Oct 30 16:45:39.435510 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:39.435481 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4a4c68-54fd7c68d8-g96m2" Oct 30 16:45:39.747575 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:39.747545 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4a4c68-54fd7c68d8-g96m2"] Oct 30 16:45:39.748539 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:45:39.748502 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e2b6cc4_8cfe_4cf3_b221_fc948169dd31.slice/crio-3f5914883a1939325821f0a189d61a11b7dfb3d10432b921d4f5cad49dbc008c WatchSource:0}: Error finding container 3f5914883a1939325821f0a189d61a11b7dfb3d10432b921d4f5cad49dbc008c: Status 404 returned error can't find the container with id 3f5914883a1939325821f0a189d61a11b7dfb3d10432b921d4f5cad49dbc008c Oct 30 16:45:40.105501 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:40.105325 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4a4c68-54fd7c68d8-g96m2" event={"ID":"8e2b6cc4-8cfe-4cf3-b221-fc948169dd31","Type":"ContainerStarted","Data":"2df3dae036ae03f8acf759fd3f1b32af478529f9adbb74111dad50ac9373260b"} Oct 30 16:45:40.105501 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:40.105397 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-4a4c68-54fd7c68d8-g96m2" Oct 30 16:45:40.105501 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:40.105420 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4a4c68-54fd7c68d8-g96m2" event={"ID":"8e2b6cc4-8cfe-4cf3-b221-fc948169dd31","Type":"ContainerStarted","Data":"3f5914883a1939325821f0a189d61a11b7dfb3d10432b921d4f5cad49dbc008c"} Oct 30 16:45:51.132199 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:51.132169 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-4a4c68-54fd7c68d8-g96m2" Oct 30 16:45:51.172492 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:51.172446 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-4a4c68-54fd7c68d8-g96m2" podStartSLOduration=13.172432544 podStartE2EDuration="13.172432544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:45:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:45:40.160626674 +0000 UTC m=+1367.387801774" watchObservedRunningTime="2026-10-30 16:45:51.172432544 +0000 UTC m=+1378.399607646" Oct 30 16:45:54.126842 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:54.126759 4493 scope.go:122] "RemoveContainer" containerID="40442b7bedd4ae44381042fbd2abb87d8c2b211e2ba7dca3fa8a7aa843e6b5d5" Oct 30 16:45:57.427811 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:57.427015 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 16:45:57.666327 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:57.666287 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0a98d1-n6gsh"] Oct 30 16:45:57.667185 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:57.667164 4493 cpu_manager.go:414] "containerMap: removing container" podUID="09d7b91f-5649-41b5-921c-80b1d25ddcb9" containerName="cleanup" Oct 30 16:45:57.667237 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:57.667191 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09d7b91f-5649-41b5-921c-80b1d25ddcb9" containerName="cleanup" Oct 30 16:45:57.667237 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:57.667218 4493 cpu_manager.go:414] "containerMap: removing container" podUID="e170303e-4547-4a45-8807-94e735e4ea91" containerName="cleanup" Oct 30 16:45:57.667237 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:57.667227 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e170303e-4547-4a45-8807-94e735e4ea91" containerName="cleanup" Oct 30 16:45:57.667334 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:57.667257 4493 cpu_manager.go:414] "containerMap: removing container" podUID="0ed89856-cce2-41c5-b13b-348bf55b3825" containerName="cleanup" Oct 30 16:45:57.667334 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:57.667266 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ed89856-cce2-41c5-b13b-348bf55b3825" containerName="cleanup" Oct 30 16:45:57.667488 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:57.667473 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8430536-01ed-4fab-833a-176ac3939e9c" containerName="cleanup" Oct 30 16:45:57.673707 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:57.673686 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0a98d1-n6gsh" Oct 30 16:45:57.691611 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:57.691581 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0a98d1-n6gsh"] Oct 30 16:45:57.696166 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:57.696140 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:45:57.696166 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:57.696155 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:45:57.696370 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:57.696219 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:45:57.696531 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:57.696517 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 16:45:57.859723 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:57.859673 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab7a55e2-443d-4079-b3e5-10846b0ef979-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0a98d1-n6gsh\" (UID: \"ab7a55e2-443d-4079-b3e5-10846b0ef979\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0a98d1-n6gsh" Oct 30 16:45:57.859967 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:57.859833 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab7a55e2-443d-4079-b3e5-10846b0ef979-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0a98d1-n6gsh\" (UID: \"ab7a55e2-443d-4079-b3e5-10846b0ef979\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0a98d1-n6gsh" Oct 30 16:45:57.860083 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:57.860050 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkkrh\" (UniqueName: \"kubernetes.io/projected/ab7a55e2-443d-4079-b3e5-10846b0ef979-kube-api-access-rkkrh\") pod \"maas-api-revoke-keys-e2e-ipp-a-0a98d1-n6gsh\" (UID: \"ab7a55e2-443d-4079-b3e5-10846b0ef979\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0a98d1-n6gsh" Oct 30 16:45:57.961530 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:57.961444 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkkrh\" (UniqueName: \"kubernetes.io/projected/ab7a55e2-443d-4079-b3e5-10846b0ef979-kube-api-access-rkkrh\") pod \"maas-api-revoke-keys-e2e-ipp-a-0a98d1-n6gsh\" (UID: \"ab7a55e2-443d-4079-b3e5-10846b0ef979\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0a98d1-n6gsh" Oct 30 16:45:57.961708 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:57.961562 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab7a55e2-443d-4079-b3e5-10846b0ef979-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0a98d1-n6gsh\" (UID: \"ab7a55e2-443d-4079-b3e5-10846b0ef979\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0a98d1-n6gsh" Oct 30 16:45:57.961708 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:57.961600 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab7a55e2-443d-4079-b3e5-10846b0ef979-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0a98d1-n6gsh\" (UID: \"ab7a55e2-443d-4079-b3e5-10846b0ef979\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0a98d1-n6gsh" Oct 30 16:45:57.962331 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:57.962306 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab7a55e2-443d-4079-b3e5-10846b0ef979-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0a98d1-n6gsh\" (UID: \"ab7a55e2-443d-4079-b3e5-10846b0ef979\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0a98d1-n6gsh" Oct 30 16:45:57.971232 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:57.971198 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab7a55e2-443d-4079-b3e5-10846b0ef979-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0a98d1-n6gsh\" (UID: \"ab7a55e2-443d-4079-b3e5-10846b0ef979\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0a98d1-n6gsh" Oct 30 16:45:57.984127 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:57.984090 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkkrh\" (UniqueName: \"kubernetes.io/projected/ab7a55e2-443d-4079-b3e5-10846b0ef979-kube-api-access-rkkrh\") pod \"maas-api-revoke-keys-e2e-ipp-a-0a98d1-n6gsh\" (UID: \"ab7a55e2-443d-4079-b3e5-10846b0ef979\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0a98d1-n6gsh" Oct 30 16:45:58.001811 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:57.998371 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0a98d1-n6gsh" Oct 30 16:45:58.425749 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:58.425723 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0a98d1-n6gsh"] Oct 30 16:45:58.428830 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:45:58.428800 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab7a55e2_443d_4079_b3e5_10846b0ef979.slice/crio-376d929f34546245e1edd709f0488bedb16a2d0f86bb2fc7ef9a1c957055f482 WatchSource:0}: Error finding container 376d929f34546245e1edd709f0488bedb16a2d0f86bb2fc7ef9a1c957055f482: Status 404 returned error can't find the container with id 376d929f34546245e1edd709f0488bedb16a2d0f86bb2fc7ef9a1c957055f482 Oct 30 16:45:59.296880 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:59.296837 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0a98d1-n6gsh" event={"ID":"ab7a55e2-443d-4079-b3e5-10846b0ef979","Type":"ContainerStarted","Data":"3cc97ce2593e0967559d03442e739ec10bcd362d7f42ca7f23606aa70c0e375e"} Oct 30 16:45:59.297066 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:59.296908 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0a98d1-n6gsh" event={"ID":"ab7a55e2-443d-4079-b3e5-10846b0ef979","Type":"ContainerStarted","Data":"376d929f34546245e1edd709f0488bedb16a2d0f86bb2fc7ef9a1c957055f482"} Oct 30 16:45:59.333883 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:45:59.333830 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0a98d1-n6gsh" podStartSLOduration=2.333817 podStartE2EDuration="2.333817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:45:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:45:59.333175269 +0000 UTC m=+1386.560350369" watchObservedRunningTime="2026-10-30 16:45:59.333817 +0000 UTC m=+1386.560992100" Oct 30 16:46:00.306326 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:00.306288 4493 generic.go:356] "Generic (PLEG): container finished" podID="ab7a55e2-443d-4079-b3e5-10846b0ef979" containerID="3cc97ce2593e0967559d03442e739ec10bcd362d7f42ca7f23606aa70c0e375e" exitCode=0 Oct 30 16:46:00.306326 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:00.306331 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0a98d1-n6gsh" event={"ID":"ab7a55e2-443d-4079-b3e5-10846b0ef979","Type":"ContainerDied","Data":"3cc97ce2593e0967559d03442e739ec10bcd362d7f42ca7f23606aa70c0e375e"} Oct 30 16:46:00.345702 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:46:00.345671 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 16:46:01.582896 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:01.582824 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0a98d1-n6gsh" Oct 30 16:46:01.679870 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:01.679833 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab7a55e2-443d-4079-b3e5-10846b0ef979-kube-api-access-rkkrh\" (UniqueName: \"kubernetes.io/projected/ab7a55e2-443d-4079-b3e5-10846b0ef979-kube-api-access-rkkrh\") pod \"ab7a55e2-443d-4079-b3e5-10846b0ef979\" (UID: \"ab7a55e2-443d-4079-b3e5-10846b0ef979\") " Oct 30 16:46:01.680044 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:01.679950 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab7a55e2-443d-4079-b3e5-10846b0ef979-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab7a55e2-443d-4079-b3e5-10846b0ef979-maas-api-xks-ca\") pod \"ab7a55e2-443d-4079-b3e5-10846b0ef979\" (UID: \"ab7a55e2-443d-4079-b3e5-10846b0ef979\") " Oct 30 16:46:01.680044 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:01.680036 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab7a55e2-443d-4079-b3e5-10846b0ef979-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab7a55e2-443d-4079-b3e5-10846b0ef979-maas-api-openshift-ca\") pod \"ab7a55e2-443d-4079-b3e5-10846b0ef979\" (UID: \"ab7a55e2-443d-4079-b3e5-10846b0ef979\") " Oct 30 16:46:01.680374 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:01.680348 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab7a55e2-443d-4079-b3e5-10846b0ef979-maas-api-openshift-ca" pod "ab7a55e2-443d-4079-b3e5-10846b0ef979" (UID: "ab7a55e2-443d-4079-b3e5-10846b0ef979"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:46:01.680505 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:01.680490 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab7a55e2-443d-4079-b3e5-10846b0ef979-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:46:01.682063 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:01.682037 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab7a55e2-443d-4079-b3e5-10846b0ef979-maas-api-xks-ca" pod "ab7a55e2-443d-4079-b3e5-10846b0ef979" (UID: "ab7a55e2-443d-4079-b3e5-10846b0ef979"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:46:01.682164 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:01.682091 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab7a55e2-443d-4079-b3e5-10846b0ef979-kube-api-access-rkkrh" pod "ab7a55e2-443d-4079-b3e5-10846b0ef979" (UID: "ab7a55e2-443d-4079-b3e5-10846b0ef979"). InnerVolumeSpecName "kube-api-access-rkkrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:46:01.781396 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:01.781367 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab7a55e2-443d-4079-b3e5-10846b0ef979-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:46:01.781396 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:01.781397 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkkrh\" (UniqueName: \"kubernetes.io/projected/ab7a55e2-443d-4079-b3e5-10846b0ef979-kube-api-access-rkkrh\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:46:02.323431 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:02.323399 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0a98d1-n6gsh" Oct 30 16:46:02.323600 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:02.323440 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0a98d1-n6gsh" event={"ID":"ab7a55e2-443d-4079-b3e5-10846b0ef979","Type":"ContainerDied","Data":"376d929f34546245e1edd709f0488bedb16a2d0f86bb2fc7ef9a1c957055f482"} Oct 30 16:46:02.323600 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:02.323489 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="376d929f34546245e1edd709f0488bedb16a2d0f86bb2fc7ef9a1c957055f482" Oct 30 16:46:16.642631 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:16.642592 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-08fcc8-g6jb8"] Oct 30 16:46:16.643528 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:16.643506 4493 cpu_manager.go:414] "containerMap: removing container" podUID="ab7a55e2-443d-4079-b3e5-10846b0ef979" containerName="revoke-keys" Oct 30 16:46:16.643572 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:16.643533 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab7a55e2-443d-4079-b3e5-10846b0ef979" containerName="revoke-keys" Oct 30 16:46:16.643715 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:16.643703 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab7a55e2-443d-4079-b3e5-10846b0ef979" containerName="revoke-keys" Oct 30 16:46:16.648019 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:16.648000 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-08fcc8-g6jb8" Oct 30 16:46:16.662711 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:16.662686 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:46:16.662931 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:16.662772 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:46:16.673175 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:16.673139 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:46:16.673356 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:16.673191 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 16:46:16.677811 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:16.677770 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-08fcc8-g6jb8"] Oct 30 16:46:16.803043 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:16.803011 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f363aff-2fc7-487f-9fd0-47fdd06276a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-08fcc8-g6jb8\" (UID: \"8f363aff-2fc7-487f-9fd0-47fdd06276a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-08fcc8-g6jb8" Oct 30 16:46:16.803187 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:16.803171 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6584\" (UniqueName: \"kubernetes.io/projected/8f363aff-2fc7-487f-9fd0-47fdd06276a3-kube-api-access-s6584\") pod \"maas-api-revoke-keys-e2e-api-a-08fcc8-g6jb8\" (UID: \"8f363aff-2fc7-487f-9fd0-47fdd06276a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-08fcc8-g6jb8" Oct 30 16:46:16.803259 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:16.803218 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f363aff-2fc7-487f-9fd0-47fdd06276a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-08fcc8-g6jb8\" (UID: \"8f363aff-2fc7-487f-9fd0-47fdd06276a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-08fcc8-g6jb8" Oct 30 16:46:16.904988 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:16.904891 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6584\" (UniqueName: \"kubernetes.io/projected/8f363aff-2fc7-487f-9fd0-47fdd06276a3-kube-api-access-s6584\") pod \"maas-api-revoke-keys-e2e-api-a-08fcc8-g6jb8\" (UID: \"8f363aff-2fc7-487f-9fd0-47fdd06276a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-08fcc8-g6jb8" Oct 30 16:46:16.904988 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:16.904960 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f363aff-2fc7-487f-9fd0-47fdd06276a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-08fcc8-g6jb8\" (UID: \"8f363aff-2fc7-487f-9fd0-47fdd06276a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-08fcc8-g6jb8" Oct 30 16:46:16.905243 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:16.905137 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f363aff-2fc7-487f-9fd0-47fdd06276a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-08fcc8-g6jb8\" (UID: \"8f363aff-2fc7-487f-9fd0-47fdd06276a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-08fcc8-g6jb8" Oct 30 16:46:16.905831 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:16.905773 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f363aff-2fc7-487f-9fd0-47fdd06276a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-08fcc8-g6jb8\" (UID: \"8f363aff-2fc7-487f-9fd0-47fdd06276a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-08fcc8-g6jb8" Oct 30 16:46:16.907533 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:16.907517 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f363aff-2fc7-487f-9fd0-47fdd06276a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-08fcc8-g6jb8\" (UID: \"8f363aff-2fc7-487f-9fd0-47fdd06276a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-08fcc8-g6jb8" Oct 30 16:46:16.926327 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:16.926303 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6584\" (UniqueName: \"kubernetes.io/projected/8f363aff-2fc7-487f-9fd0-47fdd06276a3-kube-api-access-s6584\") pod \"maas-api-revoke-keys-e2e-api-a-08fcc8-g6jb8\" (UID: \"8f363aff-2fc7-487f-9fd0-47fdd06276a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-08fcc8-g6jb8" Oct 30 16:46:16.965638 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:16.965610 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-08fcc8-g6jb8" Oct 30 16:46:17.268622 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:17.268544 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-08fcc8-g6jb8"] Oct 30 16:46:17.271400 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:46:17.271366 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f363aff_2fc7_487f_9fd0_47fdd06276a3.slice/crio-386c5096d4d780a512497fc1f347e1eaa77e2f48d35c7391b082ab731a3546d2 WatchSource:0}: Error finding container 386c5096d4d780a512497fc1f347e1eaa77e2f48d35c7391b082ab731a3546d2: Status 404 returned error can't find the container with id 386c5096d4d780a512497fc1f347e1eaa77e2f48d35c7391b082ab731a3546d2 Oct 30 16:46:17.450620 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:17.450584 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-08fcc8-g6jb8" event={"ID":"8f363aff-2fc7-487f-9fd0-47fdd06276a3","Type":"ContainerStarted","Data":"386c5096d4d780a512497fc1f347e1eaa77e2f48d35c7391b082ab731a3546d2"} Oct 30 16:46:18.459288 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:18.459252 4493 generic.go:356] "Generic (PLEG): container finished" podID="8f363aff-2fc7-487f-9fd0-47fdd06276a3" containerID="60f8d1c80965dc9cd5f4e9f056f5dd2991d09bac5991e5d29fc256597cae4fe4" exitCode=0 Oct 30 16:46:18.459754 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:18.459336 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-08fcc8-g6jb8" event={"ID":"8f363aff-2fc7-487f-9fd0-47fdd06276a3","Type":"ContainerDied","Data":"60f8d1c80965dc9cd5f4e9f056f5dd2991d09bac5991e5d29fc256597cae4fe4"} Oct 30 16:46:19.611422 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:19.611400 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-08fcc8-g6jb8" Oct 30 16:46:19.739185 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:19.739112 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f363aff-2fc7-487f-9fd0-47fdd06276a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f363aff-2fc7-487f-9fd0-47fdd06276a3-maas-api-openshift-ca\") pod \"8f363aff-2fc7-487f-9fd0-47fdd06276a3\" (UID: \"8f363aff-2fc7-487f-9fd0-47fdd06276a3\") " Oct 30 16:46:19.739354 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:19.739197 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f363aff-2fc7-487f-9fd0-47fdd06276a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f363aff-2fc7-487f-9fd0-47fdd06276a3-maas-api-xks-ca\") pod \"8f363aff-2fc7-487f-9fd0-47fdd06276a3\" (UID: \"8f363aff-2fc7-487f-9fd0-47fdd06276a3\") " Oct 30 16:46:19.739423 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:19.739373 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f363aff-2fc7-487f-9fd0-47fdd06276a3-kube-api-access-s6584\" (UniqueName: \"kubernetes.io/projected/8f363aff-2fc7-487f-9fd0-47fdd06276a3-kube-api-access-s6584\") pod \"8f363aff-2fc7-487f-9fd0-47fdd06276a3\" (UID: \"8f363aff-2fc7-487f-9fd0-47fdd06276a3\") " Oct 30 16:46:19.739544 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:19.739520 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f363aff-2fc7-487f-9fd0-47fdd06276a3-maas-api-openshift-ca" pod "8f363aff-2fc7-487f-9fd0-47fdd06276a3" (UID: "8f363aff-2fc7-487f-9fd0-47fdd06276a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:46:19.739817 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:19.739798 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f363aff-2fc7-487f-9fd0-47fdd06276a3-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:46:19.741398 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:19.741375 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f363aff-2fc7-487f-9fd0-47fdd06276a3-maas-api-xks-ca" pod "8f363aff-2fc7-487f-9fd0-47fdd06276a3" (UID: "8f363aff-2fc7-487f-9fd0-47fdd06276a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:46:19.741552 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:19.741533 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f363aff-2fc7-487f-9fd0-47fdd06276a3-kube-api-access-s6584" pod "8f363aff-2fc7-487f-9fd0-47fdd06276a3" (UID: "8f363aff-2fc7-487f-9fd0-47fdd06276a3"). InnerVolumeSpecName "kube-api-access-s6584". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:46:19.841263 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:19.841227 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6584\" (UniqueName: \"kubernetes.io/projected/8f363aff-2fc7-487f-9fd0-47fdd06276a3-kube-api-access-s6584\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:46:19.841263 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:19.841258 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f363aff-2fc7-487f-9fd0-47fdd06276a3-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:46:20.475584 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:20.475558 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-08fcc8-g6jb8" Oct 30 16:46:20.475765 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:20.475593 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-08fcc8-g6jb8" event={"ID":"8f363aff-2fc7-487f-9fd0-47fdd06276a3","Type":"ContainerDied","Data":"386c5096d4d780a512497fc1f347e1eaa77e2f48d35c7391b082ab731a3546d2"} Oct 30 16:46:20.475765 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:20.475643 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="386c5096d4d780a512497fc1f347e1eaa77e2f48d35c7391b082ab731a3546d2" Oct 30 16:46:26.110971 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:26.110933 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 16:46:30.489824 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:46:30.487326 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 16:46:31.030817 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:31.030736 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0a98d1-6b5d79cbb4-crqf4"] Oct 30 16:46:31.031084 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:31.031032 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-0a98d1-6b5d79cbb4-crqf4" podUID="dc863628-c06a-4415-961d-0179b11b3a63" containerName="payload-processing" containerID="cri-o://73e35e9b5cb1794693cb1a2fc2a90fc2ade0b9774029dc3e18bc15eaaf026069" gracePeriod=30 Oct 30 16:46:31.372383 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:31.372360 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0a98d1-6b5d79cbb4-crqf4" Oct 30 16:46:31.475811 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:31.462567 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc863628-c06a-4415-961d-0179b11b3a63-kube-api-access-28z77\" (UniqueName: \"kubernetes.io/projected/dc863628-c06a-4415-961d-0179b11b3a63-kube-api-access-28z77\") pod \"dc863628-c06a-4415-961d-0179b11b3a63\" (UID: \"dc863628-c06a-4415-961d-0179b11b3a63\") " Oct 30 16:46:31.498428 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:31.498398 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc863628-c06a-4415-961d-0179b11b3a63-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc863628-c06a-4415-961d-0179b11b3a63-plugins-config-volume\") pod \"dc863628-c06a-4415-961d-0179b11b3a63\" (UID: \"dc863628-c06a-4415-961d-0179b11b3a63\") " Oct 30 16:46:31.499930 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:31.499273 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc863628-c06a-4415-961d-0179b11b3a63-plugins-config-volume" pod "dc863628-c06a-4415-961d-0179b11b3a63" (UID: "dc863628-c06a-4415-961d-0179b11b3a63"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:46:31.558949 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:31.558907 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc863628-c06a-4415-961d-0179b11b3a63-kube-api-access-28z77" pod "dc863628-c06a-4415-961d-0179b11b3a63" (UID: "dc863628-c06a-4415-961d-0179b11b3a63"). InnerVolumeSpecName "kube-api-access-28z77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:46:31.574547 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:31.574510 4493 generic.go:356] "Generic (PLEG): container finished" podID="dc863628-c06a-4415-961d-0179b11b3a63" containerID="73e35e9b5cb1794693cb1a2fc2a90fc2ade0b9774029dc3e18bc15eaaf026069" exitCode=0 Oct 30 16:46:31.574967 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:31.574573 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0a98d1-6b5d79cbb4-crqf4" Oct 30 16:46:31.575111 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:31.574603 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0a98d1-6b5d79cbb4-crqf4" event={"ID":"dc863628-c06a-4415-961d-0179b11b3a63","Type":"ContainerDied","Data":"73e35e9b5cb1794693cb1a2fc2a90fc2ade0b9774029dc3e18bc15eaaf026069"} Oct 30 16:46:31.575320 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:31.575297 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0a98d1-6b5d79cbb4-crqf4" event={"ID":"dc863628-c06a-4415-961d-0179b11b3a63","Type":"ContainerDied","Data":"6043fab49c2a72e47a5893f55759dcf951c170bcfbad008a9a5518389dfb6131"} Oct 30 16:46:31.575415 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:31.575360 4493 scope.go:122] "RemoveContainer" containerID="73e35e9b5cb1794693cb1a2fc2a90fc2ade0b9774029dc3e18bc15eaaf026069" Oct 30 16:46:31.590097 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:31.590077 4493 scope.go:122] "RemoveContainer" containerID="73e35e9b5cb1794693cb1a2fc2a90fc2ade0b9774029dc3e18bc15eaaf026069" Oct 30 16:46:31.590444 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:46:31.590424 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73e35e9b5cb1794693cb1a2fc2a90fc2ade0b9774029dc3e18bc15eaaf026069\": container with ID starting with 73e35e9b5cb1794693cb1a2fc2a90fc2ade0b9774029dc3e18bc15eaaf026069 not found: ID does not exist" containerID="73e35e9b5cb1794693cb1a2fc2a90fc2ade0b9774029dc3e18bc15eaaf026069" Oct 30 16:46:31.590512 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:31.590453 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73e35e9b5cb1794693cb1a2fc2a90fc2ade0b9774029dc3e18bc15eaaf026069"} err="failed to get container status \"73e35e9b5cb1794693cb1a2fc2a90fc2ade0b9774029dc3e18bc15eaaf026069\": rpc error: code = NotFound desc = could not find container \"73e35e9b5cb1794693cb1a2fc2a90fc2ade0b9774029dc3e18bc15eaaf026069\": container with ID starting with 73e35e9b5cb1794693cb1a2fc2a90fc2ade0b9774029dc3e18bc15eaaf026069 not found: ID does not exist" Oct 30 16:46:31.601509 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:31.601488 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28z77\" (UniqueName: \"kubernetes.io/projected/dc863628-c06a-4415-961d-0179b11b3a63-kube-api-access-28z77\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:46:31.601611 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:31.601511 4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc863628-c06a-4415-961d-0179b11b3a63-plugins-config-volume\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:46:31.634125 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:31.634094 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0a98d1-6b5d79cbb4-crqf4"] Oct 30 16:46:31.638669 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:31.638640 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0a98d1-6b5d79cbb4-crqf4"] Oct 30 16:46:31.846026 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:31.845934 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0a98d1-n6gsh"] Oct 30 16:46:31.851370 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:31.851340 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0a98d1-n6gsh"] Oct 30 16:46:32.937020 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:32.936986 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab7a55e2-443d-4079-b3e5-10846b0ef979" path="/var/lib/kubelet/pods/ab7a55e2-443d-4079-b3e5-10846b0ef979/volumes" Oct 30 16:46:32.937436 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:32.937422 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc863628-c06a-4415-961d-0179b11b3a63" path="/var/lib/kubelet/pods/dc863628-c06a-4415-961d-0179b11b3a63/volumes" Oct 30 16:46:38.000936 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:38.000898 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-64c409-54c8f8f675-cmds8"] Oct 30 16:46:38.002802 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:38.002759 4493 cpu_manager.go:414] "containerMap: removing container" podUID="dc863628-c06a-4415-961d-0179b11b3a63" containerName="payload-processing" Oct 30 16:46:38.002995 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:38.002978 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc863628-c06a-4415-961d-0179b11b3a63" containerName="payload-processing" Oct 30 16:46:38.003139 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:38.003128 4493 cpu_manager.go:414] "containerMap: removing container" podUID="8f363aff-2fc7-487f-9fd0-47fdd06276a3" containerName="revoke-keys" Oct 30 16:46:38.003219 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:38.003209 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f363aff-2fc7-487f-9fd0-47fdd06276a3" containerName="revoke-keys" Oct 30 16:46:38.009162 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:38.009141 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc863628-c06a-4415-961d-0179b11b3a63" containerName="payload-processing" Oct 30 16:46:38.009319 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:38.009307 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f363aff-2fc7-487f-9fd0-47fdd06276a3" containerName="revoke-keys" Oct 30 16:46:38.034675 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:38.034644 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-64c409-54c8f8f675-cmds8" Oct 30 16:46:38.055374 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:38.055282 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-64c409\"" Oct 30 16:46:38.055523 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:38.052591 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-64c409-54c8f8f675-cmds8"] Oct 30 16:46:38.072806 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:38.071088 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-64c409-dockercfg-dkqvb\"" Oct 30 16:46:38.092402 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:38.092351 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6cd9cc1-7206-4868-908a-05aed37aae54-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-64c409-54c8f8f675-cmds8\" (UID: \"e6cd9cc1-7206-4868-908a-05aed37aae54\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-64c409-54c8f8f675-cmds8" Oct 30 16:46:38.092960 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:38.092919 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl58d\" (UniqueName: \"kubernetes.io/projected/e6cd9cc1-7206-4868-908a-05aed37aae54-kube-api-access-pl58d\") pod \"payload-pre-processing-e2e-ipp-b-64c409-54c8f8f675-cmds8\" (UID: \"e6cd9cc1-7206-4868-908a-05aed37aae54\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-64c409-54c8f8f675-cmds8" Oct 30 16:46:38.155417 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:38.154571 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-64c409-75bfc9f544-l6lpj"] Oct 30 16:46:38.162671 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:38.162649 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-64c409-75bfc9f544-l6lpj" Oct 30 16:46:38.179966 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:38.179945 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-64c409-75bfc9f544-l6lpj"] Oct 30 16:46:38.194775 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:38.194746 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl58d\" (UniqueName: \"kubernetes.io/projected/e6cd9cc1-7206-4868-908a-05aed37aae54-kube-api-access-pl58d\") pod \"payload-pre-processing-e2e-ipp-b-64c409-54c8f8f675-cmds8\" (UID: \"e6cd9cc1-7206-4868-908a-05aed37aae54\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-64c409-54c8f8f675-cmds8" Oct 30 16:46:38.195000 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:38.194855 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6cd9cc1-7206-4868-908a-05aed37aae54-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-64c409-54c8f8f675-cmds8\" (UID: \"e6cd9cc1-7206-4868-908a-05aed37aae54\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-64c409-54c8f8f675-cmds8" Oct 30 16:46:38.197209 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:38.197183 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6cd9cc1-7206-4868-908a-05aed37aae54-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-64c409-54c8f8f675-cmds8\" (UID: \"e6cd9cc1-7206-4868-908a-05aed37aae54\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-64c409-54c8f8f675-cmds8" Oct 30 16:46:38.216638 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:38.216613 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl58d\" (UniqueName: \"kubernetes.io/projected/e6cd9cc1-7206-4868-908a-05aed37aae54-kube-api-access-pl58d\") pod \"payload-pre-processing-e2e-ipp-b-64c409-54c8f8f675-cmds8\" (UID: \"e6cd9cc1-7206-4868-908a-05aed37aae54\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-64c409-54c8f8f675-cmds8" Oct 30 16:46:38.296456 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:38.296378 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c1ea064-422e-49c7-9577-448806fc5e33-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-64c409-75bfc9f544-l6lpj\" (UID: \"6c1ea064-422e-49c7-9577-448806fc5e33\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-64c409-75bfc9f544-l6lpj" Oct 30 16:46:38.296614 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:38.296463 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvjw9\" (UniqueName: \"kubernetes.io/projected/6c1ea064-422e-49c7-9577-448806fc5e33-kube-api-access-lvjw9\") pod \"payload-processing-e2e-ipp-b-64c409-75bfc9f544-l6lpj\" (UID: \"6c1ea064-422e-49c7-9577-448806fc5e33\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-64c409-75bfc9f544-l6lpj" Oct 30 16:46:38.398001 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:38.397966 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c1ea064-422e-49c7-9577-448806fc5e33-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-64c409-75bfc9f544-l6lpj\" (UID: \"6c1ea064-422e-49c7-9577-448806fc5e33\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-64c409-75bfc9f544-l6lpj" Oct 30 16:46:38.398160 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:38.398067 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvjw9\" (UniqueName: \"kubernetes.io/projected/6c1ea064-422e-49c7-9577-448806fc5e33-kube-api-access-lvjw9\") pod \"payload-processing-e2e-ipp-b-64c409-75bfc9f544-l6lpj\" (UID: \"6c1ea064-422e-49c7-9577-448806fc5e33\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-64c409-75bfc9f544-l6lpj" Oct 30 16:46:38.398855 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:38.398833 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c1ea064-422e-49c7-9577-448806fc5e33-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-64c409-75bfc9f544-l6lpj\" (UID: \"6c1ea064-422e-49c7-9577-448806fc5e33\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-64c409-75bfc9f544-l6lpj" Oct 30 16:46:38.414802 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:38.414757 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-64c409-54c8f8f675-cmds8" Oct 30 16:46:38.427168 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:38.427149 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvjw9\" (UniqueName: \"kubernetes.io/projected/6c1ea064-422e-49c7-9577-448806fc5e33-kube-api-access-lvjw9\") pod \"payload-processing-e2e-ipp-b-64c409-75bfc9f544-l6lpj\" (UID: \"6c1ea064-422e-49c7-9577-448806fc5e33\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-64c409-75bfc9f544-l6lpj" Oct 30 16:46:38.478245 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:38.478121 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-64c409-75bfc9f544-l6lpj" Oct 30 16:46:38.721833 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:38.721805 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-64c409-54c8f8f675-cmds8"] Oct 30 16:46:38.724008 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:46:38.723983 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6cd9cc1_7206_4868_908a_05aed37aae54.slice/crio-7f26a8a386439f72567d64f34c223c55d9ad4345494ab9b2d6ef68c99e0758e7 WatchSource:0}: Error finding container 7f26a8a386439f72567d64f34c223c55d9ad4345494ab9b2d6ef68c99e0758e7: Status 404 returned error can't find the container with id 7f26a8a386439f72567d64f34c223c55d9ad4345494ab9b2d6ef68c99e0758e7 Oct 30 16:46:38.823109 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:38.823063 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-64c409-75bfc9f544-l6lpj"] Oct 30 16:46:38.825834 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:46:38.825717 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c1ea064_422e_49c7_9577_448806fc5e33.slice/crio-2a1b7edfd841af3212768c4f377610066f4484c43c7422ba4d6d71059467bb79 WatchSource:0}: Error finding container 2a1b7edfd841af3212768c4f377610066f4484c43c7422ba4d6d71059467bb79: Status 404 returned error can't find the container with id 2a1b7edfd841af3212768c4f377610066f4484c43c7422ba4d6d71059467bb79 Oct 30 16:46:39.666097 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:39.666059 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-64c409-54c8f8f675-cmds8" event={"ID":"e6cd9cc1-7206-4868-908a-05aed37aae54","Type":"ContainerStarted","Data":"2300b1b99387b2dde1ecd9a10bfa6a93a52272763daaa8f23c426c5e49c5bdfe"} Oct 30 16:46:39.666554 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:39.666120 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-64c409-54c8f8f675-cmds8" event={"ID":"e6cd9cc1-7206-4868-908a-05aed37aae54","Type":"ContainerStarted","Data":"7f26a8a386439f72567d64f34c223c55d9ad4345494ab9b2d6ef68c99e0758e7"} Oct 30 16:46:39.666554 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:39.666331 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-64c409-54c8f8f675-cmds8" Oct 30 16:46:39.667902 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:39.667871 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-64c409-75bfc9f544-l6lpj" event={"ID":"6c1ea064-422e-49c7-9577-448806fc5e33","Type":"ContainerStarted","Data":"7e6916b16339075bacda8b732614eaca0e7cdaf56a33ad050b85ee5096db5ba3"} Oct 30 16:46:39.668034 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:39.667925 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-64c409-75bfc9f544-l6lpj" Oct 30 16:46:39.668034 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:39.667948 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-64c409-75bfc9f544-l6lpj" event={"ID":"6c1ea064-422e-49c7-9577-448806fc5e33","Type":"ContainerStarted","Data":"2a1b7edfd841af3212768c4f377610066f4484c43c7422ba4d6d71059467bb79"} Oct 30 16:46:39.703953 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:39.703899 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-64c409-54c8f8f675-cmds8" podStartSLOduration=2.70388232 podStartE2EDuration="2.70388232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:46:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:46:39.70182639 +0000 UTC m=+1426.929001491" watchObservedRunningTime="2026-10-30 16:46:39.70388232 +0000 UTC m=+1426.931057423" Oct 30 16:46:39.741141 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:39.741081 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-64c409-75bfc9f544-l6lpj" podStartSLOduration=1.741064888 podStartE2EDuration="1.741064888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:46:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:46:39.737750277 +0000 UTC m=+1426.964925377" watchObservedRunningTime="2026-10-30 16:46:39.741064888 +0000 UTC m=+1426.968239990" Oct 30 16:46:40.764456 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:40.764421 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4a4c68-km6j4"] Oct 30 16:46:40.768642 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:40.768625 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4a4c68-km6j4" Oct 30 16:46:40.781154 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:40.781133 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:46:40.791092 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:40.791069 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:46:40.796303 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:40.796271 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4a4c68-km6j4"] Oct 30 16:46:40.840810 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:40.839279 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/714be7d8-f0ae-40d4-91ef-b8743f7912fd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-4a4c68-km6j4\" (UID: \"714be7d8-f0ae-40d4-91ef-b8743f7912fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4a4c68-km6j4" Oct 30 16:46:40.840810 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:40.839396 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbwzx\" (UniqueName: \"kubernetes.io/projected/714be7d8-f0ae-40d4-91ef-b8743f7912fd-kube-api-access-xbwzx\") pod \"maas-api-revoke-keys-e2e-ipp-b-4a4c68-km6j4\" (UID: \"714be7d8-f0ae-40d4-91ef-b8743f7912fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4a4c68-km6j4" Oct 30 16:46:40.840810 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:40.839473 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/714be7d8-f0ae-40d4-91ef-b8743f7912fd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-4a4c68-km6j4\" (UID: \"714be7d8-f0ae-40d4-91ef-b8743f7912fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4a4c68-km6j4" Oct 30 16:46:40.859810 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:40.851825 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:46:40.859810 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:40.851994 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 16:46:40.940191 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:40.940163 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/714be7d8-f0ae-40d4-91ef-b8743f7912fd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-4a4c68-km6j4\" (UID: \"714be7d8-f0ae-40d4-91ef-b8743f7912fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4a4c68-km6j4" Oct 30 16:46:40.940375 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:40.940238 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbwzx\" (UniqueName: \"kubernetes.io/projected/714be7d8-f0ae-40d4-91ef-b8743f7912fd-kube-api-access-xbwzx\") pod \"maas-api-revoke-keys-e2e-ipp-b-4a4c68-km6j4\" (UID: \"714be7d8-f0ae-40d4-91ef-b8743f7912fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4a4c68-km6j4" Oct 30 16:46:40.940375 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:40.940286 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/714be7d8-f0ae-40d4-91ef-b8743f7912fd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-4a4c68-km6j4\" (UID: \"714be7d8-f0ae-40d4-91ef-b8743f7912fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4a4c68-km6j4" Oct 30 16:46:40.940996 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:40.940968 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/714be7d8-f0ae-40d4-91ef-b8743f7912fd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-4a4c68-km6j4\" (UID: \"714be7d8-f0ae-40d4-91ef-b8743f7912fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4a4c68-km6j4" Oct 30 16:46:40.942899 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:40.942868 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/714be7d8-f0ae-40d4-91ef-b8743f7912fd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-4a4c68-km6j4\" (UID: \"714be7d8-f0ae-40d4-91ef-b8743f7912fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4a4c68-km6j4" Oct 30 16:46:40.961685 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:40.961662 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbwzx\" (UniqueName: \"kubernetes.io/projected/714be7d8-f0ae-40d4-91ef-b8743f7912fd-kube-api-access-xbwzx\") pod \"maas-api-revoke-keys-e2e-ipp-b-4a4c68-km6j4\" (UID: \"714be7d8-f0ae-40d4-91ef-b8743f7912fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4a4c68-km6j4" Oct 30 16:46:41.083474 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:41.083380 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4a4c68-km6j4" Oct 30 16:46:41.403250 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:41.403197 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4a4c68-km6j4"] Oct 30 16:46:41.407590 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:46:41.407561 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod714be7d8_f0ae_40d4_91ef_b8743f7912fd.slice/crio-a7962379247ed389f6f84ebd960ee9fda620f881b6ae45fdde1e808776878de0 WatchSource:0}: Error finding container a7962379247ed389f6f84ebd960ee9fda620f881b6ae45fdde1e808776878de0: Status 404 returned error can't find the container with id a7962379247ed389f6f84ebd960ee9fda620f881b6ae45fdde1e808776878de0 Oct 30 16:46:41.684608 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:41.684530 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4a4c68-km6j4" event={"ID":"714be7d8-f0ae-40d4-91ef-b8743f7912fd","Type":"ContainerStarted","Data":"a7962379247ed389f6f84ebd960ee9fda620f881b6ae45fdde1e808776878de0"} Oct 30 16:46:42.693878 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:42.693842 4493 generic.go:356] "Generic (PLEG): container finished" podID="714be7d8-f0ae-40d4-91ef-b8743f7912fd" containerID="94e66a1c53d783955c301187030fb050970346688fb0dc9d6860a7488bb7358d" exitCode=0 Oct 30 16:46:42.694342 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:42.693893 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4a4c68-km6j4" event={"ID":"714be7d8-f0ae-40d4-91ef-b8743f7912fd","Type":"ContainerDied","Data":"94e66a1c53d783955c301187030fb050970346688fb0dc9d6860a7488bb7358d"} Oct 30 16:46:43.888158 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:43.888133 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4a4c68-km6j4" Oct 30 16:46:43.985409 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:43.985334 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/714be7d8-f0ae-40d4-91ef-b8743f7912fd-kube-api-access-xbwzx\" (UniqueName: \"kubernetes.io/projected/714be7d8-f0ae-40d4-91ef-b8743f7912fd-kube-api-access-xbwzx\") pod \"714be7d8-f0ae-40d4-91ef-b8743f7912fd\" (UID: \"714be7d8-f0ae-40d4-91ef-b8743f7912fd\") " Oct 30 16:46:43.985409 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:43.985380 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/714be7d8-f0ae-40d4-91ef-b8743f7912fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/714be7d8-f0ae-40d4-91ef-b8743f7912fd-maas-api-xks-ca\") pod \"714be7d8-f0ae-40d4-91ef-b8743f7912fd\" (UID: \"714be7d8-f0ae-40d4-91ef-b8743f7912fd\") " Oct 30 16:46:43.985605 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:43.985500 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/714be7d8-f0ae-40d4-91ef-b8743f7912fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/714be7d8-f0ae-40d4-91ef-b8743f7912fd-maas-api-openshift-ca\") pod \"714be7d8-f0ae-40d4-91ef-b8743f7912fd\" (UID: \"714be7d8-f0ae-40d4-91ef-b8743f7912fd\") " Oct 30 16:46:43.985897 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:43.985873 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/714be7d8-f0ae-40d4-91ef-b8743f7912fd-maas-api-openshift-ca" pod "714be7d8-f0ae-40d4-91ef-b8743f7912fd" (UID: "714be7d8-f0ae-40d4-91ef-b8743f7912fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:46:43.987506 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:43.987486 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/714be7d8-f0ae-40d4-91ef-b8743f7912fd-maas-api-xks-ca" pod "714be7d8-f0ae-40d4-91ef-b8743f7912fd" (UID: "714be7d8-f0ae-40d4-91ef-b8743f7912fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:46:43.987668 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:43.987653 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/714be7d8-f0ae-40d4-91ef-b8743f7912fd-kube-api-access-xbwzx" pod "714be7d8-f0ae-40d4-91ef-b8743f7912fd" (UID: "714be7d8-f0ae-40d4-91ef-b8743f7912fd"). InnerVolumeSpecName "kube-api-access-xbwzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:46:44.087227 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:44.087195 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/714be7d8-f0ae-40d4-91ef-b8743f7912fd-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:46:44.087227 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:44.087222 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbwzx\" (UniqueName: \"kubernetes.io/projected/714be7d8-f0ae-40d4-91ef-b8743f7912fd-kube-api-access-xbwzx\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:46:44.087227 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:44.087232 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/714be7d8-f0ae-40d4-91ef-b8743f7912fd-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:46:44.713817 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:44.710079 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4a4c68-km6j4" Oct 30 16:46:44.713817 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:44.710102 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4a4c68-km6j4" event={"ID":"714be7d8-f0ae-40d4-91ef-b8743f7912fd","Type":"ContainerDied","Data":"a7962379247ed389f6f84ebd960ee9fda620f881b6ae45fdde1e808776878de0"} Oct 30 16:46:44.713817 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:44.710147 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7962379247ed389f6f84ebd960ee9fda620f881b6ae45fdde1e808776878de0" Oct 30 16:46:45.033273 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:45.033187 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8077da2e-xt2z9"] Oct 30 16:46:45.038669 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:45.038643 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8077da2e-xt2z9"] Oct 30 16:46:46.955858 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:46.955349 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3a489ce-69b9-44fb-8f42-db7c25762d3c" path="/var/lib/kubelet/pods/a3a489ce-69b9-44fb-8f42-db7c25762d3c/volumes" Oct 30 16:46:48.936163 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:48.936132 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-843ab3-6qk4m"] Oct 30 16:46:48.936976 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:48.936954 4493 cpu_manager.go:414] "containerMap: removing container" podUID="714be7d8-f0ae-40d4-91ef-b8743f7912fd" containerName="revoke-keys" Oct 30 16:46:48.936976 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:48.936976 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="714be7d8-f0ae-40d4-91ef-b8743f7912fd" containerName="revoke-keys" Oct 30 16:46:48.937191 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:48.937178 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="714be7d8-f0ae-40d4-91ef-b8743f7912fd" containerName="revoke-keys" Oct 30 16:46:48.940934 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:48.940913 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-843ab3-6qk4m" Oct 30 16:46:48.959830 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:48.959799 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-843ab3-6qk4m"] Oct 30 16:46:48.964831 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:48.964807 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:46:48.964941 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:48.964892 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:46:48.996276 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:48.996244 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:46:48.996422 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:48.996406 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 16:46:49.061221 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:49.061183 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50560313-6f31-4b21-8e1d-51cb701f5e7c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-843ab3-6qk4m\" (UID: \"50560313-6f31-4b21-8e1d-51cb701f5e7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-843ab3-6qk4m" Oct 30 16:46:49.061409 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:49.061233 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50560313-6f31-4b21-8e1d-51cb701f5e7c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-843ab3-6qk4m\" (UID: \"50560313-6f31-4b21-8e1d-51cb701f5e7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-843ab3-6qk4m" Oct 30 16:46:49.061409 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:49.061272 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4rpg\" (UniqueName: \"kubernetes.io/projected/50560313-6f31-4b21-8e1d-51cb701f5e7c-kube-api-access-w4rpg\") pod \"maas-api-revoke-keys-e2e-ipp-a-843ab3-6qk4m\" (UID: \"50560313-6f31-4b21-8e1d-51cb701f5e7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-843ab3-6qk4m" Oct 30 16:46:49.162591 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:49.162560 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50560313-6f31-4b21-8e1d-51cb701f5e7c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-843ab3-6qk4m\" (UID: \"50560313-6f31-4b21-8e1d-51cb701f5e7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-843ab3-6qk4m" Oct 30 16:46:49.162811 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:49.162601 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50560313-6f31-4b21-8e1d-51cb701f5e7c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-843ab3-6qk4m\" (UID: \"50560313-6f31-4b21-8e1d-51cb701f5e7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-843ab3-6qk4m" Oct 30 16:46:49.162811 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:49.162627 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4rpg\" (UniqueName: \"kubernetes.io/projected/50560313-6f31-4b21-8e1d-51cb701f5e7c-kube-api-access-w4rpg\") pod \"maas-api-revoke-keys-e2e-ipp-a-843ab3-6qk4m\" (UID: \"50560313-6f31-4b21-8e1d-51cb701f5e7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-843ab3-6qk4m" Oct 30 16:46:49.163346 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:49.163320 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50560313-6f31-4b21-8e1d-51cb701f5e7c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-843ab3-6qk4m\" (UID: \"50560313-6f31-4b21-8e1d-51cb701f5e7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-843ab3-6qk4m" Oct 30 16:46:49.165273 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:49.165250 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50560313-6f31-4b21-8e1d-51cb701f5e7c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-843ab3-6qk4m\" (UID: \"50560313-6f31-4b21-8e1d-51cb701f5e7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-843ab3-6qk4m" Oct 30 16:46:49.184575 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:49.184541 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4rpg\" (UniqueName: \"kubernetes.io/projected/50560313-6f31-4b21-8e1d-51cb701f5e7c-kube-api-access-w4rpg\") pod \"maas-api-revoke-keys-e2e-ipp-a-843ab3-6qk4m\" (UID: \"50560313-6f31-4b21-8e1d-51cb701f5e7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-843ab3-6qk4m" Oct 30 16:46:49.254957 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:49.254869 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-843ab3-6qk4m" Oct 30 16:46:49.560195 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:49.560125 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-843ab3-6qk4m"] Oct 30 16:46:49.562849 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:46:49.562822 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50560313_6f31_4b21_8e1d_51cb701f5e7c.slice/crio-80381099fa18118350ed06c56617cec857d89260b65f9da4b82fe1d42d4589f0 WatchSource:0}: Error finding container 80381099fa18118350ed06c56617cec857d89260b65f9da4b82fe1d42d4589f0: Status 404 returned error can't find the container with id 80381099fa18118350ed06c56617cec857d89260b65f9da4b82fe1d42d4589f0 Oct 30 16:46:49.757711 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:49.757684 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-843ab3-6qk4m" event={"ID":"50560313-6f31-4b21-8e1d-51cb701f5e7c","Type":"ContainerStarted","Data":"80381099fa18118350ed06c56617cec857d89260b65f9da4b82fe1d42d4589f0"} Oct 30 16:46:50.344803 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:50.344758 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 16:46:50.680458 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:50.680418 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-64c409-54c8f8f675-cmds8" Oct 30 16:46:50.681083 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:50.681066 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-64c409-75bfc9f544-l6lpj" Oct 30 16:46:50.801837 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:50.801768 4493 generic.go:356] "Generic (PLEG): container finished" podID="50560313-6f31-4b21-8e1d-51cb701f5e7c" containerID="286ae5724b19f893f6f9880bccf62a763caa00dd1084660f32c384efc918256d" exitCode=0 Oct 30 16:46:50.802032 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:50.801833 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-843ab3-6qk4m" event={"ID":"50560313-6f31-4b21-8e1d-51cb701f5e7c","Type":"ContainerDied","Data":"286ae5724b19f893f6f9880bccf62a763caa00dd1084660f32c384efc918256d"} Oct 30 16:46:52.011232 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:52.011204 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-843ab3-6qk4m" Oct 30 16:46:52.116550 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:52.116472 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50560313-6f31-4b21-8e1d-51cb701f5e7c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50560313-6f31-4b21-8e1d-51cb701f5e7c-maas-api-xks-ca\") pod \"50560313-6f31-4b21-8e1d-51cb701f5e7c\" (UID: \"50560313-6f31-4b21-8e1d-51cb701f5e7c\") " Oct 30 16:46:52.116550 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:52.116547 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50560313-6f31-4b21-8e1d-51cb701f5e7c-kube-api-access-w4rpg\" (UniqueName: \"kubernetes.io/projected/50560313-6f31-4b21-8e1d-51cb701f5e7c-kube-api-access-w4rpg\") pod \"50560313-6f31-4b21-8e1d-51cb701f5e7c\" (UID: \"50560313-6f31-4b21-8e1d-51cb701f5e7c\") " Oct 30 16:46:52.116739 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:52.116723 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50560313-6f31-4b21-8e1d-51cb701f5e7c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50560313-6f31-4b21-8e1d-51cb701f5e7c-maas-api-openshift-ca\") pod \"50560313-6f31-4b21-8e1d-51cb701f5e7c\" (UID: \"50560313-6f31-4b21-8e1d-51cb701f5e7c\") " Oct 30 16:46:52.117109 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:52.117082 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50560313-6f31-4b21-8e1d-51cb701f5e7c-maas-api-openshift-ca" pod "50560313-6f31-4b21-8e1d-51cb701f5e7c" (UID: "50560313-6f31-4b21-8e1d-51cb701f5e7c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:46:52.117330 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:52.117313 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50560313-6f31-4b21-8e1d-51cb701f5e7c-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:46:52.119262 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:52.119236 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50560313-6f31-4b21-8e1d-51cb701f5e7c-maas-api-xks-ca" pod "50560313-6f31-4b21-8e1d-51cb701f5e7c" (UID: "50560313-6f31-4b21-8e1d-51cb701f5e7c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:46:52.119374 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:52.119266 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50560313-6f31-4b21-8e1d-51cb701f5e7c-kube-api-access-w4rpg" pod "50560313-6f31-4b21-8e1d-51cb701f5e7c" (UID: "50560313-6f31-4b21-8e1d-51cb701f5e7c"). InnerVolumeSpecName "kube-api-access-w4rpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:46:52.218884 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:52.218849 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4rpg\" (UniqueName: \"kubernetes.io/projected/50560313-6f31-4b21-8e1d-51cb701f5e7c-kube-api-access-w4rpg\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:46:52.218884 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:52.218880 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50560313-6f31-4b21-8e1d-51cb701f5e7c-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:46:52.832222 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:52.832185 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-843ab3-6qk4m" event={"ID":"50560313-6f31-4b21-8e1d-51cb701f5e7c","Type":"ContainerDied","Data":"80381099fa18118350ed06c56617cec857d89260b65f9da4b82fe1d42d4589f0"} Oct 30 16:46:52.832222 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:52.832225 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-843ab3-6qk4m" Oct 30 16:46:52.832453 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:52.832244 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80381099fa18118350ed06c56617cec857d89260b65f9da4b82fe1d42d4589f0" Oct 30 16:46:54.204531 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:54.204494 4493 scope.go:122] "RemoveContainer" containerID="0da036f25ac77111e7837f047bd4609bfe61641de2fdfcb1cfca0a9efb0915cf" Oct 30 16:46:54.216665 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:54.216636 4493 scope.go:122] "RemoveContainer" containerID="04220862c312a9077b0163a12af7aca0c11644531dcde2dd3d82564c70dd1a78" Oct 30 16:46:54.227548 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:54.227530 4493 scope.go:122] "RemoveContainer" containerID="ae3732987a32634279ae045339a6a93875ebe1f95bbfb0dadc77d355298e2c26" Oct 30 16:46:54.238701 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:54.238681 4493 scope.go:122] "RemoveContainer" containerID="00e675ce42e43826328445bb71b1af0002ea4b4988a95182465df3931af992bf" Oct 30 16:46:56.097384 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:56.097353 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7c23a4-zkbn9"] Oct 30 16:46:56.098159 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:56.098139 4493 cpu_manager.go:414] "containerMap: removing container" podUID="50560313-6f31-4b21-8e1d-51cb701f5e7c" containerName="revoke-keys" Oct 30 16:46:56.098270 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:56.098159 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50560313-6f31-4b21-8e1d-51cb701f5e7c" containerName="revoke-keys" Oct 30 16:46:56.098359 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:56.098346 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="50560313-6f31-4b21-8e1d-51cb701f5e7c" containerName="revoke-keys" Oct 30 16:46:56.103267 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:56.103247 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7c23a4-zkbn9" Oct 30 16:46:56.116011 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:56.115987 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:46:56.125863 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:56.125842 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:46:56.126285 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:56.126265 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:46:56.126285 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:56.126279 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 16:46:56.130245 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:56.130225 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7c23a4-zkbn9"] Oct 30 16:46:56.169563 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:56.169524 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7feaf66b-d3c1-4e24-b006-6104ceb28a65-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7c23a4-zkbn9\" (UID: \"7feaf66b-d3c1-4e24-b006-6104ceb28a65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7c23a4-zkbn9" Oct 30 16:46:56.169725 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:56.169563 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zxf9\" (UniqueName: \"kubernetes.io/projected/7feaf66b-d3c1-4e24-b006-6104ceb28a65-kube-api-access-2zxf9\") pod \"maas-api-revoke-keys-e2e-ipp-a-7c23a4-zkbn9\" (UID: \"7feaf66b-d3c1-4e24-b006-6104ceb28a65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7c23a4-zkbn9" Oct 30 16:46:56.169852 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:56.169747 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7feaf66b-d3c1-4e24-b006-6104ceb28a65-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7c23a4-zkbn9\" (UID: \"7feaf66b-d3c1-4e24-b006-6104ceb28a65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7c23a4-zkbn9" Oct 30 16:46:56.271032 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:56.271002 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7feaf66b-d3c1-4e24-b006-6104ceb28a65-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7c23a4-zkbn9\" (UID: \"7feaf66b-d3c1-4e24-b006-6104ceb28a65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7c23a4-zkbn9" Oct 30 16:46:56.271032 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:56.271034 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zxf9\" (UniqueName: \"kubernetes.io/projected/7feaf66b-d3c1-4e24-b006-6104ceb28a65-kube-api-access-2zxf9\") pod \"maas-api-revoke-keys-e2e-ipp-a-7c23a4-zkbn9\" (UID: \"7feaf66b-d3c1-4e24-b006-6104ceb28a65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7c23a4-zkbn9" Oct 30 16:46:56.271365 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:56.271131 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7feaf66b-d3c1-4e24-b006-6104ceb28a65-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7c23a4-zkbn9\" (UID: \"7feaf66b-d3c1-4e24-b006-6104ceb28a65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7c23a4-zkbn9" Oct 30 16:46:56.271802 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:56.271759 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7feaf66b-d3c1-4e24-b006-6104ceb28a65-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7c23a4-zkbn9\" (UID: \"7feaf66b-d3c1-4e24-b006-6104ceb28a65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7c23a4-zkbn9" Oct 30 16:46:56.273455 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:56.273429 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7feaf66b-d3c1-4e24-b006-6104ceb28a65-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7c23a4-zkbn9\" (UID: \"7feaf66b-d3c1-4e24-b006-6104ceb28a65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7c23a4-zkbn9" Oct 30 16:46:56.292071 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:56.292047 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zxf9\" (UniqueName: \"kubernetes.io/projected/7feaf66b-d3c1-4e24-b006-6104ceb28a65-kube-api-access-2zxf9\") pod \"maas-api-revoke-keys-e2e-ipp-a-7c23a4-zkbn9\" (UID: \"7feaf66b-d3c1-4e24-b006-6104ceb28a65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7c23a4-zkbn9" Oct 30 16:46:56.418988 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:56.418948 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7c23a4-zkbn9" Oct 30 16:46:56.732310 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:56.732283 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7c23a4-zkbn9"] Oct 30 16:46:56.735306 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:46:56.735280 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7feaf66b_d3c1_4e24_b006_6104ceb28a65.slice/crio-9ee92e3d589a0efe5848f8bc6c638f07cb0982243523ea612a5f552aa3528769 WatchSource:0}: Error finding container 9ee92e3d589a0efe5848f8bc6c638f07cb0982243523ea612a5f552aa3528769: Status 404 returned error can't find the container with id 9ee92e3d589a0efe5848f8bc6c638f07cb0982243523ea612a5f552aa3528769 Oct 30 16:46:56.866505 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:56.866467 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7c23a4-zkbn9" event={"ID":"7feaf66b-d3c1-4e24-b006-6104ceb28a65","Type":"ContainerStarted","Data":"9ee92e3d589a0efe5848f8bc6c638f07cb0982243523ea612a5f552aa3528769"} Oct 30 16:46:57.876731 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:57.876644 4493 generic.go:356] "Generic (PLEG): container finished" podID="7feaf66b-d3c1-4e24-b006-6104ceb28a65" containerID="eb155eb7828685f12d7685dcfd14ced8ded79e7b218ef31ada07d3ce2a1c7d49" exitCode=0 Oct 30 16:46:57.876731 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:57.876708 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7c23a4-zkbn9" event={"ID":"7feaf66b-d3c1-4e24-b006-6104ceb28a65","Type":"ContainerDied","Data":"eb155eb7828685f12d7685dcfd14ced8ded79e7b218ef31ada07d3ce2a1c7d49"} Oct 30 16:46:59.059923 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:59.059897 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7c23a4-zkbn9" Oct 30 16:46:59.120830 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:59.120771 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7feaf66b-d3c1-4e24-b006-6104ceb28a65-kube-api-access-2zxf9\" (UniqueName: \"kubernetes.io/projected/7feaf66b-d3c1-4e24-b006-6104ceb28a65-kube-api-access-2zxf9\") pod \"7feaf66b-d3c1-4e24-b006-6104ceb28a65\" (UID: \"7feaf66b-d3c1-4e24-b006-6104ceb28a65\") " Oct 30 16:46:59.120999 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:59.120869 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7feaf66b-d3c1-4e24-b006-6104ceb28a65-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7feaf66b-d3c1-4e24-b006-6104ceb28a65-maas-api-openshift-ca\") pod \"7feaf66b-d3c1-4e24-b006-6104ceb28a65\" (UID: \"7feaf66b-d3c1-4e24-b006-6104ceb28a65\") " Oct 30 16:46:59.120999 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:59.120981 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7feaf66b-d3c1-4e24-b006-6104ceb28a65-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7feaf66b-d3c1-4e24-b006-6104ceb28a65-maas-api-xks-ca\") pod \"7feaf66b-d3c1-4e24-b006-6104ceb28a65\" (UID: \"7feaf66b-d3c1-4e24-b006-6104ceb28a65\") " Oct 30 16:46:59.121245 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:59.121217 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7feaf66b-d3c1-4e24-b006-6104ceb28a65-maas-api-openshift-ca" pod "7feaf66b-d3c1-4e24-b006-6104ceb28a65" (UID: "7feaf66b-d3c1-4e24-b006-6104ceb28a65"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:46:59.121590 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:59.121569 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7feaf66b-d3c1-4e24-b006-6104ceb28a65-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:46:59.123184 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:59.123151 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7feaf66b-d3c1-4e24-b006-6104ceb28a65-maas-api-xks-ca" pod "7feaf66b-d3c1-4e24-b006-6104ceb28a65" (UID: "7feaf66b-d3c1-4e24-b006-6104ceb28a65"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:46:59.123304 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:59.123283 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7feaf66b-d3c1-4e24-b006-6104ceb28a65-kube-api-access-2zxf9" pod "7feaf66b-d3c1-4e24-b006-6104ceb28a65" (UID: "7feaf66b-d3c1-4e24-b006-6104ceb28a65"). InnerVolumeSpecName "kube-api-access-2zxf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:46:59.223167 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:59.223140 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zxf9\" (UniqueName: \"kubernetes.io/projected/7feaf66b-d3c1-4e24-b006-6104ceb28a65-kube-api-access-2zxf9\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:46:59.223167 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:59.223166 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7feaf66b-d3c1-4e24-b006-6104ceb28a65-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:46:59.904172 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:59.904128 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7c23a4-zkbn9" event={"ID":"7feaf66b-d3c1-4e24-b006-6104ceb28a65","Type":"ContainerDied","Data":"9ee92e3d589a0efe5848f8bc6c638f07cb0982243523ea612a5f552aa3528769"} Oct 30 16:46:59.904391 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:59.904200 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ee92e3d589a0efe5848f8bc6c638f07cb0982243523ea612a5f552aa3528769" Oct 30 16:46:59.904391 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:46:59.904272 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7c23a4-zkbn9" Oct 30 16:47:00.342112 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:47:00.342033 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 16:47:00.556948 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:00.556911 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-08fcc8-g6jb8"] Oct 30 16:47:00.565008 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:00.564976 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-08fcc8-g6jb8"] Oct 30 16:47:00.936491 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:00.936455 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f363aff-2fc7-487f-9fd0-47fdd06276a3" path="/var/lib/kubelet/pods/8f363aff-2fc7-487f-9fd0-47fdd06276a3/volumes" Oct 30 16:47:02.867601 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:02.867567 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-843ab3-84f5645776-wbn9k"] Oct 30 16:47:02.868477 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:02.868438 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-843ab3-84f5645776-wbn9k" podUID="47194c28-9025-4eb5-98e6-9482dc97d190" containerName="payload-processing" containerID="cri-o://3778e82bac83f982139b0da99bf0886e6fd926490253cdb8d9f42942ea4cbb7e" gracePeriod=30 Oct 30 16:47:03.196915 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:03.196890 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-843ab3-84f5645776-wbn9k" Oct 30 16:47:03.326599 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:03.326568 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47194c28-9025-4eb5-98e6-9482dc97d190-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47194c28-9025-4eb5-98e6-9482dc97d190-plugins-config-volume\") pod \"47194c28-9025-4eb5-98e6-9482dc97d190\" (UID: \"47194c28-9025-4eb5-98e6-9482dc97d190\") " Oct 30 16:47:03.327092 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:03.327069 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47194c28-9025-4eb5-98e6-9482dc97d190-kube-api-access-knz88\" (UniqueName: \"kubernetes.io/projected/47194c28-9025-4eb5-98e6-9482dc97d190-kube-api-access-knz88\") pod \"47194c28-9025-4eb5-98e6-9482dc97d190\" (UID: \"47194c28-9025-4eb5-98e6-9482dc97d190\") " Oct 30 16:47:03.336150 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:03.336075 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47194c28-9025-4eb5-98e6-9482dc97d190-plugins-config-volume" pod "47194c28-9025-4eb5-98e6-9482dc97d190" (UID: "47194c28-9025-4eb5-98e6-9482dc97d190"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:47:03.367897 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:03.362224 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47194c28-9025-4eb5-98e6-9482dc97d190-kube-api-access-knz88" pod "47194c28-9025-4eb5-98e6-9482dc97d190" (UID: "47194c28-9025-4eb5-98e6-9482dc97d190"). InnerVolumeSpecName "kube-api-access-knz88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:47:03.430765 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:03.430573 4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47194c28-9025-4eb5-98e6-9482dc97d190-plugins-config-volume\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:47:03.430765 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:03.430653 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knz88\" (UniqueName: \"kubernetes.io/projected/47194c28-9025-4eb5-98e6-9482dc97d190-kube-api-access-knz88\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:47:03.566252 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:03.566212 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-843ab3-6qk4m"] Oct 30 16:47:03.573045 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:03.573015 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-843ab3-6qk4m"] Oct 30 16:47:03.996235 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:03.996198 4493 generic.go:356] "Generic (PLEG): container finished" podID="47194c28-9025-4eb5-98e6-9482dc97d190" containerID="3778e82bac83f982139b0da99bf0886e6fd926490253cdb8d9f42942ea4cbb7e" exitCode=0 Oct 30 16:47:03.996748 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:03.996250 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-843ab3-84f5645776-wbn9k" Oct 30 16:47:03.996748 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:03.996283 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-843ab3-84f5645776-wbn9k" event={"ID":"47194c28-9025-4eb5-98e6-9482dc97d190","Type":"ContainerDied","Data":"3778e82bac83f982139b0da99bf0886e6fd926490253cdb8d9f42942ea4cbb7e"} Oct 30 16:47:03.996748 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:03.996352 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-843ab3-84f5645776-wbn9k" event={"ID":"47194c28-9025-4eb5-98e6-9482dc97d190","Type":"ContainerDied","Data":"3c01a86901465ec7f5129b5623230357e27129dace685594306fd942b7dfafe9"} Oct 30 16:47:03.996748 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:03.996385 4493 scope.go:122] "RemoveContainer" containerID="3778e82bac83f982139b0da99bf0886e6fd926490253cdb8d9f42942ea4cbb7e" Oct 30 16:47:04.008595 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:04.008578 4493 scope.go:122] "RemoveContainer" containerID="3778e82bac83f982139b0da99bf0886e6fd926490253cdb8d9f42942ea4cbb7e" Oct 30 16:47:04.008921 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:47:04.008899 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3778e82bac83f982139b0da99bf0886e6fd926490253cdb8d9f42942ea4cbb7e\": container with ID starting with 3778e82bac83f982139b0da99bf0886e6fd926490253cdb8d9f42942ea4cbb7e not found: ID does not exist" containerID="3778e82bac83f982139b0da99bf0886e6fd926490253cdb8d9f42942ea4cbb7e" Oct 30 16:47:04.009022 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:04.008928 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3778e82bac83f982139b0da99bf0886e6fd926490253cdb8d9f42942ea4cbb7e"} err="failed to get container status \"3778e82bac83f982139b0da99bf0886e6fd926490253cdb8d9f42942ea4cbb7e\": rpc error: code = NotFound desc = could not find container \"3778e82bac83f982139b0da99bf0886e6fd926490253cdb8d9f42942ea4cbb7e\": container with ID starting with 3778e82bac83f982139b0da99bf0886e6fd926490253cdb8d9f42942ea4cbb7e not found: ID does not exist" Oct 30 16:47:04.043905 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:04.043883 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-843ab3-84f5645776-wbn9k"] Oct 30 16:47:04.054147 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:04.054124 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-843ab3-84f5645776-wbn9k"] Oct 30 16:47:04.934287 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:04.934248 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47194c28-9025-4eb5-98e6-9482dc97d190" path="/var/lib/kubelet/pods/47194c28-9025-4eb5-98e6-9482dc97d190/volumes" Oct 30 16:47:04.934838 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:04.934821 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50560313-6f31-4b21-8e1d-51cb701f5e7c" path="/var/lib/kubelet/pods/50560313-6f31-4b21-8e1d-51cb701f5e7c/volumes" Oct 30 16:47:08.631301 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:08.631260 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f9970a-nbr54"] Oct 30 16:47:08.632134 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:08.632112 4493 cpu_manager.go:414] "containerMap: removing container" podUID="7feaf66b-d3c1-4e24-b006-6104ceb28a65" containerName="revoke-keys" Oct 30 16:47:08.632134 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:08.632134 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7feaf66b-d3c1-4e24-b006-6104ceb28a65" containerName="revoke-keys" Oct 30 16:47:08.632287 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:08.632150 4493 cpu_manager.go:414] "containerMap: removing container" podUID="47194c28-9025-4eb5-98e6-9482dc97d190" containerName="payload-processing" Oct 30 16:47:08.632287 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:08.632158 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47194c28-9025-4eb5-98e6-9482dc97d190" containerName="payload-processing" Oct 30 16:47:08.632398 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:08.632342 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="7feaf66b-d3c1-4e24-b006-6104ceb28a65" containerName="revoke-keys" Oct 30 16:47:08.632398 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:08.632359 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="47194c28-9025-4eb5-98e6-9482dc97d190" containerName="payload-processing" Oct 30 16:47:08.637197 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:08.637177 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f9970a-nbr54" Oct 30 16:47:08.650274 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:08.650255 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:47:08.659466 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:08.659442 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:47:08.663862 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:08.663832 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f9970a-nbr54"] Oct 30 16:47:08.667006 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:08.666990 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:47:08.667228 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:08.667209 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 16:47:08.805577 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:08.805539 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c89c2fc2-7e6a-490b-9023-f98f2b9bfd89-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f9970a-nbr54\" (UID: \"c89c2fc2-7e6a-490b-9023-f98f2b9bfd89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f9970a-nbr54" Oct 30 16:47:08.805765 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:08.805703 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlrs4\" (UniqueName: \"kubernetes.io/projected/c89c2fc2-7e6a-490b-9023-f98f2b9bfd89-kube-api-access-rlrs4\") pod \"maas-api-revoke-keys-e2e-api-b-f9970a-nbr54\" (UID: \"c89c2fc2-7e6a-490b-9023-f98f2b9bfd89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f9970a-nbr54" Oct 30 16:47:08.805765 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:08.805740 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c89c2fc2-7e6a-490b-9023-f98f2b9bfd89-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f9970a-nbr54\" (UID: \"c89c2fc2-7e6a-490b-9023-f98f2b9bfd89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f9970a-nbr54" Oct 30 16:47:08.907239 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:08.907210 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c89c2fc2-7e6a-490b-9023-f98f2b9bfd89-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f9970a-nbr54\" (UID: \"c89c2fc2-7e6a-490b-9023-f98f2b9bfd89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f9970a-nbr54" Oct 30 16:47:08.907403 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:08.907339 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlrs4\" (UniqueName: \"kubernetes.io/projected/c89c2fc2-7e6a-490b-9023-f98f2b9bfd89-kube-api-access-rlrs4\") pod \"maas-api-revoke-keys-e2e-api-b-f9970a-nbr54\" (UID: \"c89c2fc2-7e6a-490b-9023-f98f2b9bfd89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f9970a-nbr54" Oct 30 16:47:08.907403 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:08.907377 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c89c2fc2-7e6a-490b-9023-f98f2b9bfd89-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f9970a-nbr54\" (UID: \"c89c2fc2-7e6a-490b-9023-f98f2b9bfd89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f9970a-nbr54" Oct 30 16:47:08.908106 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:08.908044 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c89c2fc2-7e6a-490b-9023-f98f2b9bfd89-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f9970a-nbr54\" (UID: \"c89c2fc2-7e6a-490b-9023-f98f2b9bfd89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f9970a-nbr54" Oct 30 16:47:08.909995 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:08.909966 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c89c2fc2-7e6a-490b-9023-f98f2b9bfd89-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f9970a-nbr54\" (UID: \"c89c2fc2-7e6a-490b-9023-f98f2b9bfd89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f9970a-nbr54" Oct 30 16:47:08.928726 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:08.928706 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlrs4\" (UniqueName: \"kubernetes.io/projected/c89c2fc2-7e6a-490b-9023-f98f2b9bfd89-kube-api-access-rlrs4\") pod \"maas-api-revoke-keys-e2e-api-b-f9970a-nbr54\" (UID: \"c89c2fc2-7e6a-490b-9023-f98f2b9bfd89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f9970a-nbr54" Oct 30 16:47:08.953035 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:08.953011 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f9970a-nbr54" Oct 30 16:47:09.260588 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:09.260560 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f9970a-nbr54"] Oct 30 16:47:09.263566 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:47:09.263537 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc89c2fc2_7e6a_490b_9023_f98f2b9bfd89.slice/crio-ca6cb92e6ba92fefe5f35525b9a1813c3b5b971cd27962b9d63d4382878e629c WatchSource:0}: Error finding container ca6cb92e6ba92fefe5f35525b9a1813c3b5b971cd27962b9d63d4382878e629c: Status 404 returned error can't find the container with id ca6cb92e6ba92fefe5f35525b9a1813c3b5b971cd27962b9d63d4382878e629c Oct 30 16:47:10.056666 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:10.056627 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f9970a-nbr54" event={"ID":"c89c2fc2-7e6a-490b-9023-f98f2b9bfd89","Type":"ContainerStarted","Data":"d9873cac7ea910c33f169de39ce0609be0169e0a1db1122446be7020a2c61579"} Oct 30 16:47:10.057137 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:10.056685 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f9970a-nbr54" event={"ID":"c89c2fc2-7e6a-490b-9023-f98f2b9bfd89","Type":"ContainerStarted","Data":"ca6cb92e6ba92fefe5f35525b9a1813c3b5b971cd27962b9d63d4382878e629c"} Oct 30 16:47:10.094271 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:10.094222 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f9970a-nbr54" podStartSLOduration=2.094206536 podStartE2EDuration="2.094206536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:47:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:47:10.092455921 +0000 UTC m=+1457.319631020" watchObservedRunningTime="2026-10-30 16:47:10.094206536 +0000 UTC m=+1457.321381635" Oct 30 16:47:11.081922 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:11.081886 4493 generic.go:356] "Generic (PLEG): container finished" podID="c89c2fc2-7e6a-490b-9023-f98f2b9bfd89" containerID="d9873cac7ea910c33f169de39ce0609be0169e0a1db1122446be7020a2c61579" exitCode=0 Oct 30 16:47:11.082319 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:11.081988 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f9970a-nbr54" event={"ID":"c89c2fc2-7e6a-490b-9023-f98f2b9bfd89","Type":"ContainerDied","Data":"d9873cac7ea910c33f169de39ce0609be0169e0a1db1122446be7020a2c61579"} Oct 30 16:47:11.648964 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:11.648927 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-dc1ea0-qpwj2"] Oct 30 16:47:11.655538 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:11.655506 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-dc1ea0-qpwj2" Oct 30 16:47:11.679125 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:11.679096 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-dc1ea0-qpwj2"] Oct 30 16:47:11.793293 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:11.793258 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c92029b3-a56d-4b96-8fba-f0a2c03fa579-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-dc1ea0-qpwj2\" (UID: \"c92029b3-a56d-4b96-8fba-f0a2c03fa579\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-dc1ea0-qpwj2" Oct 30 16:47:11.793488 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:11.793321 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td27z\" (UniqueName: \"kubernetes.io/projected/c92029b3-a56d-4b96-8fba-f0a2c03fa579-kube-api-access-td27z\") pod \"maas-api-revoke-keys-e2e-ipp-b-dc1ea0-qpwj2\" (UID: \"c92029b3-a56d-4b96-8fba-f0a2c03fa579\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-dc1ea0-qpwj2" Oct 30 16:47:11.793668 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:11.793650 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c92029b3-a56d-4b96-8fba-f0a2c03fa579-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-dc1ea0-qpwj2\" (UID: \"c92029b3-a56d-4b96-8fba-f0a2c03fa579\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-dc1ea0-qpwj2" Oct 30 16:47:11.894670 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:11.894635 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c92029b3-a56d-4b96-8fba-f0a2c03fa579-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-dc1ea0-qpwj2\" (UID: \"c92029b3-a56d-4b96-8fba-f0a2c03fa579\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-dc1ea0-qpwj2" Oct 30 16:47:11.894894 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:11.894682 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c92029b3-a56d-4b96-8fba-f0a2c03fa579-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-dc1ea0-qpwj2\" (UID: \"c92029b3-a56d-4b96-8fba-f0a2c03fa579\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-dc1ea0-qpwj2" Oct 30 16:47:11.894894 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:11.894709 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td27z\" (UniqueName: \"kubernetes.io/projected/c92029b3-a56d-4b96-8fba-f0a2c03fa579-kube-api-access-td27z\") pod \"maas-api-revoke-keys-e2e-ipp-b-dc1ea0-qpwj2\" (UID: \"c92029b3-a56d-4b96-8fba-f0a2c03fa579\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-dc1ea0-qpwj2" Oct 30 16:47:11.895516 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:11.895489 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c92029b3-a56d-4b96-8fba-f0a2c03fa579-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-dc1ea0-qpwj2\" (UID: \"c92029b3-a56d-4b96-8fba-f0a2c03fa579\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-dc1ea0-qpwj2" Oct 30 16:47:11.897664 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:11.897635 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c92029b3-a56d-4b96-8fba-f0a2c03fa579-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-dc1ea0-qpwj2\" (UID: \"c92029b3-a56d-4b96-8fba-f0a2c03fa579\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-dc1ea0-qpwj2" Oct 30 16:47:11.925284 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:11.925209 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4a4c68-54fd7c68d8-g96m2"] Oct 30 16:47:11.925545 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:11.925514 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-4a4c68-54fd7c68d8-g96m2" podUID="8e2b6cc4-8cfe-4cf3-b221-fc948169dd31" containerName="payload-processing" containerID="cri-o://2df3dae036ae03f8acf759fd3f1b32af478529f9adbb74111dad50ac9373260b" gracePeriod=30 Oct 30 16:47:11.926899 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:11.926608 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-td27z\" (UniqueName: \"kubernetes.io/projected/c92029b3-a56d-4b96-8fba-f0a2c03fa579-kube-api-access-td27z\") pod \"maas-api-revoke-keys-e2e-ipp-b-dc1ea0-qpwj2\" (UID: \"c92029b3-a56d-4b96-8fba-f0a2c03fa579\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-dc1ea0-qpwj2" Oct 30 16:47:11.973157 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:11.973114 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-dc1ea0-qpwj2" Oct 30 16:47:12.133250 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:12.132122 4493 generic.go:356] "Generic (PLEG): container finished" podID="8e2b6cc4-8cfe-4cf3-b221-fc948169dd31" containerID="2df3dae036ae03f8acf759fd3f1b32af478529f9adbb74111dad50ac9373260b" exitCode=0 Oct 30 16:47:12.133250 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:12.132369 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4a4c68-54fd7c68d8-g96m2" event={"ID":"8e2b6cc4-8cfe-4cf3-b221-fc948169dd31","Type":"ContainerDied","Data":"2df3dae036ae03f8acf759fd3f1b32af478529f9adbb74111dad50ac9373260b"} Oct 30 16:47:12.444511 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:12.444473 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4a4c68-54fd7c68d8-g96m2" Oct 30 16:47:12.450306 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:12.450030 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-dc1ea0-qpwj2"] Oct 30 16:47:12.552432 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:12.552405 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f9970a-nbr54" Oct 30 16:47:12.564541 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:12.564515 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c89c2fc2-7e6a-490b-9023-f98f2b9bfd89-kube-api-access-rlrs4\" (UniqueName: \"kubernetes.io/projected/c89c2fc2-7e6a-490b-9023-f98f2b9bfd89-kube-api-access-rlrs4\") pod \"c89c2fc2-7e6a-490b-9023-f98f2b9bfd89\" (UID: \"c89c2fc2-7e6a-490b-9023-f98f2b9bfd89\") " Oct 30 16:47:12.564658 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:12.564549 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c89c2fc2-7e6a-490b-9023-f98f2b9bfd89-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c89c2fc2-7e6a-490b-9023-f98f2b9bfd89-maas-api-openshift-ca\") pod \"c89c2fc2-7e6a-490b-9023-f98f2b9bfd89\" (UID: \"c89c2fc2-7e6a-490b-9023-f98f2b9bfd89\") " Oct 30 16:47:12.564658 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:12.564612 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e2b6cc4-8cfe-4cf3-b221-fc948169dd31-kube-api-access-dstxk\" (UniqueName: \"kubernetes.io/projected/8e2b6cc4-8cfe-4cf3-b221-fc948169dd31-kube-api-access-dstxk\") pod \"8e2b6cc4-8cfe-4cf3-b221-fc948169dd31\" (UID: \"8e2b6cc4-8cfe-4cf3-b221-fc948169dd31\") " Oct 30 16:47:12.564758 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:12.564666 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c89c2fc2-7e6a-490b-9023-f98f2b9bfd89-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c89c2fc2-7e6a-490b-9023-f98f2b9bfd89-maas-api-xks-ca\") pod \"c89c2fc2-7e6a-490b-9023-f98f2b9bfd89\" (UID: \"c89c2fc2-7e6a-490b-9023-f98f2b9bfd89\") " Oct 30 16:47:12.564845 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:12.564778 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e2b6cc4-8cfe-4cf3-b221-fc948169dd31-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e2b6cc4-8cfe-4cf3-b221-fc948169dd31-plugins-config-volume\") pod \"8e2b6cc4-8cfe-4cf3-b221-fc948169dd31\" (UID: \"8e2b6cc4-8cfe-4cf3-b221-fc948169dd31\") " Oct 30 16:47:12.564972 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:12.564946 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c89c2fc2-7e6a-490b-9023-f98f2b9bfd89-maas-api-openshift-ca" pod "c89c2fc2-7e6a-490b-9023-f98f2b9bfd89" (UID: "c89c2fc2-7e6a-490b-9023-f98f2b9bfd89"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:47:12.565268 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:12.565241 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e2b6cc4-8cfe-4cf3-b221-fc948169dd31-plugins-config-volume" pod "8e2b6cc4-8cfe-4cf3-b221-fc948169dd31" (UID: "8e2b6cc4-8cfe-4cf3-b221-fc948169dd31"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:47:12.565595 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:12.565574 4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e2b6cc4-8cfe-4cf3-b221-fc948169dd31-plugins-config-volume\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:47:12.565668 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:12.565601 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c89c2fc2-7e6a-490b-9023-f98f2b9bfd89-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:47:12.567046 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:12.567026 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c89c2fc2-7e6a-490b-9023-f98f2b9bfd89-kube-api-access-rlrs4" pod "c89c2fc2-7e6a-490b-9023-f98f2b9bfd89" (UID: "c89c2fc2-7e6a-490b-9023-f98f2b9bfd89"). InnerVolumeSpecName "kube-api-access-rlrs4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:47:12.567046 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:12.567038 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c89c2fc2-7e6a-490b-9023-f98f2b9bfd89-maas-api-xks-ca" pod "c89c2fc2-7e6a-490b-9023-f98f2b9bfd89" (UID: "c89c2fc2-7e6a-490b-9023-f98f2b9bfd89"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:47:12.567164 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:12.567097 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e2b6cc4-8cfe-4cf3-b221-fc948169dd31-kube-api-access-dstxk" pod "8e2b6cc4-8cfe-4cf3-b221-fc948169dd31" (UID: "8e2b6cc4-8cfe-4cf3-b221-fc948169dd31"). InnerVolumeSpecName "kube-api-access-dstxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:47:12.667293 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:12.667260 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dstxk\" (UniqueName: \"kubernetes.io/projected/8e2b6cc4-8cfe-4cf3-b221-fc948169dd31-kube-api-access-dstxk\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:47:12.667293 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:12.667289 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c89c2fc2-7e6a-490b-9023-f98f2b9bfd89-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:47:12.667293 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:12.667299 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlrs4\" (UniqueName: \"kubernetes.io/projected/c89c2fc2-7e6a-490b-9023-f98f2b9bfd89-kube-api-access-rlrs4\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:47:12.740430 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:12.740396 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4a4c68-km6j4"] Oct 30 16:47:12.746116 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:12.746085 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-4a4c68-km6j4"] Oct 30 16:47:12.947930 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:12.947897 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="714be7d8-f0ae-40d4-91ef-b8743f7912fd" path="/var/lib/kubelet/pods/714be7d8-f0ae-40d4-91ef-b8743f7912fd/volumes" Oct 30 16:47:13.143595 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:13.143568 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f9970a-nbr54" Oct 30 16:47:13.144093 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:13.143597 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f9970a-nbr54" event={"ID":"c89c2fc2-7e6a-490b-9023-f98f2b9bfd89","Type":"ContainerDied","Data":"ca6cb92e6ba92fefe5f35525b9a1813c3b5b971cd27962b9d63d4382878e629c"} Oct 30 16:47:13.144093 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:13.143654 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca6cb92e6ba92fefe5f35525b9a1813c3b5b971cd27962b9d63d4382878e629c" Oct 30 16:47:13.145462 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:13.145431 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-4a4c68-54fd7c68d8-g96m2" Oct 30 16:47:13.145624 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:13.145474 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-4a4c68-54fd7c68d8-g96m2" event={"ID":"8e2b6cc4-8cfe-4cf3-b221-fc948169dd31","Type":"ContainerDied","Data":"3f5914883a1939325821f0a189d61a11b7dfb3d10432b921d4f5cad49dbc008c"} Oct 30 16:47:13.145624 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:13.145605 4493 scope.go:122] "RemoveContainer" containerID="2df3dae036ae03f8acf759fd3f1b32af478529f9adbb74111dad50ac9373260b" Oct 30 16:47:13.148849 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:13.148820 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-dc1ea0-qpwj2" event={"ID":"c92029b3-a56d-4b96-8fba-f0a2c03fa579","Type":"ContainerStarted","Data":"4fc9d519c010f36746c0a8cea13fce07edfb89a83c1fac078e468f390ac3409e"} Oct 30 16:47:13.148956 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:13.148874 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-dc1ea0-qpwj2" event={"ID":"c92029b3-a56d-4b96-8fba-f0a2c03fa579","Type":"ContainerStarted","Data":"69228881ccfbf8bb87a319bc5e961110d5f1c2c97912997f76fe4752a8195619"} Oct 30 16:47:13.198085 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:13.198040 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-dc1ea0-qpwj2" podStartSLOduration=2.198028452 podStartE2EDuration="2.198028452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:47:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:47:13.19702673 +0000 UTC m=+1460.424201868" watchObservedRunningTime="2026-10-30 16:47:13.198028452 +0000 UTC m=+1460.425203552" Oct 30 16:47:13.255027 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:13.254774 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4a4c68-54fd7c68d8-g96m2"] Oct 30 16:47:13.262933 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:13.262884 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-4a4c68-54fd7c68d8-g96m2"] Oct 30 16:47:14.178361 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:14.178326 4493 generic.go:356] "Generic (PLEG): container finished" podID="c92029b3-a56d-4b96-8fba-f0a2c03fa579" containerID="4fc9d519c010f36746c0a8cea13fce07edfb89a83c1fac078e468f390ac3409e" exitCode=0 Oct 30 16:47:14.178812 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:14.178406 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-dc1ea0-qpwj2" event={"ID":"c92029b3-a56d-4b96-8fba-f0a2c03fa579","Type":"ContainerDied","Data":"4fc9d519c010f36746c0a8cea13fce07edfb89a83c1fac078e468f390ac3409e"} Oct 30 16:47:14.956353 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:14.956313 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e2b6cc4-8cfe-4cf3-b221-fc948169dd31" path="/var/lib/kubelet/pods/8e2b6cc4-8cfe-4cf3-b221-fc948169dd31/volumes" Oct 30 16:47:15.345764 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:15.345742 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-dc1ea0-qpwj2" Oct 30 16:47:15.418992 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:15.418953 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c92029b3-a56d-4b96-8fba-f0a2c03fa579-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c92029b3-a56d-4b96-8fba-f0a2c03fa579-maas-api-xks-ca\") pod \"c92029b3-a56d-4b96-8fba-f0a2c03fa579\" (UID: \"c92029b3-a56d-4b96-8fba-f0a2c03fa579\") " Oct 30 16:47:15.419195 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:15.419173 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c92029b3-a56d-4b96-8fba-f0a2c03fa579-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c92029b3-a56d-4b96-8fba-f0a2c03fa579-maas-api-openshift-ca\") pod \"c92029b3-a56d-4b96-8fba-f0a2c03fa579\" (UID: \"c92029b3-a56d-4b96-8fba-f0a2c03fa579\") " Oct 30 16:47:15.419317 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:15.419295 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c92029b3-a56d-4b96-8fba-f0a2c03fa579-kube-api-access-td27z\" (UniqueName: \"kubernetes.io/projected/c92029b3-a56d-4b96-8fba-f0a2c03fa579-kube-api-access-td27z\") pod \"c92029b3-a56d-4b96-8fba-f0a2c03fa579\" (UID: \"c92029b3-a56d-4b96-8fba-f0a2c03fa579\") " Oct 30 16:47:15.419757 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:15.419692 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c92029b3-a56d-4b96-8fba-f0a2c03fa579-maas-api-openshift-ca" pod "c92029b3-a56d-4b96-8fba-f0a2c03fa579" (UID: "c92029b3-a56d-4b96-8fba-f0a2c03fa579"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:47:15.422188 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:15.422152 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c92029b3-a56d-4b96-8fba-f0a2c03fa579-maas-api-xks-ca" pod "c92029b3-a56d-4b96-8fba-f0a2c03fa579" (UID: "c92029b3-a56d-4b96-8fba-f0a2c03fa579"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:47:15.424273 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:15.424240 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c92029b3-a56d-4b96-8fba-f0a2c03fa579-kube-api-access-td27z" pod "c92029b3-a56d-4b96-8fba-f0a2c03fa579" (UID: "c92029b3-a56d-4b96-8fba-f0a2c03fa579"). InnerVolumeSpecName "kube-api-access-td27z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:47:15.521311 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:15.521236 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c92029b3-a56d-4b96-8fba-f0a2c03fa579-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:47:15.521311 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:15.521267 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-td27z\" (UniqueName: \"kubernetes.io/projected/c92029b3-a56d-4b96-8fba-f0a2c03fa579-kube-api-access-td27z\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:47:15.521311 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:15.521277 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c92029b3-a56d-4b96-8fba-f0a2c03fa579-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:47:16.197378 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:16.197347 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-dc1ea0-qpwj2" Oct 30 16:47:16.197572 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:16.197389 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-dc1ea0-qpwj2" event={"ID":"c92029b3-a56d-4b96-8fba-f0a2c03fa579","Type":"ContainerDied","Data":"69228881ccfbf8bb87a319bc5e961110d5f1c2c97912997f76fe4752a8195619"} Oct 30 16:47:16.197572 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:16.197437 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69228881ccfbf8bb87a319bc5e961110d5f1c2c97912997f76fe4752a8195619" Oct 30 16:47:18.266762 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:18.266719 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7c23a4-zkbn9"] Oct 30 16:47:18.274652 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:18.274622 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7c23a4-zkbn9"] Oct 30 16:47:18.935398 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:18.935362 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7feaf66b-d3c1-4e24-b006-6104ceb28a65" path="/var/lib/kubelet/pods/7feaf66b-d3c1-4e24-b006-6104ceb28a65/volumes" Oct 30 16:47:26.646817 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:26.646769 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-64c409-86xd9"] Oct 30 16:47:26.647381 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:26.647366 4493 cpu_manager.go:414] "containerMap: removing container" podUID="c89c2fc2-7e6a-490b-9023-f98f2b9bfd89" containerName="revoke-keys" Oct 30 16:47:26.647426 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:26.647383 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c89c2fc2-7e6a-490b-9023-f98f2b9bfd89" containerName="revoke-keys" Oct 30 16:47:26.647426 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:26.647401 4493 cpu_manager.go:414] "containerMap: removing container" podUID="8e2b6cc4-8cfe-4cf3-b221-fc948169dd31" containerName="payload-processing" Oct 30 16:47:26.647426 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:26.647406 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e2b6cc4-8cfe-4cf3-b221-fc948169dd31" containerName="payload-processing" Oct 30 16:47:26.647519 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:26.647433 4493 cpu_manager.go:414] "containerMap: removing container" podUID="c92029b3-a56d-4b96-8fba-f0a2c03fa579" containerName="revoke-keys" Oct 30 16:47:26.647519 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:26.647438 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c92029b3-a56d-4b96-8fba-f0a2c03fa579" containerName="revoke-keys" Oct 30 16:47:26.647585 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:26.647543 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="c92029b3-a56d-4b96-8fba-f0a2c03fa579" containerName="revoke-keys" Oct 30 16:47:26.647585 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:26.647553 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="c89c2fc2-7e6a-490b-9023-f98f2b9bfd89" containerName="revoke-keys" Oct 30 16:47:26.647585 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:26.647560 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e2b6cc4-8cfe-4cf3-b221-fc948169dd31" containerName="payload-processing" Oct 30 16:47:26.652057 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:26.652037 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-64c409-86xd9" Oct 30 16:47:26.664865 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:26.664834 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:47:26.664999 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:26.664879 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:47:26.674731 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:26.674703 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:47:26.674731 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:26.674723 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 16:47:26.678423 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:26.678399 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-64c409-86xd9"] Oct 30 16:47:26.753513 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:26.753479 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90841876-5f2a-47e9-8b7c-b5731a8c49cf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-64c409-86xd9\" (UID: \"90841876-5f2a-47e9-8b7c-b5731a8c49cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-64c409-86xd9" Oct 30 16:47:26.753759 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:26.753732 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkmhh\" (UniqueName: \"kubernetes.io/projected/90841876-5f2a-47e9-8b7c-b5731a8c49cf-kube-api-access-pkmhh\") pod \"maas-api-revoke-keys-e2e-ipp-b-64c409-86xd9\" (UID: \"90841876-5f2a-47e9-8b7c-b5731a8c49cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-64c409-86xd9" Oct 30 16:47:26.753856 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:26.753835 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90841876-5f2a-47e9-8b7c-b5731a8c49cf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-64c409-86xd9\" (UID: \"90841876-5f2a-47e9-8b7c-b5731a8c49cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-64c409-86xd9" Oct 30 16:47:26.855145 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:26.855105 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90841876-5f2a-47e9-8b7c-b5731a8c49cf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-64c409-86xd9\" (UID: \"90841876-5f2a-47e9-8b7c-b5731a8c49cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-64c409-86xd9" Oct 30 16:47:26.855349 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:26.855266 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkmhh\" (UniqueName: \"kubernetes.io/projected/90841876-5f2a-47e9-8b7c-b5731a8c49cf-kube-api-access-pkmhh\") pod \"maas-api-revoke-keys-e2e-ipp-b-64c409-86xd9\" (UID: \"90841876-5f2a-47e9-8b7c-b5731a8c49cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-64c409-86xd9" Oct 30 16:47:26.855349 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:26.855331 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90841876-5f2a-47e9-8b7c-b5731a8c49cf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-64c409-86xd9\" (UID: \"90841876-5f2a-47e9-8b7c-b5731a8c49cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-64c409-86xd9" Oct 30 16:47:26.855997 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:26.855967 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90841876-5f2a-47e9-8b7c-b5731a8c49cf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-64c409-86xd9\" (UID: \"90841876-5f2a-47e9-8b7c-b5731a8c49cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-64c409-86xd9" Oct 30 16:47:26.858138 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:26.858111 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90841876-5f2a-47e9-8b7c-b5731a8c49cf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-64c409-86xd9\" (UID: \"90841876-5f2a-47e9-8b7c-b5731a8c49cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-64c409-86xd9" Oct 30 16:47:26.878207 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:26.878173 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkmhh\" (UniqueName: \"kubernetes.io/projected/90841876-5f2a-47e9-8b7c-b5731a8c49cf-kube-api-access-pkmhh\") pod \"maas-api-revoke-keys-e2e-ipp-b-64c409-86xd9\" (UID: \"90841876-5f2a-47e9-8b7c-b5731a8c49cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-64c409-86xd9" Oct 30 16:47:26.968434 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:26.968021 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-64c409-86xd9" Oct 30 16:47:27.340921 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:27.331734 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-64c409-86xd9"] Oct 30 16:47:27.349187 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:47:27.345027 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90841876_5f2a_47e9_8b7c_b5731a8c49cf.slice/crio-2c0c65aec0106251c149a75579d493adeb1449b1405a81e3029b17d6f4a3dae4 WatchSource:0}: Error finding container 2c0c65aec0106251c149a75579d493adeb1449b1405a81e3029b17d6f4a3dae4: Status 404 returned error can't find the container with id 2c0c65aec0106251c149a75579d493adeb1449b1405a81e3029b17d6f4a3dae4 Oct 30 16:47:27.692095 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:27.690104 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-dc1ea0-qpwj2"] Oct 30 16:47:27.696335 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:27.696300 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-dc1ea0-qpwj2"] Oct 30 16:47:28.362704 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:28.362662 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-64c409-86xd9" event={"ID":"90841876-5f2a-47e9-8b7c-b5731a8c49cf","Type":"ContainerStarted","Data":"b87a436b329c5eaeca15c7101851d4cd8ef395e45f78c1a653f1d3f2e8a2f2fe"} Oct 30 16:47:28.362889 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:28.362736 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-64c409-86xd9" event={"ID":"90841876-5f2a-47e9-8b7c-b5731a8c49cf","Type":"ContainerStarted","Data":"2c0c65aec0106251c149a75579d493adeb1449b1405a81e3029b17d6f4a3dae4"} Oct 30 16:47:28.934492 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:28.934456 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c92029b3-a56d-4b96-8fba-f0a2c03fa579" path="/var/lib/kubelet/pods/c92029b3-a56d-4b96-8fba-f0a2c03fa579/volumes" Oct 30 16:47:29.371400 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:29.371308 4493 generic.go:356] "Generic (PLEG): container finished" podID="90841876-5f2a-47e9-8b7c-b5731a8c49cf" containerID="b87a436b329c5eaeca15c7101851d4cd8ef395e45f78c1a653f1d3f2e8a2f2fe" exitCode=0 Oct 30 16:47:29.371571 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:29.371399 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-64c409-86xd9" event={"ID":"90841876-5f2a-47e9-8b7c-b5731a8c49cf","Type":"ContainerDied","Data":"b87a436b329c5eaeca15c7101851d4cd8ef395e45f78c1a653f1d3f2e8a2f2fe"} Oct 30 16:47:29.543338 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:29.543312 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-64c409-86xd9" Oct 30 16:47:29.631885 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:29.631775 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90841876-5f2a-47e9-8b7c-b5731a8c49cf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90841876-5f2a-47e9-8b7c-b5731a8c49cf-maas-api-xks-ca\") pod \"90841876-5f2a-47e9-8b7c-b5731a8c49cf\" (UID: \"90841876-5f2a-47e9-8b7c-b5731a8c49cf\") " Oct 30 16:47:29.632173 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:29.632151 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90841876-5f2a-47e9-8b7c-b5731a8c49cf-kube-api-access-pkmhh\" (UniqueName: \"kubernetes.io/projected/90841876-5f2a-47e9-8b7c-b5731a8c49cf-kube-api-access-pkmhh\") pod \"90841876-5f2a-47e9-8b7c-b5731a8c49cf\" (UID: \"90841876-5f2a-47e9-8b7c-b5731a8c49cf\") " Oct 30 16:47:29.632251 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:29.632200 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90841876-5f2a-47e9-8b7c-b5731a8c49cf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90841876-5f2a-47e9-8b7c-b5731a8c49cf-maas-api-openshift-ca\") pod \"90841876-5f2a-47e9-8b7c-b5731a8c49cf\" (UID: \"90841876-5f2a-47e9-8b7c-b5731a8c49cf\") " Oct 30 16:47:29.632831 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:29.632765 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90841876-5f2a-47e9-8b7c-b5731a8c49cf-maas-api-openshift-ca" pod "90841876-5f2a-47e9-8b7c-b5731a8c49cf" (UID: "90841876-5f2a-47e9-8b7c-b5731a8c49cf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:47:29.634103 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:29.634077 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90841876-5f2a-47e9-8b7c-b5731a8c49cf-maas-api-xks-ca" pod "90841876-5f2a-47e9-8b7c-b5731a8c49cf" (UID: "90841876-5f2a-47e9-8b7c-b5731a8c49cf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:47:29.634562 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:29.634539 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90841876-5f2a-47e9-8b7c-b5731a8c49cf-kube-api-access-pkmhh" pod "90841876-5f2a-47e9-8b7c-b5731a8c49cf" (UID: "90841876-5f2a-47e9-8b7c-b5731a8c49cf"). InnerVolumeSpecName "kube-api-access-pkmhh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:47:29.733888 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:29.733855 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90841876-5f2a-47e9-8b7c-b5731a8c49cf-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:47:29.733888 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:29.733881 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkmhh\" (UniqueName: \"kubernetes.io/projected/90841876-5f2a-47e9-8b7c-b5731a8c49cf-kube-api-access-pkmhh\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:47:29.733888 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:29.733893 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90841876-5f2a-47e9-8b7c-b5731a8c49cf-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:47:30.342238 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:47:30.342198 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 16:47:30.379554 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:30.379531 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-64c409-86xd9" Oct 30 16:47:30.379705 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:30.379610 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-64c409-86xd9" event={"ID":"90841876-5f2a-47e9-8b7c-b5731a8c49cf","Type":"ContainerDied","Data":"2c0c65aec0106251c149a75579d493adeb1449b1405a81e3029b17d6f4a3dae4"} Oct 30 16:47:30.379705 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:30.379663 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c0c65aec0106251c149a75579d493adeb1449b1405a81e3029b17d6f4a3dae4" Oct 30 16:47:38.334496 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:38.334387 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 16:47:46.033186 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:46.033089 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e22aa1c8-7bj5d"] Oct 30 16:47:46.044166 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:46.044134 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e22aa1c8-7bj5d"] Oct 30 16:47:46.933562 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:46.933525 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4487028d-34c7-4bee-abd2-28cc0a3e6b51" path="/var/lib/kubelet/pods/4487028d-34c7-4bee-abd2-28cc0a3e6b51/volumes" Oct 30 16:47:51.107630 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:51.107504 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-64c409-75bfc9f544-l6lpj"] Oct 30 16:47:51.108175 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:51.107807 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-64c409-75bfc9f544-l6lpj" podUID="6c1ea064-422e-49c7-9577-448806fc5e33" containerName="payload-processing" containerID="cri-o://7e6916b16339075bacda8b732614eaca0e7cdaf56a33ad050b85ee5096db5ba3" gracePeriod=30 Oct 30 16:47:51.137315 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:51.137279 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-64c409-54c8f8f675-cmds8"] Oct 30 16:47:51.137766 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:51.137714 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-64c409-54c8f8f675-cmds8" podUID="e6cd9cc1-7206-4868-908a-05aed37aae54" containerName="payload-pre-processing" containerID="cri-o://2300b1b99387b2dde1ecd9a10bfa6a93a52272763daaa8f23c426c5e49c5bdfe" gracePeriod=30 Oct 30 16:47:51.634324 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:51.634197 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-64c409-75bfc9f544-l6lpj" Oct 30 16:47:51.636641 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:51.636615 4493 generic.go:356] "Generic (PLEG): container finished" podID="e6cd9cc1-7206-4868-908a-05aed37aae54" containerID="2300b1b99387b2dde1ecd9a10bfa6a93a52272763daaa8f23c426c5e49c5bdfe" exitCode=0 Oct 30 16:47:51.636837 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:51.636687 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-64c409-54c8f8f675-cmds8" event={"ID":"e6cd9cc1-7206-4868-908a-05aed37aae54","Type":"ContainerDied","Data":"2300b1b99387b2dde1ecd9a10bfa6a93a52272763daaa8f23c426c5e49c5bdfe"} Oct 30 16:47:51.638625 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:51.638598 4493 generic.go:356] "Generic (PLEG): container finished" podID="6c1ea064-422e-49c7-9577-448806fc5e33" containerID="7e6916b16339075bacda8b732614eaca0e7cdaf56a33ad050b85ee5096db5ba3" exitCode=0 Oct 30 16:47:51.638752 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:51.638631 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-64c409-75bfc9f544-l6lpj" Oct 30 16:47:51.638752 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:51.638670 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-64c409-75bfc9f544-l6lpj" event={"ID":"6c1ea064-422e-49c7-9577-448806fc5e33","Type":"ContainerDied","Data":"7e6916b16339075bacda8b732614eaca0e7cdaf56a33ad050b85ee5096db5ba3"} Oct 30 16:47:51.638752 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:51.638713 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-64c409-75bfc9f544-l6lpj" event={"ID":"6c1ea064-422e-49c7-9577-448806fc5e33","Type":"ContainerDied","Data":"2a1b7edfd841af3212768c4f377610066f4484c43c7422ba4d6d71059467bb79"} Oct 30 16:47:51.638752 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:51.638727 4493 scope.go:122] "RemoveContainer" containerID="7e6916b16339075bacda8b732614eaca0e7cdaf56a33ad050b85ee5096db5ba3" Oct 30 16:47:51.651597 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:51.651575 4493 scope.go:122] "RemoveContainer" containerID="7e6916b16339075bacda8b732614eaca0e7cdaf56a33ad050b85ee5096db5ba3" Oct 30 16:47:51.651966 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:47:51.651940 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e6916b16339075bacda8b732614eaca0e7cdaf56a33ad050b85ee5096db5ba3\": container with ID starting with 7e6916b16339075bacda8b732614eaca0e7cdaf56a33ad050b85ee5096db5ba3 not found: ID does not exist" containerID="7e6916b16339075bacda8b732614eaca0e7cdaf56a33ad050b85ee5096db5ba3" Oct 30 16:47:51.652085 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:51.651975 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e6916b16339075bacda8b732614eaca0e7cdaf56a33ad050b85ee5096db5ba3"} err="failed to get container status \"7e6916b16339075bacda8b732614eaca0e7cdaf56a33ad050b85ee5096db5ba3\": rpc error: code = NotFound desc = could not find container \"7e6916b16339075bacda8b732614eaca0e7cdaf56a33ad050b85ee5096db5ba3\": container with ID starting with 7e6916b16339075bacda8b732614eaca0e7cdaf56a33ad050b85ee5096db5ba3 not found: ID does not exist" Oct 30 16:47:51.677292 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:51.677263 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-64c409-54c8f8f675-cmds8" Oct 30 16:47:51.693193 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:51.693160 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c1ea064-422e-49c7-9577-448806fc5e33-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c1ea064-422e-49c7-9577-448806fc5e33-plugins-config-volume\") pod \"6c1ea064-422e-49c7-9577-448806fc5e33\" (UID: \"6c1ea064-422e-49c7-9577-448806fc5e33\") " Oct 30 16:47:51.693616 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:51.693375 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c1ea064-422e-49c7-9577-448806fc5e33-kube-api-access-lvjw9\" (UniqueName: \"kubernetes.io/projected/6c1ea064-422e-49c7-9577-448806fc5e33-kube-api-access-lvjw9\") pod \"6c1ea064-422e-49c7-9577-448806fc5e33\" (UID: \"6c1ea064-422e-49c7-9577-448806fc5e33\") " Oct 30 16:47:51.694463 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:51.694433 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c1ea064-422e-49c7-9577-448806fc5e33-plugins-config-volume" pod "6c1ea064-422e-49c7-9577-448806fc5e33" (UID: "6c1ea064-422e-49c7-9577-448806fc5e33"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:47:51.697730 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:51.697705 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c1ea064-422e-49c7-9577-448806fc5e33-kube-api-access-lvjw9" pod "6c1ea064-422e-49c7-9577-448806fc5e33" (UID: "6c1ea064-422e-49c7-9577-448806fc5e33"). InnerVolumeSpecName "kube-api-access-lvjw9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:47:51.794389 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:51.794351 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6cd9cc1-7206-4868-908a-05aed37aae54-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6cd9cc1-7206-4868-908a-05aed37aae54-plugins-config-volume\") pod \"e6cd9cc1-7206-4868-908a-05aed37aae54\" (UID: \"e6cd9cc1-7206-4868-908a-05aed37aae54\") " Oct 30 16:47:51.794813 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:51.794771 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6cd9cc1-7206-4868-908a-05aed37aae54-kube-api-access-pl58d\" (UniqueName: \"kubernetes.io/projected/e6cd9cc1-7206-4868-908a-05aed37aae54-kube-api-access-pl58d\") pod \"e6cd9cc1-7206-4868-908a-05aed37aae54\" (UID: \"e6cd9cc1-7206-4868-908a-05aed37aae54\") " Oct 30 16:47:51.794940 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:51.794875 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6cd9cc1-7206-4868-908a-05aed37aae54-plugins-config-volume" pod "e6cd9cc1-7206-4868-908a-05aed37aae54" (UID: "e6cd9cc1-7206-4868-908a-05aed37aae54"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:47:51.795190 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:51.795172 4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c1ea064-422e-49c7-9577-448806fc5e33-plugins-config-volume\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:47:51.795258 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:51.795190 4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6cd9cc1-7206-4868-908a-05aed37aae54-plugins-config-volume\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:47:51.795258 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:51.795200 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvjw9\" (UniqueName: \"kubernetes.io/projected/6c1ea064-422e-49c7-9577-448806fc5e33-kube-api-access-lvjw9\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:47:51.797184 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:51.797162 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6cd9cc1-7206-4868-908a-05aed37aae54-kube-api-access-pl58d" pod "e6cd9cc1-7206-4868-908a-05aed37aae54" (UID: "e6cd9cc1-7206-4868-908a-05aed37aae54"). InnerVolumeSpecName "kube-api-access-pl58d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:47:51.855456 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:51.855408 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-64c409-86xd9"] Oct 30 16:47:51.858847 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:51.858820 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-64c409-86xd9"] Oct 30 16:47:51.897069 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:51.897036 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pl58d\" (UniqueName: \"kubernetes.io/projected/e6cd9cc1-7206-4868-908a-05aed37aae54-kube-api-access-pl58d\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:47:52.004095 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:52.004066 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-64c409-75bfc9f544-l6lpj"] Oct 30 16:47:52.011860 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:52.011832 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-64c409-75bfc9f544-l6lpj"] Oct 30 16:47:52.647456 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:52.647428 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-64c409-54c8f8f675-cmds8" Oct 30 16:47:52.648002 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:52.647462 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-64c409-54c8f8f675-cmds8" event={"ID":"e6cd9cc1-7206-4868-908a-05aed37aae54","Type":"ContainerDied","Data":"7f26a8a386439f72567d64f34c223c55d9ad4345494ab9b2d6ef68c99e0758e7"} Oct 30 16:47:52.648002 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:52.647522 4493 scope.go:122] "RemoveContainer" containerID="2300b1b99387b2dde1ecd9a10bfa6a93a52272763daaa8f23c426c5e49c5bdfe" Oct 30 16:47:52.694559 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:52.694531 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-64c409-54c8f8f675-cmds8"] Oct 30 16:47:52.699484 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:52.699459 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-64c409-54c8f8f675-cmds8"] Oct 30 16:47:52.934723 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:52.934642 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c1ea064-422e-49c7-9577-448806fc5e33" path="/var/lib/kubelet/pods/6c1ea064-422e-49c7-9577-448806fc5e33/volumes" Oct 30 16:47:52.935127 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:52.935112 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90841876-5f2a-47e9-8b7c-b5731a8c49cf" path="/var/lib/kubelet/pods/90841876-5f2a-47e9-8b7c-b5731a8c49cf/volumes" Oct 30 16:47:52.935515 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:52.935500 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6cd9cc1-7206-4868-908a-05aed37aae54" path="/var/lib/kubelet/pods/e6cd9cc1-7206-4868-908a-05aed37aae54/volumes" Oct 30 16:47:53.493066 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:53.493041 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-56v9s_d3a1c417-2698-4462-b7ff-1d2e66d7a10f/ovn-acl-logging/0.log" Oct 30 16:47:53.547330 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:53.547304 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-56v9s_d3a1c417-2698-4462-b7ff-1d2e66d7a10f/ovn-acl-logging/0.log" Oct 30 16:47:54.285039 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:54.285008 4493 scope.go:122] "RemoveContainer" containerID="8ad5bc622b1639235757cb8e7e0848634750b8e103bfa77bcaae2439c2efd7ad" Oct 30 16:47:54.297674 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:54.297607 4493 scope.go:122] "RemoveContainer" containerID="eab562a57122e92a51f90c1da6a0a8eb2100f243cad1c3ef2917b90a74a7bf9b" Oct 30 16:47:54.309314 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:54.309273 4493 scope.go:122] "RemoveContainer" containerID="44aa3f71f557e86ab87a9047b80e37faf78e6154ad9a6f82fbd3ca159c75bfa1" Oct 30 16:47:54.319048 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:47:54.319028 4493 scope.go:122] "RemoveContainer" containerID="887dda815dd0e14d0c496d00c6038fb3b25facbf2e286af1238f99acbbabc6d7" Oct 30 16:48:00.343196 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:48:00.343156 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 16:48:00.486478 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:48:00.486444 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f9970a-nbr54"] Oct 30 16:48:00.496038 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:48:00.496008 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f9970a-nbr54"] Oct 30 16:48:00.839220 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:48:00.839174 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 16:48:00.944951 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:48:00.944916 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c89c2fc2-7e6a-490b-9023-f98f2b9bfd89" path="/var/lib/kubelet/pods/c89c2fc2-7e6a-490b-9023-f98f2b9bfd89/volumes" Oct 30 16:48:23.115209 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:48:23.115156 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 16:48:30.341314 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:48:30.341274 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 16:48:54.385009 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:48:54.384977 4493 scope.go:122] "RemoveContainer" containerID="b692ba76caaeba582441eb7f419abae2652ba0c53af07d8bef220dc04a90b04a" Oct 30 16:48:54.394914 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:48:54.394893 4493 scope.go:122] "RemoveContainer" containerID="e9c58f902473dcda44002369ebdee2bb11b263038dcf0ed4f4d0bf01a01de474" Oct 30 16:48:54.408009 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:48:54.407982 4493 scope.go:122] "RemoveContainer" containerID="dbd3646e32a56cea6902a1c6ca53ecf70d3018d6e0d211ee3368b003f30358a2" Oct 30 16:49:00.021131 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:00.021095 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 16:49:00.362272 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:49:00.362130 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 16:49:09.799980 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:09.799900 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-504442-fq8px"] Oct 30 16:49:09.800528 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:09.800513 4493 cpu_manager.go:414] "containerMap: removing container" podUID="90841876-5f2a-47e9-8b7c-b5731a8c49cf" containerName="revoke-keys" Oct 30 16:49:09.800576 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:09.800530 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90841876-5f2a-47e9-8b7c-b5731a8c49cf" containerName="revoke-keys" Oct 30 16:49:09.800576 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:09.800564 4493 cpu_manager.go:414] "containerMap: removing container" podUID="e6cd9cc1-7206-4868-908a-05aed37aae54" containerName="payload-pre-processing" Oct 30 16:49:09.800576 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:09.800570 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6cd9cc1-7206-4868-908a-05aed37aae54" containerName="payload-pre-processing" Oct 30 16:49:09.800677 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:09.800599 4493 cpu_manager.go:414] "containerMap: removing container" podUID="6c1ea064-422e-49c7-9577-448806fc5e33" containerName="payload-processing" Oct 30 16:49:09.800677 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:09.800604 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c1ea064-422e-49c7-9577-448806fc5e33" containerName="payload-processing" Oct 30 16:49:09.800740 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:09.800685 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6cd9cc1-7206-4868-908a-05aed37aae54" containerName="payload-pre-processing" Oct 30 16:49:09.800740 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:09.800697 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c1ea064-422e-49c7-9577-448806fc5e33" containerName="payload-processing" Oct 30 16:49:09.800740 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:09.800706 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="90841876-5f2a-47e9-8b7c-b5731a8c49cf" containerName="revoke-keys" Oct 30 16:49:09.803775 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:09.803759 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-504442-fq8px" Oct 30 16:49:09.816200 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:09.816168 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:49:09.816323 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:09.816197 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:49:09.826141 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:09.826115 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 16:49:09.826811 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:09.826759 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:49:09.830835 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:09.830774 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-504442-fq8px"] Oct 30 16:49:09.862164 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:09.862102 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ad5370d-702b-410e-967b-9728ac70dba5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-504442-fq8px\" (UID: \"4ad5370d-702b-410e-967b-9728ac70dba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-504442-fq8px" Oct 30 16:49:09.862528 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:09.862503 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd7jz\" (UniqueName: \"kubernetes.io/projected/4ad5370d-702b-410e-967b-9728ac70dba5-kube-api-access-nd7jz\") pod \"maas-api-revoke-keys-e2e-ipp-a-504442-fq8px\" (UID: \"4ad5370d-702b-410e-967b-9728ac70dba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-504442-fq8px" Oct 30 16:49:09.877851 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:09.870048 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ad5370d-702b-410e-967b-9728ac70dba5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-504442-fq8px\" (UID: \"4ad5370d-702b-410e-967b-9728ac70dba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-504442-fq8px" Oct 30 16:49:09.971342 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:09.971298 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ad5370d-702b-410e-967b-9728ac70dba5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-504442-fq8px\" (UID: \"4ad5370d-702b-410e-967b-9728ac70dba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-504442-fq8px" Oct 30 16:49:09.971533 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:09.971370 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd7jz\" (UniqueName: \"kubernetes.io/projected/4ad5370d-702b-410e-967b-9728ac70dba5-kube-api-access-nd7jz\") pod \"maas-api-revoke-keys-e2e-ipp-a-504442-fq8px\" (UID: \"4ad5370d-702b-410e-967b-9728ac70dba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-504442-fq8px" Oct 30 16:49:09.971533 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:09.971516 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ad5370d-702b-410e-967b-9728ac70dba5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-504442-fq8px\" (UID: \"4ad5370d-702b-410e-967b-9728ac70dba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-504442-fq8px" Oct 30 16:49:09.972149 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:09.972130 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ad5370d-702b-410e-967b-9728ac70dba5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-504442-fq8px\" (UID: \"4ad5370d-702b-410e-967b-9728ac70dba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-504442-fq8px" Oct 30 16:49:09.973749 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:09.973730 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ad5370d-702b-410e-967b-9728ac70dba5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-504442-fq8px\" (UID: \"4ad5370d-702b-410e-967b-9728ac70dba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-504442-fq8px" Oct 30 16:49:09.992926 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:09.992902 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd7jz\" (UniqueName: \"kubernetes.io/projected/4ad5370d-702b-410e-967b-9728ac70dba5-kube-api-access-nd7jz\") pod \"maas-api-revoke-keys-e2e-ipp-a-504442-fq8px\" (UID: \"4ad5370d-702b-410e-967b-9728ac70dba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-504442-fq8px" Oct 30 16:49:10.119049 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:10.118975 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-504442-fq8px" Oct 30 16:49:10.425443 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:10.425306 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-504442-fq8px"] Oct 30 16:49:10.429175 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:49:10.429144 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ad5370d_702b_410e_967b_9728ac70dba5.slice/crio-26fcfecce17ef782bd5595815b0a853e80b77ccec6eff69af43151aeb5abdf57 WatchSource:0}: Error finding container 26fcfecce17ef782bd5595815b0a853e80b77ccec6eff69af43151aeb5abdf57: Status 404 returned error can't find the container with id 26fcfecce17ef782bd5595815b0a853e80b77ccec6eff69af43151aeb5abdf57 Oct 30 16:49:11.359520 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:11.359461 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-504442-fq8px" event={"ID":"4ad5370d-702b-410e-967b-9728ac70dba5","Type":"ContainerStarted","Data":"94d860faaf973cd622a3d4c5952c3039dfcfd3f0196ed5a2ff08c77e691b5987"} Oct 30 16:49:11.360005 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:11.359550 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-504442-fq8px" event={"ID":"4ad5370d-702b-410e-967b-9728ac70dba5","Type":"ContainerStarted","Data":"26fcfecce17ef782bd5595815b0a853e80b77ccec6eff69af43151aeb5abdf57"} Oct 30 16:49:11.409411 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:11.409353 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-504442-fq8px" podStartSLOduration=2.409332989 podStartE2EDuration="2.409332989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:49:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:49:11.404213251 +0000 UTC m=+1578.631388351" watchObservedRunningTime="2026-10-30 16:49:11.409332989 +0000 UTC m=+1578.636508091" Oct 30 16:49:11.922456 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:11.922331 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7"] Oct 30 16:49:11.936615 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:11.936576 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7" Oct 30 16:49:11.956340 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:11.956312 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7"] Oct 30 16:49:11.961805 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:11.961766 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-6d5903-openshift-default-dockercfg-fnbtc\"" Oct 30 16:49:12.013706 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:12.009723 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86b5095e-c77a-478b-b5ba-ff477800a066-istio-data\") pod \"e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7\" (UID: \"86b5095e-c77a-478b-b5ba-ff477800a066\") " pod="openshift-ingress/e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7" Oct 30 16:49:12.013706 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:12.009801 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86b5095e-c77a-478b-b5ba-ff477800a066-istio-envoy\") pod \"e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7\" (UID: \"86b5095e-c77a-478b-b5ba-ff477800a066\") " pod="openshift-ingress/e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7" Oct 30 16:49:12.013706 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:12.009905 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86b5095e-c77a-478b-b5ba-ff477800a066-istio-podinfo\") pod \"e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7\" (UID: \"86b5095e-c77a-478b-b5ba-ff477800a066\") " pod="openshift-ingress/e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7" Oct 30 16:49:12.013706 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:12.009950 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86b5095e-c77a-478b-b5ba-ff477800a066-istio-token\") pod \"e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7\" (UID: \"86b5095e-c77a-478b-b5ba-ff477800a066\") " pod="openshift-ingress/e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7" Oct 30 16:49:12.013706 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:12.009998 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgvs8\" (UniqueName: \"kubernetes.io/projected/86b5095e-c77a-478b-b5ba-ff477800a066-kube-api-access-pgvs8\") pod \"e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7\" (UID: \"86b5095e-c77a-478b-b5ba-ff477800a066\") " pod="openshift-ingress/e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7" Oct 30 16:49:12.013706 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:12.010052 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86b5095e-c77a-478b-b5ba-ff477800a066-workload-certs\") pod \"e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7\" (UID: \"86b5095e-c77a-478b-b5ba-ff477800a066\") " pod="openshift-ingress/e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7" Oct 30 16:49:12.013706 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:12.010087 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86b5095e-c77a-478b-b5ba-ff477800a066-istiod-ca-cert\") pod \"e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7\" (UID: \"86b5095e-c77a-478b-b5ba-ff477800a066\") " pod="openshift-ingress/e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7" Oct 30 16:49:12.013706 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:12.010119 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b5095e-c77a-478b-b5ba-ff477800a066-credential-socket\") pod \"e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7\" (UID: \"86b5095e-c77a-478b-b5ba-ff477800a066\") " pod="openshift-ingress/e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7" Oct 30 16:49:12.013706 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:12.010243 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b5095e-c77a-478b-b5ba-ff477800a066-workload-socket\") pod \"e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7\" (UID: \"86b5095e-c77a-478b-b5ba-ff477800a066\") " pod="openshift-ingress/e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7" Oct 30 16:49:12.111920 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:12.111885 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b5095e-c77a-478b-b5ba-ff477800a066-workload-socket\") pod \"e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7\" (UID: \"86b5095e-c77a-478b-b5ba-ff477800a066\") " pod="openshift-ingress/e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7" Oct 30 16:49:12.112106 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:12.111953 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86b5095e-c77a-478b-b5ba-ff477800a066-istio-data\") pod \"e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7\" (UID: \"86b5095e-c77a-478b-b5ba-ff477800a066\") " pod="openshift-ingress/e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7" Oct 30 16:49:12.112106 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:12.111987 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86b5095e-c77a-478b-b5ba-ff477800a066-istio-envoy\") pod \"e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7\" (UID: \"86b5095e-c77a-478b-b5ba-ff477800a066\") " pod="openshift-ingress/e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7" Oct 30 16:49:12.112106 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:12.112074 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86b5095e-c77a-478b-b5ba-ff477800a066-istio-podinfo\") pod \"e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7\" (UID: \"86b5095e-c77a-478b-b5ba-ff477800a066\") " pod="openshift-ingress/e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7" Oct 30 16:49:12.112299 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:12.112120 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86b5095e-c77a-478b-b5ba-ff477800a066-istio-token\") pod \"e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7\" (UID: \"86b5095e-c77a-478b-b5ba-ff477800a066\") " pod="openshift-ingress/e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7" Oct 30 16:49:12.112299 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:12.112155 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgvs8\" (UniqueName: \"kubernetes.io/projected/86b5095e-c77a-478b-b5ba-ff477800a066-kube-api-access-pgvs8\") pod \"e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7\" (UID: \"86b5095e-c77a-478b-b5ba-ff477800a066\") " pod="openshift-ingress/e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7" Oct 30 16:49:12.112299 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:12.112204 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86b5095e-c77a-478b-b5ba-ff477800a066-workload-certs\") pod \"e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7\" (UID: \"86b5095e-c77a-478b-b5ba-ff477800a066\") " pod="openshift-ingress/e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7" Oct 30 16:49:12.112299 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:12.112249 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86b5095e-c77a-478b-b5ba-ff477800a066-istiod-ca-cert\") pod \"e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7\" (UID: \"86b5095e-c77a-478b-b5ba-ff477800a066\") " pod="openshift-ingress/e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7" Oct 30 16:49:12.112299 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:12.112291 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b5095e-c77a-478b-b5ba-ff477800a066-credential-socket\") pod \"e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7\" (UID: \"86b5095e-c77a-478b-b5ba-ff477800a066\") " pod="openshift-ingress/e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7" Oct 30 16:49:12.112603 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:12.112403 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b5095e-c77a-478b-b5ba-ff477800a066-workload-socket\") pod \"e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7\" (UID: \"86b5095e-c77a-478b-b5ba-ff477800a066\") " pod="openshift-ingress/e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7" Oct 30 16:49:12.112603 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:12.112581 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86b5095e-c77a-478b-b5ba-ff477800a066-workload-certs\") pod \"e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7\" (UID: \"86b5095e-c77a-478b-b5ba-ff477800a066\") " pod="openshift-ingress/e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7" Oct 30 16:49:12.112738 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:12.112645 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b5095e-c77a-478b-b5ba-ff477800a066-credential-socket\") pod \"e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7\" (UID: \"86b5095e-c77a-478b-b5ba-ff477800a066\") " pod="openshift-ingress/e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7" Oct 30 16:49:12.112738 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:12.112697 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86b5095e-c77a-478b-b5ba-ff477800a066-istio-data\") pod \"e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7\" (UID: \"86b5095e-c77a-478b-b5ba-ff477800a066\") " pod="openshift-ingress/e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7" Oct 30 16:49:12.113199 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:12.113175 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86b5095e-c77a-478b-b5ba-ff477800a066-istiod-ca-cert\") pod \"e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7\" (UID: \"86b5095e-c77a-478b-b5ba-ff477800a066\") " pod="openshift-ingress/e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7" Oct 30 16:49:12.114984 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:12.114939 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86b5095e-c77a-478b-b5ba-ff477800a066-istio-envoy\") pod \"e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7\" (UID: \"86b5095e-c77a-478b-b5ba-ff477800a066\") " pod="openshift-ingress/e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7" Oct 30 16:49:12.115210 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:12.115193 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86b5095e-c77a-478b-b5ba-ff477800a066-istio-podinfo\") pod \"e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7\" (UID: \"86b5095e-c77a-478b-b5ba-ff477800a066\") " pod="openshift-ingress/e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7" Oct 30 16:49:12.144128 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:12.144062 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86b5095e-c77a-478b-b5ba-ff477800a066-istio-token\") pod \"e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7\" (UID: \"86b5095e-c77a-478b-b5ba-ff477800a066\") " pod="openshift-ingress/e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7" Oct 30 16:49:12.145857 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:12.144461 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgvs8\" (UniqueName: \"kubernetes.io/projected/86b5095e-c77a-478b-b5ba-ff477800a066-kube-api-access-pgvs8\") pod \"e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7\" (UID: \"86b5095e-c77a-478b-b5ba-ff477800a066\") " pod="openshift-ingress/e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7" Oct 30 16:49:12.282894 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:12.282122 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7" Oct 30 16:49:12.387578 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:12.387545 4493 generic.go:356] "Generic (PLEG): container finished" podID="4ad5370d-702b-410e-967b-9728ac70dba5" containerID="94d860faaf973cd622a3d4c5952c3039dfcfd3f0196ed5a2ff08c77e691b5987" exitCode=0 Oct 30 16:49:12.388747 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:12.388460 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-504442-fq8px" event={"ID":"4ad5370d-702b-410e-967b-9728ac70dba5","Type":"ContainerDied","Data":"94d860faaf973cd622a3d4c5952c3039dfcfd3f0196ed5a2ff08c77e691b5987"} Oct 30 16:49:12.726931 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:12.726895 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7"] Oct 30 16:49:12.727449 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:49:12.727422 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86b5095e_c77a_478b_b5ba_ff477800a066.slice/crio-00c479a6c8ec2127d1b886cb6211355120edfdaa95fed21974a71774853175a7 WatchSource:0}: Error finding container 00c479a6c8ec2127d1b886cb6211355120edfdaa95fed21974a71774853175a7: Status 404 returned error can't find the container with id 00c479a6c8ec2127d1b886cb6211355120edfdaa95fed21974a71774853175a7 Oct 30 16:49:12.730286 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:12.730252 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:49:12.730395 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:12.730322 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:49:12.730395 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:12.730355 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:49:13.396849 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:13.396813 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7" event={"ID":"86b5095e-c77a-478b-b5ba-ff477800a066","Type":"ContainerStarted","Data":"7621daac5dad9e6f7afc2c9eb3238930f657b8d9e1ff1d6cdf16adf54de856b0"} Oct 30 16:49:13.397296 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:13.396874 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7" event={"ID":"86b5095e-c77a-478b-b5ba-ff477800a066","Type":"ContainerStarted","Data":"00c479a6c8ec2127d1b886cb6211355120edfdaa95fed21974a71774853175a7"} Oct 30 16:49:13.451205 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:13.451119 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7" podStartSLOduration=2.451098251 podStartE2EDuration="2.451098251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:49:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:49:13.44840733 +0000 UTC m=+1580.675582430" watchObservedRunningTime="2026-10-30 16:49:13.451098251 +0000 UTC m=+1580.678273355" Oct 30 16:49:13.570236 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:13.570173 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-504442-fq8px" Oct 30 16:49:13.647371 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:13.647279 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ad5370d-702b-410e-967b-9728ac70dba5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ad5370d-702b-410e-967b-9728ac70dba5-maas-api-xks-ca\") pod \"4ad5370d-702b-410e-967b-9728ac70dba5\" (UID: \"4ad5370d-702b-410e-967b-9728ac70dba5\") " Oct 30 16:49:13.647546 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:13.647373 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ad5370d-702b-410e-967b-9728ac70dba5-kube-api-access-nd7jz\" (UniqueName: \"kubernetes.io/projected/4ad5370d-702b-410e-967b-9728ac70dba5-kube-api-access-nd7jz\") pod \"4ad5370d-702b-410e-967b-9728ac70dba5\" (UID: \"4ad5370d-702b-410e-967b-9728ac70dba5\") " Oct 30 16:49:13.647546 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:13.647539 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ad5370d-702b-410e-967b-9728ac70dba5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ad5370d-702b-410e-967b-9728ac70dba5-maas-api-openshift-ca\") pod \"4ad5370d-702b-410e-967b-9728ac70dba5\" (UID: \"4ad5370d-702b-410e-967b-9728ac70dba5\") " Oct 30 16:49:13.648427 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:13.648397 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ad5370d-702b-410e-967b-9728ac70dba5-maas-api-openshift-ca" pod "4ad5370d-702b-410e-967b-9728ac70dba5" (UID: "4ad5370d-702b-410e-967b-9728ac70dba5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:49:13.650721 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:13.650693 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ad5370d-702b-410e-967b-9728ac70dba5-maas-api-xks-ca" pod "4ad5370d-702b-410e-967b-9728ac70dba5" (UID: "4ad5370d-702b-410e-967b-9728ac70dba5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:49:13.651914 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:13.651886 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ad5370d-702b-410e-967b-9728ac70dba5-kube-api-access-nd7jz" pod "4ad5370d-702b-410e-967b-9728ac70dba5" (UID: "4ad5370d-702b-410e-967b-9728ac70dba5"). InnerVolumeSpecName "kube-api-access-nd7jz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:49:13.750216 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:13.750175 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nd7jz\" (UniqueName: \"kubernetes.io/projected/4ad5370d-702b-410e-967b-9728ac70dba5-kube-api-access-nd7jz\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:49:13.750216 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:13.750205 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ad5370d-702b-410e-967b-9728ac70dba5-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:49:13.750216 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:13.750220 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ad5370d-702b-410e-967b-9728ac70dba5-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:49:14.283632 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:14.283600 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7" Oct 30 16:49:14.290432 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:14.290408 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7" Oct 30 16:49:14.415189 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:14.415156 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-504442-fq8px" Oct 30 16:49:14.415189 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:14.415186 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-504442-fq8px" event={"ID":"4ad5370d-702b-410e-967b-9728ac70dba5","Type":"ContainerDied","Data":"26fcfecce17ef782bd5595815b0a853e80b77ccec6eff69af43151aeb5abdf57"} Oct 30 16:49:14.415744 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:14.415231 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26fcfecce17ef782bd5595815b0a853e80b77ccec6eff69af43151aeb5abdf57" Oct 30 16:49:14.415744 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:14.415426 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7" Oct 30 16:49:14.416512 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:14.416492 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7" Oct 30 16:49:23.810809 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:23.807935 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 16:49:29.575253 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:29.575222 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2"] Oct 30 16:49:29.575749 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:29.575499 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2" podUID="c7ae7018-798b-4a3a-afef-b2ed1a13b05f" containerName="istio-proxy" containerID="cri-o://77995f75476d1b422cb7d950bac96dace02d103308e06c8c1b9649b5c577b969" gracePeriod=30 Oct 30 16:49:29.584263 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:29.584236 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-d9rhc"] Oct 30 16:49:29.584492 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:29.584447 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-d9rhc" podUID="0476eb7d-91e3-4f24-bfbc-245775c8e93b" containerName="istio-proxy" containerID="cri-o://97a8e4b2ca4d74a421648cb3113f545ac021c7a7ca3deead0f96b8539a6f5358" gracePeriod=30 Oct 30 16:49:29.750207 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:29.750141 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7686c34-glx8c"] Oct 30 16:49:29.751470 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:29.751443 4493 cpu_manager.go:414] "containerMap: removing container" podUID="4ad5370d-702b-410e-967b-9728ac70dba5" containerName="revoke-keys" Oct 30 16:49:29.751470 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:29.751472 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ad5370d-702b-410e-967b-9728ac70dba5" containerName="revoke-keys" Oct 30 16:49:29.751659 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:29.751641 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ad5370d-702b-410e-967b-9728ac70dba5" containerName="revoke-keys" Oct 30 16:49:29.756049 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:29.756027 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7686c34-glx8c" Oct 30 16:49:29.770228 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:29.770204 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:49:29.770517 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:29.770502 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 16:49:29.770687 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:29.770673 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:49:29.770860 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:29.770846 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:49:29.809873 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:29.809839 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7686c34-glx8c"] Oct 30 16:49:29.893411 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:29.893363 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bc54dd3-3017-4f7c-a6d7-3b81a363f3b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7686c34-glx8c\" (UID: \"8bc54dd3-3017-4f7c-a6d7-3b81a363f3b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7686c34-glx8c" Oct 30 16:49:29.893838 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:29.893745 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bc54dd3-3017-4f7c-a6d7-3b81a363f3b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7686c34-glx8c\" (UID: \"8bc54dd3-3017-4f7c-a6d7-3b81a363f3b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7686c34-glx8c" Oct 30 16:49:29.894139 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:29.894120 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cddw\" (UniqueName: \"kubernetes.io/projected/8bc54dd3-3017-4f7c-a6d7-3b81a363f3b7-kube-api-access-5cddw\") pod \"maas-api-revoke-sub-a7686c34-glx8c\" (UID: \"8bc54dd3-3017-4f7c-a6d7-3b81a363f3b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7686c34-glx8c" Oct 30 16:49:29.995286 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:29.995248 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bc54dd3-3017-4f7c-a6d7-3b81a363f3b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7686c34-glx8c\" (UID: \"8bc54dd3-3017-4f7c-a6d7-3b81a363f3b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7686c34-glx8c" Oct 30 16:49:29.995488 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:29.995323 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cddw\" (UniqueName: \"kubernetes.io/projected/8bc54dd3-3017-4f7c-a6d7-3b81a363f3b7-kube-api-access-5cddw\") pod \"maas-api-revoke-sub-a7686c34-glx8c\" (UID: \"8bc54dd3-3017-4f7c-a6d7-3b81a363f3b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7686c34-glx8c" Oct 30 16:49:29.995562 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:29.995516 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bc54dd3-3017-4f7c-a6d7-3b81a363f3b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7686c34-glx8c\" (UID: \"8bc54dd3-3017-4f7c-a6d7-3b81a363f3b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7686c34-glx8c" Oct 30 16:49:29.996025 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:29.996000 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bc54dd3-3017-4f7c-a6d7-3b81a363f3b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7686c34-glx8c\" (UID: \"8bc54dd3-3017-4f7c-a6d7-3b81a363f3b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7686c34-glx8c" Oct 30 16:49:29.998211 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:29.998181 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bc54dd3-3017-4f7c-a6d7-3b81a363f3b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7686c34-glx8c\" (UID: \"8bc54dd3-3017-4f7c-a6d7-3b81a363f3b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7686c34-glx8c" Oct 30 16:49:30.016279 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:30.016258 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cddw\" (UniqueName: \"kubernetes.io/projected/8bc54dd3-3017-4f7c-a6d7-3b81a363f3b7-kube-api-access-5cddw\") pod \"maas-api-revoke-sub-a7686c34-glx8c\" (UID: \"8bc54dd3-3017-4f7c-a6d7-3b81a363f3b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7686c34-glx8c" Oct 30 16:49:30.111375 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:30.111342 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7686c34-glx8c" Oct 30 16:49:30.132050 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:30.132022 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t"] Oct 30 16:49:30.132254 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:30.132230 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t" podUID="ad417aab-5686-4ae2-9163-91af4a79b356" containerName="istio-proxy" containerID="cri-o://d266cdc2c7712fcb40ea4d7125601d5ddf3af201f1a5c5b8fd9217fd245504ef" gracePeriod=30 Oct 30 16:49:30.342126 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:49:30.342083 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 16:49:30.433482 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:30.433408 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7686c34-glx8c"] Oct 30 16:49:30.437426 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:49:30.437394 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bc54dd3_3017_4f7c_a6d7_3b81a363f3b7.slice/crio-901cbb14ca2853c6a1c1ae4292573196714aac9cdaa20e1c47b8fce1a409e66e WatchSource:0}: Error finding container 901cbb14ca2853c6a1c1ae4292573196714aac9cdaa20e1c47b8fce1a409e66e: Status 404 returned error can't find the container with id 901cbb14ca2853c6a1c1ae4292573196714aac9cdaa20e1c47b8fce1a409e66e Oct 30 16:49:30.543959 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:30.543929 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-569b62-2mzzp"] Oct 30 16:49:30.549908 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:30.549885 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-569b62-2mzzp" Oct 30 16:49:30.570283 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:30.570258 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-569b62-2mzzp"] Oct 30 16:49:30.602544 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:30.602506 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7686c34-glx8c" event={"ID":"8bc54dd3-3017-4f7c-a6d7-3b81a363f3b7","Type":"ContainerStarted","Data":"901cbb14ca2853c6a1c1ae4292573196714aac9cdaa20e1c47b8fce1a409e66e"} Oct 30 16:49:30.604627 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:30.604602 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b86ls\" (UniqueName: \"kubernetes.io/projected/b43de74a-bb00-40ab-b99d-7ffec5b50bc7-kube-api-access-b86ls\") pod \"maas-api-revoke-keys-e2e-praxis-a-569b62-2mzzp\" (UID: \"b43de74a-bb00-40ab-b99d-7ffec5b50bc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-569b62-2mzzp" Oct 30 16:49:30.604762 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:30.604734 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b43de74a-bb00-40ab-b99d-7ffec5b50bc7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-569b62-2mzzp\" (UID: \"b43de74a-bb00-40ab-b99d-7ffec5b50bc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-569b62-2mzzp" Oct 30 16:49:30.604944 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:30.604929 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b43de74a-bb00-40ab-b99d-7ffec5b50bc7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-569b62-2mzzp\" (UID: \"b43de74a-bb00-40ab-b99d-7ffec5b50bc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-569b62-2mzzp" Oct 30 16:49:30.611529 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:30.611502 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn"] Oct 30 16:49:30.611988 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:30.611711 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn" podUID="84009ea2-157b-45c6-b4bd-6df55fc7c72d" containerName="istio-proxy" containerID="cri-o://8ebf076f93f55dae5e773e774ad3dc10673d4b61201803708b086c5436da423e" gracePeriod=30 Oct 30 16:49:30.705829 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:30.705732 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b86ls\" (UniqueName: \"kubernetes.io/projected/b43de74a-bb00-40ab-b99d-7ffec5b50bc7-kube-api-access-b86ls\") pod \"maas-api-revoke-keys-e2e-praxis-a-569b62-2mzzp\" (UID: \"b43de74a-bb00-40ab-b99d-7ffec5b50bc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-569b62-2mzzp" Oct 30 16:49:30.706018 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:30.705852 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b43de74a-bb00-40ab-b99d-7ffec5b50bc7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-569b62-2mzzp\" (UID: \"b43de74a-bb00-40ab-b99d-7ffec5b50bc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-569b62-2mzzp" Oct 30 16:49:30.706018 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:30.705958 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b43de74a-bb00-40ab-b99d-7ffec5b50bc7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-569b62-2mzzp\" (UID: \"b43de74a-bb00-40ab-b99d-7ffec5b50bc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-569b62-2mzzp" Oct 30 16:49:30.706578 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:30.706552 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b43de74a-bb00-40ab-b99d-7ffec5b50bc7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-569b62-2mzzp\" (UID: \"b43de74a-bb00-40ab-b99d-7ffec5b50bc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-569b62-2mzzp" Oct 30 16:49:30.708326 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:30.708308 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b43de74a-bb00-40ab-b99d-7ffec5b50bc7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-569b62-2mzzp\" (UID: \"b43de74a-bb00-40ab-b99d-7ffec5b50bc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-569b62-2mzzp" Oct 30 16:49:30.733208 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:30.733102 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b86ls\" (UniqueName: \"kubernetes.io/projected/b43de74a-bb00-40ab-b99d-7ffec5b50bc7-kube-api-access-b86ls\") pod \"maas-api-revoke-keys-e2e-praxis-a-569b62-2mzzp\" (UID: \"b43de74a-bb00-40ab-b99d-7ffec5b50bc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-569b62-2mzzp" Oct 30 16:49:30.863841 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:30.863815 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-569b62-2mzzp" Oct 30 16:49:31.175720 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:31.175660 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-569b62-2mzzp"] Oct 30 16:49:31.178606 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:49:31.178573 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb43de74a_bb00_40ab_b99d_7ffec5b50bc7.slice/crio-33a85a450db1c2037118bf2520d676cd1455625300b20a3ca99bba7496a8acf2 WatchSource:0}: Error finding container 33a85a450db1c2037118bf2520d676cd1455625300b20a3ca99bba7496a8acf2: Status 404 returned error can't find the container with id 33a85a450db1c2037118bf2520d676cd1455625300b20a3ca99bba7496a8acf2 Oct 30 16:49:31.612842 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:31.612802 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-569b62-2mzzp" event={"ID":"b43de74a-bb00-40ab-b99d-7ffec5b50bc7","Type":"ContainerStarted","Data":"f4957884cf37e49002efd32ccfadb4df27c13e1a37d4996fbda091a438e26da4"} Oct 30 16:49:31.613292 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:31.612865 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-569b62-2mzzp" event={"ID":"b43de74a-bb00-40ab-b99d-7ffec5b50bc7","Type":"ContainerStarted","Data":"33a85a450db1c2037118bf2520d676cd1455625300b20a3ca99bba7496a8acf2"} Oct 30 16:49:31.614455 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:31.614431 4493 generic.go:356] "Generic (PLEG): container finished" podID="8bc54dd3-3017-4f7c-a6d7-3b81a363f3b7" containerID="c560012373a63803dbb746dc6b8ea1989475d264d545a2bafbd4d100d0936aca" exitCode=0 Oct 30 16:49:31.614561 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:31.614503 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7686c34-glx8c" event={"ID":"8bc54dd3-3017-4f7c-a6d7-3b81a363f3b7","Type":"ContainerDied","Data":"c560012373a63803dbb746dc6b8ea1989475d264d545a2bafbd4d100d0936aca"} Oct 30 16:49:31.652246 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:31.652206 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-569b62-2mzzp" podStartSLOduration=1.65219067 podStartE2EDuration="1.65219067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:49:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:49:31.649021187 +0000 UTC m=+1598.876196288" watchObservedRunningTime="2026-10-30 16:49:31.65219067 +0000 UTC m=+1598.879365766" Oct 30 16:49:31.938273 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:31.938236 4493 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 16:49:31.938429 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:31.938298 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2" podUID="c7ae7018-798b-4a3a-afef-b2ed1a13b05f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 16:49:32.624229 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:32.624139 4493 generic.go:356] "Generic (PLEG): container finished" podID="b43de74a-bb00-40ab-b99d-7ffec5b50bc7" containerID="f4957884cf37e49002efd32ccfadb4df27c13e1a37d4996fbda091a438e26da4" exitCode=0 Oct 30 16:49:32.624641 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:32.624223 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-569b62-2mzzp" event={"ID":"b43de74a-bb00-40ab-b99d-7ffec5b50bc7","Type":"ContainerDied","Data":"f4957884cf37e49002efd32ccfadb4df27c13e1a37d4996fbda091a438e26da4"} Oct 30 16:49:32.809034 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:32.809004 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7686c34-glx8c" Oct 30 16:49:32.833230 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:32.833193 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bc54dd3-3017-4f7c-a6d7-3b81a363f3b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bc54dd3-3017-4f7c-a6d7-3b81a363f3b7-maas-api-openshift-ca\") pod \"8bc54dd3-3017-4f7c-a6d7-3b81a363f3b7\" (UID: \"8bc54dd3-3017-4f7c-a6d7-3b81a363f3b7\") " Oct 30 16:49:32.833418 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:32.833309 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bc54dd3-3017-4f7c-a6d7-3b81a363f3b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bc54dd3-3017-4f7c-a6d7-3b81a363f3b7-maas-api-xks-ca\") pod \"8bc54dd3-3017-4f7c-a6d7-3b81a363f3b7\" (UID: \"8bc54dd3-3017-4f7c-a6d7-3b81a363f3b7\") " Oct 30 16:49:32.833519 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:32.833485 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bc54dd3-3017-4f7c-a6d7-3b81a363f3b7-kube-api-access-5cddw\" (UniqueName: \"kubernetes.io/projected/8bc54dd3-3017-4f7c-a6d7-3b81a363f3b7-kube-api-access-5cddw\") pod \"8bc54dd3-3017-4f7c-a6d7-3b81a363f3b7\" (UID: \"8bc54dd3-3017-4f7c-a6d7-3b81a363f3b7\") " Oct 30 16:49:32.833630 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:32.833604 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bc54dd3-3017-4f7c-a6d7-3b81a363f3b7-maas-api-openshift-ca" pod "8bc54dd3-3017-4f7c-a6d7-3b81a363f3b7" (UID: "8bc54dd3-3017-4f7c-a6d7-3b81a363f3b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:49:32.834104 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:32.834056 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bc54dd3-3017-4f7c-a6d7-3b81a363f3b7-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:49:32.835623 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:32.835595 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bc54dd3-3017-4f7c-a6d7-3b81a363f3b7-maas-api-xks-ca" pod "8bc54dd3-3017-4f7c-a6d7-3b81a363f3b7" (UID: "8bc54dd3-3017-4f7c-a6d7-3b81a363f3b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:49:32.835732 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:32.835662 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bc54dd3-3017-4f7c-a6d7-3b81a363f3b7-kube-api-access-5cddw" pod "8bc54dd3-3017-4f7c-a6d7-3b81a363f3b7" (UID: "8bc54dd3-3017-4f7c-a6d7-3b81a363f3b7"). InnerVolumeSpecName "kube-api-access-5cddw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:49:32.934753 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:32.934724 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cddw\" (UniqueName: \"kubernetes.io/projected/8bc54dd3-3017-4f7c-a6d7-3b81a363f3b7-kube-api-access-5cddw\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:49:32.934753 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:32.934752 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bc54dd3-3017-4f7c-a6d7-3b81a363f3b7-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:49:33.634186 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:33.634152 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7686c34-glx8c" Oct 30 16:49:33.634675 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:33.634187 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7686c34-glx8c" event={"ID":"8bc54dd3-3017-4f7c-a6d7-3b81a363f3b7","Type":"ContainerDied","Data":"901cbb14ca2853c6a1c1ae4292573196714aac9cdaa20e1c47b8fce1a409e66e"} Oct 30 16:49:33.634675 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:33.634241 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="901cbb14ca2853c6a1c1ae4292573196714aac9cdaa20e1c47b8fce1a409e66e" Oct 30 16:49:33.785204 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:33.785182 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-569b62-2mzzp" Oct 30 16:49:33.849585 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:33.847150 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b43de74a-bb00-40ab-b99d-7ffec5b50bc7-kube-api-access-b86ls\" (UniqueName: \"kubernetes.io/projected/b43de74a-bb00-40ab-b99d-7ffec5b50bc7-kube-api-access-b86ls\") pod \"b43de74a-bb00-40ab-b99d-7ffec5b50bc7\" (UID: \"b43de74a-bb00-40ab-b99d-7ffec5b50bc7\") " Oct 30 16:49:33.849585 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:33.847358 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b43de74a-bb00-40ab-b99d-7ffec5b50bc7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b43de74a-bb00-40ab-b99d-7ffec5b50bc7-maas-api-openshift-ca\") pod \"b43de74a-bb00-40ab-b99d-7ffec5b50bc7\" (UID: \"b43de74a-bb00-40ab-b99d-7ffec5b50bc7\") " Oct 30 16:49:33.849585 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:33.847457 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b43de74a-bb00-40ab-b99d-7ffec5b50bc7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b43de74a-bb00-40ab-b99d-7ffec5b50bc7-maas-api-xks-ca\") pod \"b43de74a-bb00-40ab-b99d-7ffec5b50bc7\" (UID: \"b43de74a-bb00-40ab-b99d-7ffec5b50bc7\") " Oct 30 16:49:33.849585 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:33.849371 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b43de74a-bb00-40ab-b99d-7ffec5b50bc7-maas-api-openshift-ca" pod "b43de74a-bb00-40ab-b99d-7ffec5b50bc7" (UID: "b43de74a-bb00-40ab-b99d-7ffec5b50bc7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:49:33.851508 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:33.851480 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b43de74a-bb00-40ab-b99d-7ffec5b50bc7-kube-api-access-b86ls" pod "b43de74a-bb00-40ab-b99d-7ffec5b50bc7" (UID: "b43de74a-bb00-40ab-b99d-7ffec5b50bc7"). InnerVolumeSpecName "kube-api-access-b86ls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:49:33.852101 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:33.852073 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b43de74a-bb00-40ab-b99d-7ffec5b50bc7-maas-api-xks-ca" pod "b43de74a-bb00-40ab-b99d-7ffec5b50bc7" (UID: "b43de74a-bb00-40ab-b99d-7ffec5b50bc7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:49:33.949479 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:33.949449 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b43de74a-bb00-40ab-b99d-7ffec5b50bc7-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:49:33.949479 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:33.949477 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b43de74a-bb00-40ab-b99d-7ffec5b50bc7-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:49:33.949703 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:33.949490 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b86ls\" (UniqueName: \"kubernetes.io/projected/b43de74a-bb00-40ab-b99d-7ffec5b50bc7-kube-api-access-b86ls\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:49:34.553904 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:34.553873 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-76cccb-m4q75"] Oct 30 16:49:34.554412 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:34.554400 4493 cpu_manager.go:414] "containerMap: removing container" podUID="8bc54dd3-3017-4f7c-a6d7-3b81a363f3b7" containerName="revoke-keys" Oct 30 16:49:34.554456 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:34.554414 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bc54dd3-3017-4f7c-a6d7-3b81a363f3b7" containerName="revoke-keys" Oct 30 16:49:34.554503 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:34.554473 4493 cpu_manager.go:414] "containerMap: removing container" podUID="b43de74a-bb00-40ab-b99d-7ffec5b50bc7" containerName="revoke-keys" Oct 30 16:49:34.554503 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:34.554482 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b43de74a-bb00-40ab-b99d-7ffec5b50bc7" containerName="revoke-keys" Oct 30 16:49:34.554586 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:34.554576 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bc54dd3-3017-4f7c-a6d7-3b81a363f3b7" containerName="revoke-keys" Oct 30 16:49:34.554624 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:34.554594 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="b43de74a-bb00-40ab-b99d-7ffec5b50bc7" containerName="revoke-keys" Oct 30 16:49:34.558064 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:34.558046 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-76cccb-m4q75" Oct 30 16:49:34.579019 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:34.578992 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-76cccb-m4q75"] Oct 30 16:49:34.647849 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:34.647827 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-569b62-2mzzp" Oct 30 16:49:34.648274 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:34.647850 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-569b62-2mzzp" event={"ID":"b43de74a-bb00-40ab-b99d-7ffec5b50bc7","Type":"ContainerDied","Data":"33a85a450db1c2037118bf2520d676cd1455625300b20a3ca99bba7496a8acf2"} Oct 30 16:49:34.648414 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:34.648302 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33a85a450db1c2037118bf2520d676cd1455625300b20a3ca99bba7496a8acf2" Oct 30 16:49:34.661661 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:34.661638 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/232a337f-6b37-4c14-8d9b-fabda3c1f93b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-76cccb-m4q75\" (UID: \"232a337f-6b37-4c14-8d9b-fabda3c1f93b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-76cccb-m4q75" Oct 30 16:49:34.661767 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:34.661675 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkjmx\" (UniqueName: \"kubernetes.io/projected/232a337f-6b37-4c14-8d9b-fabda3c1f93b-kube-api-access-jkjmx\") pod \"maas-api-revoke-keys-e2e-praxis-a-76cccb-m4q75\" (UID: \"232a337f-6b37-4c14-8d9b-fabda3c1f93b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-76cccb-m4q75" Oct 30 16:49:34.661850 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:34.661832 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/232a337f-6b37-4c14-8d9b-fabda3c1f93b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-76cccb-m4q75\" (UID: \"232a337f-6b37-4c14-8d9b-fabda3c1f93b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-76cccb-m4q75" Oct 30 16:49:34.768666 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:34.768411 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/232a337f-6b37-4c14-8d9b-fabda3c1f93b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-76cccb-m4q75\" (UID: \"232a337f-6b37-4c14-8d9b-fabda3c1f93b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-76cccb-m4q75" Oct 30 16:49:34.768666 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:34.768641 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/232a337f-6b37-4c14-8d9b-fabda3c1f93b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-76cccb-m4q75\" (UID: \"232a337f-6b37-4c14-8d9b-fabda3c1f93b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-76cccb-m4q75" Oct 30 16:49:34.768908 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:34.768684 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkjmx\" (UniqueName: \"kubernetes.io/projected/232a337f-6b37-4c14-8d9b-fabda3c1f93b-kube-api-access-jkjmx\") pod \"maas-api-revoke-keys-e2e-praxis-a-76cccb-m4q75\" (UID: \"232a337f-6b37-4c14-8d9b-fabda3c1f93b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-76cccb-m4q75" Oct 30 16:49:34.771093 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:34.770448 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/232a337f-6b37-4c14-8d9b-fabda3c1f93b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-76cccb-m4q75\" (UID: \"232a337f-6b37-4c14-8d9b-fabda3c1f93b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-76cccb-m4q75" Oct 30 16:49:34.791329 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:34.790953 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/232a337f-6b37-4c14-8d9b-fabda3c1f93b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-76cccb-m4q75\" (UID: \"232a337f-6b37-4c14-8d9b-fabda3c1f93b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-76cccb-m4q75" Oct 30 16:49:34.801376 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:34.801319 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkjmx\" (UniqueName: \"kubernetes.io/projected/232a337f-6b37-4c14-8d9b-fabda3c1f93b-kube-api-access-jkjmx\") pod \"maas-api-revoke-keys-e2e-praxis-a-76cccb-m4q75\" (UID: \"232a337f-6b37-4c14-8d9b-fabda3c1f93b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-76cccb-m4q75" Oct 30 16:49:34.883008 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:34.880199 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-76cccb-m4q75" Oct 30 16:49:34.971643 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:34.971361 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2" Oct 30 16:49:35.069609 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.066123 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-d9rhc" Oct 30 16:49:35.081773 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.081741 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7ae7018-798b-4a3a-afef-b2ed1a13b05f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7ae7018-798b-4a3a-afef-b2ed1a13b05f-credential-socket\") pod \"c7ae7018-798b-4a3a-afef-b2ed1a13b05f\" (UID: \"c7ae7018-798b-4a3a-afef-b2ed1a13b05f\") " Oct 30 16:49:35.081953 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.081839 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7ae7018-798b-4a3a-afef-b2ed1a13b05f-istio-token\" (UniqueName: \"kubernetes.io/projected/c7ae7018-798b-4a3a-afef-b2ed1a13b05f-istio-token\") pod \"c7ae7018-798b-4a3a-afef-b2ed1a13b05f\" (UID: \"c7ae7018-798b-4a3a-afef-b2ed1a13b05f\") " Oct 30 16:49:35.081953 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.081903 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7ae7018-798b-4a3a-afef-b2ed1a13b05f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7ae7018-798b-4a3a-afef-b2ed1a13b05f-workload-socket\") pod \"c7ae7018-798b-4a3a-afef-b2ed1a13b05f\" (UID: \"c7ae7018-798b-4a3a-afef-b2ed1a13b05f\") " Oct 30 16:49:35.081953 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.081941 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7ae7018-798b-4a3a-afef-b2ed1a13b05f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7ae7018-798b-4a3a-afef-b2ed1a13b05f-istio-data\") pod \"c7ae7018-798b-4a3a-afef-b2ed1a13b05f\" (UID: \"c7ae7018-798b-4a3a-afef-b2ed1a13b05f\") " Oct 30 16:49:35.082067 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.082045 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7ae7018-798b-4a3a-afef-b2ed1a13b05f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7ae7018-798b-4a3a-afef-b2ed1a13b05f-istio-envoy\") pod \"c7ae7018-798b-4a3a-afef-b2ed1a13b05f\" (UID: \"c7ae7018-798b-4a3a-afef-b2ed1a13b05f\") " Oct 30 16:49:35.082121 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.082107 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7ae7018-798b-4a3a-afef-b2ed1a13b05f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7ae7018-798b-4a3a-afef-b2ed1a13b05f-workload-certs\") pod \"c7ae7018-798b-4a3a-afef-b2ed1a13b05f\" (UID: \"c7ae7018-798b-4a3a-afef-b2ed1a13b05f\") " Oct 30 16:49:35.082156 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.082147 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c7ae7018-798b-4a3a-afef-b2ed1a13b05f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7ae7018-798b-4a3a-afef-b2ed1a13b05f-istio-podinfo\") pod \"c7ae7018-798b-4a3a-afef-b2ed1a13b05f\" (UID: \"c7ae7018-798b-4a3a-afef-b2ed1a13b05f\") " Oct 30 16:49:35.082244 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.082230 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7ae7018-798b-4a3a-afef-b2ed1a13b05f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7ae7018-798b-4a3a-afef-b2ed1a13b05f-istiod-ca-cert\") pod \"c7ae7018-798b-4a3a-afef-b2ed1a13b05f\" (UID: \"c7ae7018-798b-4a3a-afef-b2ed1a13b05f\") " Oct 30 16:49:35.082293 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.082280 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7ae7018-798b-4a3a-afef-b2ed1a13b05f-kube-api-access-q99cb\" (UniqueName: \"kubernetes.io/projected/c7ae7018-798b-4a3a-afef-b2ed1a13b05f-kube-api-access-q99cb\") pod \"c7ae7018-798b-4a3a-afef-b2ed1a13b05f\" (UID: \"c7ae7018-798b-4a3a-afef-b2ed1a13b05f\") " Oct 30 16:49:35.083356 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.083330 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7ae7018-798b-4a3a-afef-b2ed1a13b05f-istio-data" pod "c7ae7018-798b-4a3a-afef-b2ed1a13b05f" (UID: "c7ae7018-798b-4a3a-afef-b2ed1a13b05f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:49:35.083424 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.083405 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk"] Oct 30 16:49:35.083531 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.083514 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7ae7018-798b-4a3a-afef-b2ed1a13b05f-credential-socket" pod "c7ae7018-798b-4a3a-afef-b2ed1a13b05f" (UID: "c7ae7018-798b-4a3a-afef-b2ed1a13b05f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:49:35.083648 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.083624 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk" podUID="320f1fb7-deb8-4a82-b468-f92066eca8a1" containerName="istio-proxy" containerID="cri-o://24a87beb129a10f854bcadc0f61d76be963d76dbd791cb56e7a80316215ae361" gracePeriod=30 Oct 30 16:49:35.084598 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.084514 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7ae7018-798b-4a3a-afef-b2ed1a13b05f-istiod-ca-cert" pod "c7ae7018-798b-4a3a-afef-b2ed1a13b05f" (UID: "c7ae7018-798b-4a3a-afef-b2ed1a13b05f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:49:35.084598 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.084527 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7ae7018-798b-4a3a-afef-b2ed1a13b05f-workload-socket" pod "c7ae7018-798b-4a3a-afef-b2ed1a13b05f" (UID: "c7ae7018-798b-4a3a-afef-b2ed1a13b05f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:49:35.084598 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.084579 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7ae7018-798b-4a3a-afef-b2ed1a13b05f-workload-certs" pod "c7ae7018-798b-4a3a-afef-b2ed1a13b05f" (UID: "c7ae7018-798b-4a3a-afef-b2ed1a13b05f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:49:35.088249 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.088220 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7ae7018-798b-4a3a-afef-b2ed1a13b05f-istio-token" pod "c7ae7018-798b-4a3a-afef-b2ed1a13b05f" (UID: "c7ae7018-798b-4a3a-afef-b2ed1a13b05f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:49:35.088581 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.088559 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7ae7018-798b-4a3a-afef-b2ed1a13b05f-istio-envoy" pod "c7ae7018-798b-4a3a-afef-b2ed1a13b05f" (UID: "c7ae7018-798b-4a3a-afef-b2ed1a13b05f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:49:35.091668 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.091637 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c7ae7018-798b-4a3a-afef-b2ed1a13b05f-istio-podinfo" pod "c7ae7018-798b-4a3a-afef-b2ed1a13b05f" (UID: "c7ae7018-798b-4a3a-afef-b2ed1a13b05f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:49:35.096008 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.095980 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7ae7018-798b-4a3a-afef-b2ed1a13b05f-kube-api-access-q99cb" pod "c7ae7018-798b-4a3a-afef-b2ed1a13b05f" (UID: "c7ae7018-798b-4a3a-afef-b2ed1a13b05f"). InnerVolumeSpecName "kube-api-access-q99cb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:49:35.184122 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.184099 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0476eb7d-91e3-4f24-bfbc-245775c8e93b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0476eb7d-91e3-4f24-bfbc-245775c8e93b-istio-podinfo\") pod \"0476eb7d-91e3-4f24-bfbc-245775c8e93b\" (UID: \"0476eb7d-91e3-4f24-bfbc-245775c8e93b\") " Oct 30 16:49:35.184239 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.184218 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0476eb7d-91e3-4f24-bfbc-245775c8e93b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0476eb7d-91e3-4f24-bfbc-245775c8e93b-credential-socket\") pod \"0476eb7d-91e3-4f24-bfbc-245775c8e93b\" (UID: \"0476eb7d-91e3-4f24-bfbc-245775c8e93b\") " Oct 30 16:49:35.184297 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.184244 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0476eb7d-91e3-4f24-bfbc-245775c8e93b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0476eb7d-91e3-4f24-bfbc-245775c8e93b-istio-data\") pod \"0476eb7d-91e3-4f24-bfbc-245775c8e93b\" (UID: \"0476eb7d-91e3-4f24-bfbc-245775c8e93b\") " Oct 30 16:49:35.184334 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.184304 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0476eb7d-91e3-4f24-bfbc-245775c8e93b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0476eb7d-91e3-4f24-bfbc-245775c8e93b-workload-certs\") pod \"0476eb7d-91e3-4f24-bfbc-245775c8e93b\" (UID: \"0476eb7d-91e3-4f24-bfbc-245775c8e93b\") " Oct 30 16:49:35.184387 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.184360 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0476eb7d-91e3-4f24-bfbc-245775c8e93b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0476eb7d-91e3-4f24-bfbc-245775c8e93b-istiod-ca-cert\") pod \"0476eb7d-91e3-4f24-bfbc-245775c8e93b\" (UID: \"0476eb7d-91e3-4f24-bfbc-245775c8e93b\") " Oct 30 16:49:35.184434 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.184394 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0476eb7d-91e3-4f24-bfbc-245775c8e93b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0476eb7d-91e3-4f24-bfbc-245775c8e93b-istio-envoy\") pod \"0476eb7d-91e3-4f24-bfbc-245775c8e93b\" (UID: \"0476eb7d-91e3-4f24-bfbc-245775c8e93b\") " Oct 30 16:49:35.184586 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.184509 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0476eb7d-91e3-4f24-bfbc-245775c8e93b-istio-token\" (UniqueName: \"kubernetes.io/projected/0476eb7d-91e3-4f24-bfbc-245775c8e93b-istio-token\") pod \"0476eb7d-91e3-4f24-bfbc-245775c8e93b\" (UID: \"0476eb7d-91e3-4f24-bfbc-245775c8e93b\") " Oct 30 16:49:35.184586 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.184554 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0476eb7d-91e3-4f24-bfbc-245775c8e93b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0476eb7d-91e3-4f24-bfbc-245775c8e93b-workload-socket\") pod \"0476eb7d-91e3-4f24-bfbc-245775c8e93b\" (UID: \"0476eb7d-91e3-4f24-bfbc-245775c8e93b\") " Oct 30 16:49:35.184776 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.184594 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0476eb7d-91e3-4f24-bfbc-245775c8e93b-kube-api-access-fkhtm\" (UniqueName: \"kubernetes.io/projected/0476eb7d-91e3-4f24-bfbc-245775c8e93b-kube-api-access-fkhtm\") pod \"0476eb7d-91e3-4f24-bfbc-245775c8e93b\" (UID: \"0476eb7d-91e3-4f24-bfbc-245775c8e93b\") " Oct 30 16:49:35.184776 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.184510 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0476eb7d-91e3-4f24-bfbc-245775c8e93b-credential-socket" pod "0476eb7d-91e3-4f24-bfbc-245775c8e93b" (UID: "0476eb7d-91e3-4f24-bfbc-245775c8e93b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:49:35.184776 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.184614 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0476eb7d-91e3-4f24-bfbc-245775c8e93b-workload-certs" pod "0476eb7d-91e3-4f24-bfbc-245775c8e93b" (UID: "0476eb7d-91e3-4f24-bfbc-245775c8e93b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:49:35.184776 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.184627 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0476eb7d-91e3-4f24-bfbc-245775c8e93b-istio-data" pod "0476eb7d-91e3-4f24-bfbc-245775c8e93b" (UID: "0476eb7d-91e3-4f24-bfbc-245775c8e93b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:49:35.185028 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.184835 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0476eb7d-91e3-4f24-bfbc-245775c8e93b-workload-socket" pod "0476eb7d-91e3-4f24-bfbc-245775c8e93b" (UID: "0476eb7d-91e3-4f24-bfbc-245775c8e93b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:49:35.185028 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.184925 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0476eb7d-91e3-4f24-bfbc-245775c8e93b-istiod-ca-cert" pod "0476eb7d-91e3-4f24-bfbc-245775c8e93b" (UID: "0476eb7d-91e3-4f24-bfbc-245775c8e93b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:49:35.185473 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.185453 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0476eb7d-91e3-4f24-bfbc-245775c8e93b-workload-certs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:49:35.185562 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.185478 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0476eb7d-91e3-4f24-bfbc-245775c8e93b-istiod-ca-cert\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:49:35.185562 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.185491 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7ae7018-798b-4a3a-afef-b2ed1a13b05f-istio-envoy\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:49:35.185562 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.185504 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7ae7018-798b-4a3a-afef-b2ed1a13b05f-workload-certs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:49:35.185562 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.185517 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7ae7018-798b-4a3a-afef-b2ed1a13b05f-istio-podinfo\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:49:35.185562 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.185529 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7ae7018-798b-4a3a-afef-b2ed1a13b05f-istiod-ca-cert\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:49:35.185562 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.185544 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0476eb7d-91e3-4f24-bfbc-245775c8e93b-workload-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:49:35.185562 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.185557 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q99cb\" (UniqueName: \"kubernetes.io/projected/c7ae7018-798b-4a3a-afef-b2ed1a13b05f-kube-api-access-q99cb\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:49:35.185865 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.185572 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7ae7018-798b-4a3a-afef-b2ed1a13b05f-credential-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:49:35.185865 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.185585 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7ae7018-798b-4a3a-afef-b2ed1a13b05f-istio-token\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:49:35.185865 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.185599 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7ae7018-798b-4a3a-afef-b2ed1a13b05f-workload-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:49:35.185865 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.185612 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0476eb7d-91e3-4f24-bfbc-245775c8e93b-credential-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:49:35.185865 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.185625 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7ae7018-798b-4a3a-afef-b2ed1a13b05f-istio-data\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:49:35.185865 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.185637 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0476eb7d-91e3-4f24-bfbc-245775c8e93b-istio-data\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:49:35.186632 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.186610 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0476eb7d-91e3-4f24-bfbc-245775c8e93b-istio-token" pod "0476eb7d-91e3-4f24-bfbc-245775c8e93b" (UID: "0476eb7d-91e3-4f24-bfbc-245775c8e93b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:49:35.186843 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.186800 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0476eb7d-91e3-4f24-bfbc-245775c8e93b-istio-podinfo" pod "0476eb7d-91e3-4f24-bfbc-245775c8e93b" (UID: "0476eb7d-91e3-4f24-bfbc-245775c8e93b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:49:35.187455 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.187437 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0476eb7d-91e3-4f24-bfbc-245775c8e93b-istio-envoy" pod "0476eb7d-91e3-4f24-bfbc-245775c8e93b" (UID: "0476eb7d-91e3-4f24-bfbc-245775c8e93b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:49:35.187514 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.187458 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0476eb7d-91e3-4f24-bfbc-245775c8e93b-kube-api-access-fkhtm" pod "0476eb7d-91e3-4f24-bfbc-245775c8e93b" (UID: "0476eb7d-91e3-4f24-bfbc-245775c8e93b"). InnerVolumeSpecName "kube-api-access-fkhtm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:49:35.282520 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.274965 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-76cccb-m4q75"] Oct 30 16:49:35.287250 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.286889 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0476eb7d-91e3-4f24-bfbc-245775c8e93b-istio-envoy\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:49:35.287250 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.286918 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0476eb7d-91e3-4f24-bfbc-245775c8e93b-istio-token\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:49:35.287250 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.286933 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkhtm\" (UniqueName: \"kubernetes.io/projected/0476eb7d-91e3-4f24-bfbc-245775c8e93b-kube-api-access-fkhtm\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:49:35.287250 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.286947 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0476eb7d-91e3-4f24-bfbc-245775c8e93b-istio-podinfo\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:49:35.437848 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.437721 4493 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 16:49:35.437848 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.437779 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn" podUID="84009ea2-157b-45c6-b4bd-6df55fc7c72d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 16:49:35.455317 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.455287 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t" Oct 30 16:49:35.489642 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.489619 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad417aab-5686-4ae2-9163-91af4a79b356-kube-api-access-lkh2h\" (UniqueName: \"kubernetes.io/projected/ad417aab-5686-4ae2-9163-91af4a79b356-kube-api-access-lkh2h\") pod \"ad417aab-5686-4ae2-9163-91af4a79b356\" (UID: \"ad417aab-5686-4ae2-9163-91af4a79b356\") " Oct 30 16:49:35.489776 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.489677 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad417aab-5686-4ae2-9163-91af4a79b356-istio-token\" (UniqueName: \"kubernetes.io/projected/ad417aab-5686-4ae2-9163-91af4a79b356-istio-token\") pod \"ad417aab-5686-4ae2-9163-91af4a79b356\" (UID: \"ad417aab-5686-4ae2-9163-91af4a79b356\") " Oct 30 16:49:35.489776 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.489728 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad417aab-5686-4ae2-9163-91af4a79b356-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad417aab-5686-4ae2-9163-91af4a79b356-workload-socket\") pod \"ad417aab-5686-4ae2-9163-91af4a79b356\" (UID: \"ad417aab-5686-4ae2-9163-91af4a79b356\") " Oct 30 16:49:35.489881 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.489776 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad417aab-5686-4ae2-9163-91af4a79b356-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad417aab-5686-4ae2-9163-91af4a79b356-credential-socket\") pod \"ad417aab-5686-4ae2-9163-91af4a79b356\" (UID: \"ad417aab-5686-4ae2-9163-91af4a79b356\") " Oct 30 16:49:35.489881 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.489845 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad417aab-5686-4ae2-9163-91af4a79b356-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad417aab-5686-4ae2-9163-91af4a79b356-istiod-ca-cert\") pod \"ad417aab-5686-4ae2-9163-91af4a79b356\" (UID: \"ad417aab-5686-4ae2-9163-91af4a79b356\") " Oct 30 16:49:35.489982 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.489964 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad417aab-5686-4ae2-9163-91af4a79b356-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad417aab-5686-4ae2-9163-91af4a79b356-workload-certs\") pod \"ad417aab-5686-4ae2-9163-91af4a79b356\" (UID: \"ad417aab-5686-4ae2-9163-91af4a79b356\") " Oct 30 16:49:35.490044 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.490000 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ad417aab-5686-4ae2-9163-91af4a79b356-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad417aab-5686-4ae2-9163-91af4a79b356-istio-podinfo\") pod \"ad417aab-5686-4ae2-9163-91af4a79b356\" (UID: \"ad417aab-5686-4ae2-9163-91af4a79b356\") " Oct 30 16:49:35.490112 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.490096 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad417aab-5686-4ae2-9163-91af4a79b356-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad417aab-5686-4ae2-9163-91af4a79b356-istio-data\") pod \"ad417aab-5686-4ae2-9163-91af4a79b356\" (UID: \"ad417aab-5686-4ae2-9163-91af4a79b356\") " Oct 30 16:49:35.490164 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.490098 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad417aab-5686-4ae2-9163-91af4a79b356-workload-socket" pod "ad417aab-5686-4ae2-9163-91af4a79b356" (UID: "ad417aab-5686-4ae2-9163-91af4a79b356"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:49:35.490164 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.490084 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad417aab-5686-4ae2-9163-91af4a79b356-credential-socket" pod "ad417aab-5686-4ae2-9163-91af4a79b356" (UID: "ad417aab-5686-4ae2-9163-91af4a79b356"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:49:35.490281 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.490157 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad417aab-5686-4ae2-9163-91af4a79b356-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad417aab-5686-4ae2-9163-91af4a79b356-istio-envoy\") pod \"ad417aab-5686-4ae2-9163-91af4a79b356\" (UID: \"ad417aab-5686-4ae2-9163-91af4a79b356\") " Oct 30 16:49:35.490442 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.490274 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad417aab-5686-4ae2-9163-91af4a79b356-istiod-ca-cert" pod "ad417aab-5686-4ae2-9163-91af4a79b356" (UID: "ad417aab-5686-4ae2-9163-91af4a79b356"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:49:35.490442 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.490319 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad417aab-5686-4ae2-9163-91af4a79b356-istio-data" pod "ad417aab-5686-4ae2-9163-91af4a79b356" (UID: "ad417aab-5686-4ae2-9163-91af4a79b356"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:49:35.490442 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.490272 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad417aab-5686-4ae2-9163-91af4a79b356-workload-certs" pod "ad417aab-5686-4ae2-9163-91af4a79b356" (UID: "ad417aab-5686-4ae2-9163-91af4a79b356"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:49:35.490895 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.490850 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad417aab-5686-4ae2-9163-91af4a79b356-workload-certs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:49:35.490895 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.490877 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad417aab-5686-4ae2-9163-91af4a79b356-istio-data\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:49:35.490895 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.490893 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad417aab-5686-4ae2-9163-91af4a79b356-workload-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:49:35.491218 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.490906 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad417aab-5686-4ae2-9163-91af4a79b356-credential-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:49:35.491218 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.490918 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad417aab-5686-4ae2-9163-91af4a79b356-istiod-ca-cert\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:49:35.491995 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.491975 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad417aab-5686-4ae2-9163-91af4a79b356-kube-api-access-lkh2h" pod "ad417aab-5686-4ae2-9163-91af4a79b356" (UID: "ad417aab-5686-4ae2-9163-91af4a79b356"). InnerVolumeSpecName "kube-api-access-lkh2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:49:35.492112 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.492060 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad417aab-5686-4ae2-9163-91af4a79b356-istio-token" pod "ad417aab-5686-4ae2-9163-91af4a79b356" (UID: "ad417aab-5686-4ae2-9163-91af4a79b356"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:49:35.492247 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.492228 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ad417aab-5686-4ae2-9163-91af4a79b356-istio-podinfo" pod "ad417aab-5686-4ae2-9163-91af4a79b356" (UID: "ad417aab-5686-4ae2-9163-91af4a79b356"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:49:35.492633 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.492613 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad417aab-5686-4ae2-9163-91af4a79b356-istio-envoy" pod "ad417aab-5686-4ae2-9163-91af4a79b356" (UID: "ad417aab-5686-4ae2-9163-91af4a79b356"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:49:35.592474 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.592436 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad417aab-5686-4ae2-9163-91af4a79b356-istio-envoy\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:49:35.592474 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.592467 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkh2h\" (UniqueName: \"kubernetes.io/projected/ad417aab-5686-4ae2-9163-91af4a79b356-kube-api-access-lkh2h\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:49:35.592649 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.592480 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad417aab-5686-4ae2-9163-91af4a79b356-istio-token\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:49:35.592649 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.592493 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad417aab-5686-4ae2-9163-91af4a79b356-istio-podinfo\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:49:35.659915 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.659882 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-76cccb-m4q75" event={"ID":"232a337f-6b37-4c14-8d9b-fabda3c1f93b","Type":"ContainerStarted","Data":"0ecc74b6275225a68a2a4d76183555daa06dd9ae80797214ce31f1f506c91418"} Oct 30 16:49:35.660593 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.660561 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-76cccb-m4q75" event={"ID":"232a337f-6b37-4c14-8d9b-fabda3c1f93b","Type":"ContainerStarted","Data":"a33c307e5891d41c856f1482a65b02fd88b5f599fa10ac1f8a9e60dfb717ee88"} Oct 30 16:49:35.661409 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.661383 4493 generic.go:356] "Generic (PLEG): container finished" podID="0476eb7d-91e3-4f24-bfbc-245775c8e93b" containerID="97a8e4b2ca4d74a421648cb3113f545ac021c7a7ca3deead0f96b8539a6f5358" exitCode=0 Oct 30 16:49:35.661520 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.661437 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-d9rhc" Oct 30 16:49:35.661520 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.661505 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-d9rhc" event={"ID":"0476eb7d-91e3-4f24-bfbc-245775c8e93b","Type":"ContainerDied","Data":"97a8e4b2ca4d74a421648cb3113f545ac021c7a7ca3deead0f96b8539a6f5358"} Oct 30 16:49:35.661636 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.661547 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-d9rhc" event={"ID":"0476eb7d-91e3-4f24-bfbc-245775c8e93b","Type":"ContainerDied","Data":"6e33692044b6afe333d583e9f6f9bafef51cf3261a33bbc33fa06592dee278b3"} Oct 30 16:49:35.661636 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.661580 4493 scope.go:122] "RemoveContainer" containerID="97a8e4b2ca4d74a421648cb3113f545ac021c7a7ca3deead0f96b8539a6f5358" Oct 30 16:49:35.663863 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.663848 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2" Oct 30 16:49:35.663977 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.663866 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2" event={"ID":"c7ae7018-798b-4a3a-afef-b2ed1a13b05f","Type":"ContainerDied","Data":"77995f75476d1b422cb7d950bac96dace02d103308e06c8c1b9649b5c577b969"} Oct 30 16:49:35.663977 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.663844 4493 generic.go:356] "Generic (PLEG): container finished" podID="c7ae7018-798b-4a3a-afef-b2ed1a13b05f" containerID="77995f75476d1b422cb7d950bac96dace02d103308e06c8c1b9649b5c577b969" exitCode=0 Oct 30 16:49:35.664275 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.664053 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2" event={"ID":"c7ae7018-798b-4a3a-afef-b2ed1a13b05f","Type":"ContainerDied","Data":"de9909497d0546b2a0f5f881fde3cdc0d52397ce9b48048cac6f9586467bd377"} Oct 30 16:49:35.672903 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.672878 4493 generic.go:356] "Generic (PLEG): container finished" podID="ad417aab-5686-4ae2-9163-91af4a79b356" containerID="d266cdc2c7712fcb40ea4d7125601d5ddf3af201f1a5c5b8fd9217fd245504ef" exitCode=0 Oct 30 16:49:35.673060 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.673046 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t" event={"ID":"ad417aab-5686-4ae2-9163-91af4a79b356","Type":"ContainerDied","Data":"d266cdc2c7712fcb40ea4d7125601d5ddf3af201f1a5c5b8fd9217fd245504ef"} Oct 30 16:49:35.673117 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.673083 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t" event={"ID":"ad417aab-5686-4ae2-9163-91af4a79b356","Type":"ContainerDied","Data":"a8620de0a9e9313eb120bb19036cfdb35cad98bd9cd63f2764eb33e27aa559d1"} Oct 30 16:49:35.673173 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.673142 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t" Oct 30 16:49:35.698588 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.698527 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-76cccb-m4q75" podStartSLOduration=1.698515681 podStartE2EDuration="1.698515681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:49:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:49:35.696517995 +0000 UTC m=+1602.923693096" watchObservedRunningTime="2026-10-30 16:49:35.698515681 +0000 UTC m=+1602.925690782" Oct 30 16:49:35.746824 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.743917 4493 scope.go:122] "RemoveContainer" containerID="97a8e4b2ca4d74a421648cb3113f545ac021c7a7ca3deead0f96b8539a6f5358" Oct 30 16:49:35.753846 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:49:35.747241 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97a8e4b2ca4d74a421648cb3113f545ac021c7a7ca3deead0f96b8539a6f5358\": container with ID starting with 97a8e4b2ca4d74a421648cb3113f545ac021c7a7ca3deead0f96b8539a6f5358 not found: ID does not exist" containerID="97a8e4b2ca4d74a421648cb3113f545ac021c7a7ca3deead0f96b8539a6f5358" Oct 30 16:49:35.753846 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.747289 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97a8e4b2ca4d74a421648cb3113f545ac021c7a7ca3deead0f96b8539a6f5358"} err="failed to get container status \"97a8e4b2ca4d74a421648cb3113f545ac021c7a7ca3deead0f96b8539a6f5358\": rpc error: code = NotFound desc = could not find container \"97a8e4b2ca4d74a421648cb3113f545ac021c7a7ca3deead0f96b8539a6f5358\": container with ID starting with 97a8e4b2ca4d74a421648cb3113f545ac021c7a7ca3deead0f96b8539a6f5358 not found: ID does not exist" Oct 30 16:49:35.753846 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.747319 4493 scope.go:122] "RemoveContainer" containerID="77995f75476d1b422cb7d950bac96dace02d103308e06c8c1b9649b5c577b969" Oct 30 16:49:35.756570 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.756257 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t"] Oct 30 16:49:35.763229 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.763196 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-rdv2t"] Oct 30 16:49:35.764876 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.764192 4493 scope.go:122] "RemoveContainer" containerID="2b851adaac88aed5e816a099ff1adcf586423cae5811e65fa657dd00ed2b18ae" Oct 30 16:49:35.776275 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.774849 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-d9rhc"] Oct 30 16:49:35.782050 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.779904 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-d9rhc"] Oct 30 16:49:35.807174 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.807136 4493 scope.go:122] "RemoveContainer" containerID="77995f75476d1b422cb7d950bac96dace02d103308e06c8c1b9649b5c577b969" Oct 30 16:49:35.807739 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:49:35.807712 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77995f75476d1b422cb7d950bac96dace02d103308e06c8c1b9649b5c577b969\": container with ID starting with 77995f75476d1b422cb7d950bac96dace02d103308e06c8c1b9649b5c577b969 not found: ID does not exist" containerID="77995f75476d1b422cb7d950bac96dace02d103308e06c8c1b9649b5c577b969" Oct 30 16:49:35.808372 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.807753 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77995f75476d1b422cb7d950bac96dace02d103308e06c8c1b9649b5c577b969"} err="failed to get container status \"77995f75476d1b422cb7d950bac96dace02d103308e06c8c1b9649b5c577b969\": rpc error: code = NotFound desc = could not find container \"77995f75476d1b422cb7d950bac96dace02d103308e06c8c1b9649b5c577b969\": container with ID starting with 77995f75476d1b422cb7d950bac96dace02d103308e06c8c1b9649b5c577b969 not found: ID does not exist" Oct 30 16:49:35.808372 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.807778 4493 scope.go:122] "RemoveContainer" containerID="2b851adaac88aed5e816a099ff1adcf586423cae5811e65fa657dd00ed2b18ae" Oct 30 16:49:35.808537 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:49:35.808504 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b851adaac88aed5e816a099ff1adcf586423cae5811e65fa657dd00ed2b18ae\": container with ID starting with 2b851adaac88aed5e816a099ff1adcf586423cae5811e65fa657dd00ed2b18ae not found: ID does not exist" containerID="2b851adaac88aed5e816a099ff1adcf586423cae5811e65fa657dd00ed2b18ae" Oct 30 16:49:35.808601 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.808534 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b851adaac88aed5e816a099ff1adcf586423cae5811e65fa657dd00ed2b18ae"} err="failed to get container status \"2b851adaac88aed5e816a099ff1adcf586423cae5811e65fa657dd00ed2b18ae\": rpc error: code = NotFound desc = could not find container \"2b851adaac88aed5e816a099ff1adcf586423cae5811e65fa657dd00ed2b18ae\": container with ID starting with 2b851adaac88aed5e816a099ff1adcf586423cae5811e65fa657dd00ed2b18ae not found: ID does not exist" Oct 30 16:49:35.808601 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.808554 4493 scope.go:122] "RemoveContainer" containerID="d266cdc2c7712fcb40ea4d7125601d5ddf3af201f1a5c5b8fd9217fd245504ef" Oct 30 16:49:35.810711 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.810576 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2"] Oct 30 16:49:35.815930 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.815880 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2k4f2"] Oct 30 16:49:35.822952 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.822628 4493 scope.go:122] "RemoveContainer" containerID="d266cdc2c7712fcb40ea4d7125601d5ddf3af201f1a5c5b8fd9217fd245504ef" Oct 30 16:49:35.823170 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:49:35.823090 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d266cdc2c7712fcb40ea4d7125601d5ddf3af201f1a5c5b8fd9217fd245504ef\": container with ID starting with d266cdc2c7712fcb40ea4d7125601d5ddf3af201f1a5c5b8fd9217fd245504ef not found: ID does not exist" containerID="d266cdc2c7712fcb40ea4d7125601d5ddf3af201f1a5c5b8fd9217fd245504ef" Oct 30 16:49:35.823170 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.823114 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d266cdc2c7712fcb40ea4d7125601d5ddf3af201f1a5c5b8fd9217fd245504ef"} err="failed to get container status \"d266cdc2c7712fcb40ea4d7125601d5ddf3af201f1a5c5b8fd9217fd245504ef\": rpc error: code = NotFound desc = could not find container \"d266cdc2c7712fcb40ea4d7125601d5ddf3af201f1a5c5b8fd9217fd245504ef\": container with ID starting with d266cdc2c7712fcb40ea4d7125601d5ddf3af201f1a5c5b8fd9217fd245504ef not found: ID does not exist" Oct 30 16:49:35.948955 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:35.948893 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn" Oct 30 16:49:36.005192 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:36.005153 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84009ea2-157b-45c6-b4bd-6df55fc7c72d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84009ea2-157b-45c6-b4bd-6df55fc7c72d-workload-socket\") pod \"84009ea2-157b-45c6-b4bd-6df55fc7c72d\" (UID: \"84009ea2-157b-45c6-b4bd-6df55fc7c72d\") " Oct 30 16:49:36.005378 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:36.005216 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84009ea2-157b-45c6-b4bd-6df55fc7c72d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84009ea2-157b-45c6-b4bd-6df55fc7c72d-istiod-ca-cert\") pod \"84009ea2-157b-45c6-b4bd-6df55fc7c72d\" (UID: \"84009ea2-157b-45c6-b4bd-6df55fc7c72d\") " Oct 30 16:49:36.005378 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:36.005259 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84009ea2-157b-45c6-b4bd-6df55fc7c72d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84009ea2-157b-45c6-b4bd-6df55fc7c72d-credential-socket\") pod \"84009ea2-157b-45c6-b4bd-6df55fc7c72d\" (UID: \"84009ea2-157b-45c6-b4bd-6df55fc7c72d\") " Oct 30 16:49:36.005503 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:36.005375 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84009ea2-157b-45c6-b4bd-6df55fc7c72d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84009ea2-157b-45c6-b4bd-6df55fc7c72d-workload-certs\") pod \"84009ea2-157b-45c6-b4bd-6df55fc7c72d\" (UID: \"84009ea2-157b-45c6-b4bd-6df55fc7c72d\") " Oct 30 16:49:36.005503 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:36.005449 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84009ea2-157b-45c6-b4bd-6df55fc7c72d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84009ea2-157b-45c6-b4bd-6df55fc7c72d-istio-data\") pod \"84009ea2-157b-45c6-b4bd-6df55fc7c72d\" (UID: \"84009ea2-157b-45c6-b4bd-6df55fc7c72d\") " Oct 30 16:49:36.005503 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:36.005451 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84009ea2-157b-45c6-b4bd-6df55fc7c72d-workload-socket" pod "84009ea2-157b-45c6-b4bd-6df55fc7c72d" (UID: "84009ea2-157b-45c6-b4bd-6df55fc7c72d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:49:36.005503 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:36.005490 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84009ea2-157b-45c6-b4bd-6df55fc7c72d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84009ea2-157b-45c6-b4bd-6df55fc7c72d-istio-envoy\") pod \"84009ea2-157b-45c6-b4bd-6df55fc7c72d\" (UID: \"84009ea2-157b-45c6-b4bd-6df55fc7c72d\") " Oct 30 16:49:36.005720 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:36.005530 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/84009ea2-157b-45c6-b4bd-6df55fc7c72d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84009ea2-157b-45c6-b4bd-6df55fc7c72d-istio-podinfo\") pod \"84009ea2-157b-45c6-b4bd-6df55fc7c72d\" (UID: \"84009ea2-157b-45c6-b4bd-6df55fc7c72d\") " Oct 30 16:49:36.005720 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:36.005585 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84009ea2-157b-45c6-b4bd-6df55fc7c72d-istio-token\" (UniqueName: \"kubernetes.io/projected/84009ea2-157b-45c6-b4bd-6df55fc7c72d-istio-token\") pod \"84009ea2-157b-45c6-b4bd-6df55fc7c72d\" (UID: \"84009ea2-157b-45c6-b4bd-6df55fc7c72d\") " Oct 30 16:49:36.005720 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:36.005623 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84009ea2-157b-45c6-b4bd-6df55fc7c72d-kube-api-access-kl95m\" (UniqueName: \"kubernetes.io/projected/84009ea2-157b-45c6-b4bd-6df55fc7c72d-kube-api-access-kl95m\") pod \"84009ea2-157b-45c6-b4bd-6df55fc7c72d\" (UID: \"84009ea2-157b-45c6-b4bd-6df55fc7c72d\") " Oct 30 16:49:36.005720 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:36.005659 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84009ea2-157b-45c6-b4bd-6df55fc7c72d-credential-socket" pod "84009ea2-157b-45c6-b4bd-6df55fc7c72d" (UID: "84009ea2-157b-45c6-b4bd-6df55fc7c72d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:49:36.006020 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:36.005978 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84009ea2-157b-45c6-b4bd-6df55fc7c72d-istiod-ca-cert" pod "84009ea2-157b-45c6-b4bd-6df55fc7c72d" (UID: "84009ea2-157b-45c6-b4bd-6df55fc7c72d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:49:36.006323 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:36.006299 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84009ea2-157b-45c6-b4bd-6df55fc7c72d-workload-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:49:36.006426 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:36.006327 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84009ea2-157b-45c6-b4bd-6df55fc7c72d-istiod-ca-cert\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:49:36.006426 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:36.006342 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84009ea2-157b-45c6-b4bd-6df55fc7c72d-credential-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:49:36.007322 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:36.007227 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84009ea2-157b-45c6-b4bd-6df55fc7c72d-workload-certs" pod "84009ea2-157b-45c6-b4bd-6df55fc7c72d" (UID: "84009ea2-157b-45c6-b4bd-6df55fc7c72d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:49:36.007322 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:36.007291 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84009ea2-157b-45c6-b4bd-6df55fc7c72d-istio-data" pod "84009ea2-157b-45c6-b4bd-6df55fc7c72d" (UID: "84009ea2-157b-45c6-b4bd-6df55fc7c72d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:49:36.009196 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:36.009167 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84009ea2-157b-45c6-b4bd-6df55fc7c72d-kube-api-access-kl95m" pod "84009ea2-157b-45c6-b4bd-6df55fc7c72d" (UID: "84009ea2-157b-45c6-b4bd-6df55fc7c72d"). InnerVolumeSpecName "kube-api-access-kl95m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:49:36.010570 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:36.010535 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84009ea2-157b-45c6-b4bd-6df55fc7c72d-istio-token" pod "84009ea2-157b-45c6-b4bd-6df55fc7c72d" (UID: "84009ea2-157b-45c6-b4bd-6df55fc7c72d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:49:36.010853 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:36.010813 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/84009ea2-157b-45c6-b4bd-6df55fc7c72d-istio-podinfo" pod "84009ea2-157b-45c6-b4bd-6df55fc7c72d" (UID: "84009ea2-157b-45c6-b4bd-6df55fc7c72d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:49:36.033850 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:36.024704 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84009ea2-157b-45c6-b4bd-6df55fc7c72d-istio-envoy" pod "84009ea2-157b-45c6-b4bd-6df55fc7c72d" (UID: "84009ea2-157b-45c6-b4bd-6df55fc7c72d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:49:36.107546 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:36.107514 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84009ea2-157b-45c6-b4bd-6df55fc7c72d-workload-certs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:49:36.107546 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:36.107544 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84009ea2-157b-45c6-b4bd-6df55fc7c72d-istio-data\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:49:36.107754 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:36.107558 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84009ea2-157b-45c6-b4bd-6df55fc7c72d-istio-envoy\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:49:36.107754 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:36.107571 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84009ea2-157b-45c6-b4bd-6df55fc7c72d-istio-podinfo\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:49:36.107754 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:36.107581 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84009ea2-157b-45c6-b4bd-6df55fc7c72d-istio-token\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:49:36.107754 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:36.107594 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kl95m\" (UniqueName: \"kubernetes.io/projected/84009ea2-157b-45c6-b4bd-6df55fc7c72d-kube-api-access-kl95m\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:49:36.683217 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:36.683186 4493 generic.go:356] "Generic (PLEG): container finished" podID="84009ea2-157b-45c6-b4bd-6df55fc7c72d" containerID="8ebf076f93f55dae5e773e774ad3dc10673d4b61201803708b086c5436da423e" exitCode=0 Oct 30 16:49:36.683655 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:36.683234 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn" Oct 30 16:49:36.683655 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:36.683274 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn" event={"ID":"84009ea2-157b-45c6-b4bd-6df55fc7c72d","Type":"ContainerDied","Data":"8ebf076f93f55dae5e773e774ad3dc10673d4b61201803708b086c5436da423e"} Oct 30 16:49:36.683655 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:36.683330 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn" event={"ID":"84009ea2-157b-45c6-b4bd-6df55fc7c72d","Type":"ContainerDied","Data":"3fa2d42ff03e1f548a6d0ea01bdf19510d2f3afa2fe170640c9bcc4322a34ca9"} Oct 30 16:49:36.683655 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:36.683361 4493 scope.go:122] "RemoveContainer" containerID="8ebf076f93f55dae5e773e774ad3dc10673d4b61201803708b086c5436da423e" Oct 30 16:49:36.684928 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:36.684906 4493 generic.go:356] "Generic (PLEG): container finished" podID="232a337f-6b37-4c14-8d9b-fabda3c1f93b" containerID="0ecc74b6275225a68a2a4d76183555daa06dd9ae80797214ce31f1f506c91418" exitCode=0 Oct 30 16:49:36.685027 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:36.684991 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-76cccb-m4q75" event={"ID":"232a337f-6b37-4c14-8d9b-fabda3c1f93b","Type":"ContainerDied","Data":"0ecc74b6275225a68a2a4d76183555daa06dd9ae80797214ce31f1f506c91418"} Oct 30 16:49:36.694166 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:36.694150 4493 scope.go:122] "RemoveContainer" containerID="8ebf076f93f55dae5e773e774ad3dc10673d4b61201803708b086c5436da423e" Oct 30 16:49:36.694419 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:49:36.694404 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ebf076f93f55dae5e773e774ad3dc10673d4b61201803708b086c5436da423e\": container with ID starting with 8ebf076f93f55dae5e773e774ad3dc10673d4b61201803708b086c5436da423e not found: ID does not exist" containerID="8ebf076f93f55dae5e773e774ad3dc10673d4b61201803708b086c5436da423e" Oct 30 16:49:36.694463 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:36.694428 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ebf076f93f55dae5e773e774ad3dc10673d4b61201803708b086c5436da423e"} err="failed to get container status \"8ebf076f93f55dae5e773e774ad3dc10673d4b61201803708b086c5436da423e\": rpc error: code = NotFound desc = could not find container \"8ebf076f93f55dae5e773e774ad3dc10673d4b61201803708b086c5436da423e\": container with ID starting with 8ebf076f93f55dae5e773e774ad3dc10673d4b61201803708b086c5436da423e not found: ID does not exist" Oct 30 16:49:36.730144 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:36.730107 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn"] Oct 30 16:49:36.739681 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:36.739658 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-2lnwn"] Oct 30 16:49:36.934251 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:36.934178 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0476eb7d-91e3-4f24-bfbc-245775c8e93b" path="/var/lib/kubelet/pods/0476eb7d-91e3-4f24-bfbc-245775c8e93b/volumes" Oct 30 16:49:36.934623 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:36.934610 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84009ea2-157b-45c6-b4bd-6df55fc7c72d" path="/var/lib/kubelet/pods/84009ea2-157b-45c6-b4bd-6df55fc7c72d/volumes" Oct 30 16:49:36.935129 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:36.935114 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad417aab-5686-4ae2-9163-91af4a79b356" path="/var/lib/kubelet/pods/ad417aab-5686-4ae2-9163-91af4a79b356/volumes" Oct 30 16:49:36.936005 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:36.935984 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7ae7018-798b-4a3a-afef-b2ed1a13b05f" path="/var/lib/kubelet/pods/c7ae7018-798b-4a3a-afef-b2ed1a13b05f/volumes" Oct 30 16:49:38.070809 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:38.070707 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-cfe9f0-76f945bc5f-wt6jt"] Oct 30 16:49:38.082859 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:38.071651 4493 cpu_manager.go:414] "containerMap: removing container" podUID="0476eb7d-91e3-4f24-bfbc-245775c8e93b" containerName="istio-proxy" Oct 30 16:49:38.082859 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:38.071675 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0476eb7d-91e3-4f24-bfbc-245775c8e93b" containerName="istio-proxy" Oct 30 16:49:38.082859 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:38.071726 4493 cpu_manager.go:414] "containerMap: removing container" podUID="84009ea2-157b-45c6-b4bd-6df55fc7c72d" containerName="istio-proxy" Oct 30 16:49:38.082859 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:38.071734 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84009ea2-157b-45c6-b4bd-6df55fc7c72d" containerName="istio-proxy" Oct 30 16:49:38.082859 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:38.071747 4493 cpu_manager.go:414] "containerMap: removing container" podUID="c7ae7018-798b-4a3a-afef-b2ed1a13b05f" containerName="istio-proxy" Oct 30 16:49:38.082859 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:38.071754 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7ae7018-798b-4a3a-afef-b2ed1a13b05f" containerName="istio-proxy" Oct 30 16:49:38.082859 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:38.071765 4493 cpu_manager.go:414] "containerMap: removing container" podUID="ad417aab-5686-4ae2-9163-91af4a79b356" containerName="istio-proxy" Oct 30 16:49:38.082859 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:38.071772 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad417aab-5686-4ae2-9163-91af4a79b356" containerName="istio-proxy" Oct 30 16:49:38.082859 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:38.071940 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="0476eb7d-91e3-4f24-bfbc-245775c8e93b" containerName="istio-proxy" Oct 30 16:49:38.082859 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:38.071959 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="84009ea2-157b-45c6-b4bd-6df55fc7c72d" containerName="istio-proxy" Oct 30 16:49:38.082859 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:38.071971 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7ae7018-798b-4a3a-afef-b2ed1a13b05f" containerName="istio-proxy" Oct 30 16:49:38.082859 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:38.071979 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad417aab-5686-4ae2-9163-91af4a79b356" containerName="istio-proxy" Oct 30 16:49:38.082859 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:38.071989 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7ae7018-798b-4a3a-afef-b2ed1a13b05f" containerName="istio-proxy" Oct 30 16:49:38.121243 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:38.121211 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-cfe9f0-76f945bc5f-wt6jt"] Oct 30 16:49:38.121680 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:38.121658 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cfe9f0-76f945bc5f-wt6jt" Oct 30 16:49:38.153090 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:38.153059 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-cfe9f0-dockercfg-n46q4\"" Oct 30 16:49:38.153310 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:38.153289 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-cfe9f0\"" Oct 30 16:49:38.171054 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:38.171032 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-cfe9f0-69b4bdf948-tvs7d"] Oct 30 16:49:38.171686 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:38.171663 4493 cpu_manager.go:414] "containerMap: removing container" podUID="c7ae7018-798b-4a3a-afef-b2ed1a13b05f" containerName="istio-proxy" Oct 30 16:49:38.171686 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:38.171686 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7ae7018-798b-4a3a-afef-b2ed1a13b05f" containerName="istio-proxy" Oct 30 16:49:38.173989 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:38.173972 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-76cccb-m4q75" Oct 30 16:49:38.175027 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:38.175009 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-cfe9f0-69b4bdf948-tvs7d" Oct 30 16:49:38.197378 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:38.197342 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-cfe9f0-69b4bdf948-tvs7d"] Oct 30 16:49:38.283193 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:38.283161 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/232a337f-6b37-4c14-8d9b-fabda3c1f93b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/232a337f-6b37-4c14-8d9b-fabda3c1f93b-maas-api-openshift-ca\") pod \"232a337f-6b37-4c14-8d9b-fabda3c1f93b\" (UID: \"232a337f-6b37-4c14-8d9b-fabda3c1f93b\") " Oct 30 16:49:38.283362 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:38.283278 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/232a337f-6b37-4c14-8d9b-fabda3c1f93b-kube-api-access-jkjmx\" (UniqueName: \"kubernetes.io/projected/232a337f-6b37-4c14-8d9b-fabda3c1f93b-kube-api-access-jkjmx\") pod \"232a337f-6b37-4c14-8d9b-fabda3c1f93b\" (UID: \"232a337f-6b37-4c14-8d9b-fabda3c1f93b\") " Oct 30 16:49:38.283543 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:38.283525 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/232a337f-6b37-4c14-8d9b-fabda3c1f93b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/232a337f-6b37-4c14-8d9b-fabda3c1f93b-maas-api-xks-ca\") pod \"232a337f-6b37-4c14-8d9b-fabda3c1f93b\" (UID: \"232a337f-6b37-4c14-8d9b-fabda3c1f93b\") " Oct 30 16:49:38.283670 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:38.283529 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/232a337f-6b37-4c14-8d9b-fabda3c1f93b-maas-api-openshift-ca" pod "232a337f-6b37-4c14-8d9b-fabda3c1f93b" (UID: "232a337f-6b37-4c14-8d9b-fabda3c1f93b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:49:38.283839 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:38.283821 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gh97\" (UniqueName: \"kubernetes.io/projected/5943c777-6afd-4066-b448-1f273de792f0-kube-api-access-5gh97\") pod \"payload-pre-processing-e2e-ipp-a-cfe9f0-76f945bc5f-wt6jt\" (UID: \"5943c777-6afd-4066-b448-1f273de792f0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cfe9f0-76f945bc5f-wt6jt" Oct 30 16:49:38.283936 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:38.283851 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5943c777-6afd-4066-b448-1f273de792f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-cfe9f0-76f945bc5f-wt6jt\" (UID: \"5943c777-6afd-4066-b448-1f273de792f0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cfe9f0-76f945bc5f-wt6jt" Oct 30 16:49:38.283936 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:38.283878 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn9gd\" (UniqueName: \"kubernetes.io/projected/13cca9a3-0b99-497e-9abd-22b739f7a1b3-kube-api-access-bn9gd\") pod \"payload-processing-e2e-ipp-a-cfe9f0-69b4bdf948-tvs7d\" (UID: \"13cca9a3-0b99-497e-9abd-22b739f7a1b3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-cfe9f0-69b4bdf948-tvs7d" Oct 30 16:49:38.284341 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:38.284316 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13cca9a3-0b99-497e-9abd-22b739f7a1b3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-cfe9f0-69b4bdf948-tvs7d\" (UID: \"13cca9a3-0b99-497e-9abd-22b739f7a1b3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-cfe9f0-69b4bdf948-tvs7d" Oct 30 16:49:38.284494 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:38.284476 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/232a337f-6b37-4c14-8d9b-fabda3c1f93b-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:49:38.285655 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:38.285634 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/232a337f-6b37-4c14-8d9b-fabda3c1f93b-maas-api-xks-ca" pod "232a337f-6b37-4c14-8d9b-fabda3c1f93b" (UID: "232a337f-6b37-4c14-8d9b-fabda3c1f93b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:49:38.285655 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:38.285647 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/232a337f-6b37-4c14-8d9b-fabda3c1f93b-kube-api-access-jkjmx" pod "232a337f-6b37-4c14-8d9b-fabda3c1f93b" (UID: "232a337f-6b37-4c14-8d9b-fabda3c1f93b"). InnerVolumeSpecName "kube-api-access-jkjmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:49:38.386221 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:38.386128 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gh97\" (UniqueName: \"kubernetes.io/projected/5943c777-6afd-4066-b448-1f273de792f0-kube-api-access-5gh97\") pod \"payload-pre-processing-e2e-ipp-a-cfe9f0-76f945bc5f-wt6jt\" (UID: \"5943c777-6afd-4066-b448-1f273de792f0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cfe9f0-76f945bc5f-wt6jt" Oct 30 16:49:38.386221 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:38.386180 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5943c777-6afd-4066-b448-1f273de792f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-cfe9f0-76f945bc5f-wt6jt\" (UID: \"5943c777-6afd-4066-b448-1f273de792f0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cfe9f0-76f945bc5f-wt6jt" Oct 30 16:49:38.386221 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:38.386208 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn9gd\" (UniqueName: \"kubernetes.io/projected/13cca9a3-0b99-497e-9abd-22b739f7a1b3-kube-api-access-bn9gd\") pod \"payload-processing-e2e-ipp-a-cfe9f0-69b4bdf948-tvs7d\" (UID: \"13cca9a3-0b99-497e-9abd-22b739f7a1b3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-cfe9f0-69b4bdf948-tvs7d" Oct 30 16:49:38.386444 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:38.386380 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13cca9a3-0b99-497e-9abd-22b739f7a1b3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-cfe9f0-69b4bdf948-tvs7d\" (UID: \"13cca9a3-0b99-497e-9abd-22b739f7a1b3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-cfe9f0-69b4bdf948-tvs7d" Oct 30 16:49:38.386506 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:38.386491 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/232a337f-6b37-4c14-8d9b-fabda3c1f93b-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:49:38.386541 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:38.386510 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkjmx\" (UniqueName: \"kubernetes.io/projected/232a337f-6b37-4c14-8d9b-fabda3c1f93b-kube-api-access-jkjmx\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:49:38.386982 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:38.386964 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5943c777-6afd-4066-b448-1f273de792f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-cfe9f0-76f945bc5f-wt6jt\" (UID: \"5943c777-6afd-4066-b448-1f273de792f0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cfe9f0-76f945bc5f-wt6jt" Oct 30 16:49:38.387092 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:38.387075 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13cca9a3-0b99-497e-9abd-22b739f7a1b3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-cfe9f0-69b4bdf948-tvs7d\" (UID: \"13cca9a3-0b99-497e-9abd-22b739f7a1b3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-cfe9f0-69b4bdf948-tvs7d" Oct 30 16:49:38.407532 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:38.407507 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gh97\" (UniqueName: \"kubernetes.io/projected/5943c777-6afd-4066-b448-1f273de792f0-kube-api-access-5gh97\") pod \"payload-pre-processing-e2e-ipp-a-cfe9f0-76f945bc5f-wt6jt\" (UID: \"5943c777-6afd-4066-b448-1f273de792f0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cfe9f0-76f945bc5f-wt6jt" Oct 30 16:49:38.407659 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:38.407636 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn9gd\" (UniqueName: \"kubernetes.io/projected/13cca9a3-0b99-497e-9abd-22b739f7a1b3-kube-api-access-bn9gd\") pod \"payload-processing-e2e-ipp-a-cfe9f0-69b4bdf948-tvs7d\" (UID: \"13cca9a3-0b99-497e-9abd-22b739f7a1b3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-cfe9f0-69b4bdf948-tvs7d" Oct 30 16:49:38.472405 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:38.472372 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cfe9f0-76f945bc5f-wt6jt" Oct 30 16:49:38.488250 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:38.488225 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-cfe9f0-69b4bdf948-tvs7d" Oct 30 16:49:38.780070 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:38.780047 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-cfe9f0-76f945bc5f-wt6jt"] Oct 30 16:49:38.783037 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:49:38.783012 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5943c777_6afd_4066_b448_1f273de792f0.slice/crio-bc53880f3679df1b2f3e9cc1774f6333db97a0d085a5a6ff941a589d05b2d63d WatchSource:0}: Error finding container bc53880f3679df1b2f3e9cc1774f6333db97a0d085a5a6ff941a589d05b2d63d: Status 404 returned error can't find the container with id bc53880f3679df1b2f3e9cc1774f6333db97a0d085a5a6ff941a589d05b2d63d Oct 30 16:49:38.819896 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:38.819854 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-cfe9f0-69b4bdf948-tvs7d"] Oct 30 16:49:38.824952 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:49:38.824924 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13cca9a3_0b99_497e_9abd_22b739f7a1b3.slice/crio-40f96042024e35ef25c06c6021563c7c95c4f35e41ab00b29edef5b8c977cc86 WatchSource:0}: Error finding container 40f96042024e35ef25c06c6021563c7c95c4f35e41ab00b29edef5b8c977cc86: Status 404 returned error can't find the container with id 40f96042024e35ef25c06c6021563c7c95c4f35e41ab00b29edef5b8c977cc86 Oct 30 16:49:38.834249 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:38.833313 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-76cccb-m4q75" event={"ID":"232a337f-6b37-4c14-8d9b-fabda3c1f93b","Type":"ContainerDied","Data":"a33c307e5891d41c856f1482a65b02fd88b5f599fa10ac1f8a9e60dfb717ee88"} Oct 30 16:49:38.834249 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:38.833366 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a33c307e5891d41c856f1482a65b02fd88b5f599fa10ac1f8a9e60dfb717ee88" Oct 30 16:49:38.834249 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:38.833430 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-76cccb-m4q75" Oct 30 16:49:38.843246 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:38.843179 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cfe9f0-76f945bc5f-wt6jt" event={"ID":"5943c777-6afd-4066-b448-1f273de792f0","Type":"ContainerStarted","Data":"bc53880f3679df1b2f3e9cc1774f6333db97a0d085a5a6ff941a589d05b2d63d"} Oct 30 16:49:39.853856 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:39.853815 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cfe9f0-76f945bc5f-wt6jt" event={"ID":"5943c777-6afd-4066-b448-1f273de792f0","Type":"ContainerStarted","Data":"ca433fe9b7cbefa4da621d37fe9ef7a771a81bd2217c1d8a95cb52e381f2e10f"} Oct 30 16:49:39.854290 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:39.853893 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cfe9f0-76f945bc5f-wt6jt" Oct 30 16:49:39.855709 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:39.855682 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-cfe9f0-69b4bdf948-tvs7d" event={"ID":"13cca9a3-0b99-497e-9abd-22b739f7a1b3","Type":"ContainerStarted","Data":"ace9d19c2f6fa5ffc2356c5e256c4783fa4285087ddc41cb1a530996e1465244"} Oct 30 16:49:39.855879 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:39.855734 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-cfe9f0-69b4bdf948-tvs7d" Oct 30 16:49:39.855879 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:39.855754 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-cfe9f0-69b4bdf948-tvs7d" event={"ID":"13cca9a3-0b99-497e-9abd-22b739f7a1b3","Type":"ContainerStarted","Data":"40f96042024e35ef25c06c6021563c7c95c4f35e41ab00b29edef5b8c977cc86"} Oct 30 16:49:39.891189 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:39.891148 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cfe9f0-76f945bc5f-wt6jt" podStartSLOduration=2.891136703 podStartE2EDuration="2.891136703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:49:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:49:39.889486296 +0000 UTC m=+1607.116661400" watchObservedRunningTime="2026-10-30 16:49:39.891136703 +0000 UTC m=+1607.118311802" Oct 30 16:49:39.931049 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:39.930992 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-cfe9f0-69b4bdf948-tvs7d" podStartSLOduration=1.930974064 podStartE2EDuration="1.930974064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:49:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:49:39.925642866 +0000 UTC m=+1607.152817979" watchObservedRunningTime="2026-10-30 16:49:39.930974064 +0000 UTC m=+1607.158149165" Oct 30 16:49:40.409296 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:40.409267 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk" Oct 30 16:49:40.513021 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:40.512985 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/320f1fb7-deb8-4a82-b468-f92066eca8a1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/320f1fb7-deb8-4a82-b468-f92066eca8a1-workload-certs\") pod \"320f1fb7-deb8-4a82-b468-f92066eca8a1\" (UID: \"320f1fb7-deb8-4a82-b468-f92066eca8a1\") " Oct 30 16:49:40.513021 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:40.513026 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/320f1fb7-deb8-4a82-b468-f92066eca8a1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/320f1fb7-deb8-4a82-b468-f92066eca8a1-istiod-ca-cert\") pod \"320f1fb7-deb8-4a82-b468-f92066eca8a1\" (UID: \"320f1fb7-deb8-4a82-b468-f92066eca8a1\") " Oct 30 16:49:40.513283 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:40.513054 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/320f1fb7-deb8-4a82-b468-f92066eca8a1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/320f1fb7-deb8-4a82-b468-f92066eca8a1-istio-envoy\") pod \"320f1fb7-deb8-4a82-b468-f92066eca8a1\" (UID: \"320f1fb7-deb8-4a82-b468-f92066eca8a1\") " Oct 30 16:49:40.513283 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:40.513090 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/320f1fb7-deb8-4a82-b468-f92066eca8a1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/320f1fb7-deb8-4a82-b468-f92066eca8a1-workload-socket\") pod \"320f1fb7-deb8-4a82-b468-f92066eca8a1\" (UID: \"320f1fb7-deb8-4a82-b468-f92066eca8a1\") " Oct 30 16:49:40.513283 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:40.513163 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/320f1fb7-deb8-4a82-b468-f92066eca8a1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/320f1fb7-deb8-4a82-b468-f92066eca8a1-istio-data\") pod \"320f1fb7-deb8-4a82-b468-f92066eca8a1\" (UID: \"320f1fb7-deb8-4a82-b468-f92066eca8a1\") " Oct 30 16:49:40.513447 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:40.513305 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/320f1fb7-deb8-4a82-b468-f92066eca8a1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/320f1fb7-deb8-4a82-b468-f92066eca8a1-credential-socket\") pod \"320f1fb7-deb8-4a82-b468-f92066eca8a1\" (UID: \"320f1fb7-deb8-4a82-b468-f92066eca8a1\") " Oct 30 16:49:40.513447 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:40.513337 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/320f1fb7-deb8-4a82-b468-f92066eca8a1-istio-token\" (UniqueName: \"kubernetes.io/projected/320f1fb7-deb8-4a82-b468-f92066eca8a1-istio-token\") pod \"320f1fb7-deb8-4a82-b468-f92066eca8a1\" (UID: \"320f1fb7-deb8-4a82-b468-f92066eca8a1\") " Oct 30 16:49:40.513447 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:40.513370 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/320f1fb7-deb8-4a82-b468-f92066eca8a1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/320f1fb7-deb8-4a82-b468-f92066eca8a1-istio-podinfo\") pod \"320f1fb7-deb8-4a82-b468-f92066eca8a1\" (UID: \"320f1fb7-deb8-4a82-b468-f92066eca8a1\") " Oct 30 16:49:40.513595 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:40.513484 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/320f1fb7-deb8-4a82-b468-f92066eca8a1-kube-api-access-mc8rj\" (UniqueName: \"kubernetes.io/projected/320f1fb7-deb8-4a82-b468-f92066eca8a1-kube-api-access-mc8rj\") pod \"320f1fb7-deb8-4a82-b468-f92066eca8a1\" (UID: \"320f1fb7-deb8-4a82-b468-f92066eca8a1\") " Oct 30 16:49:40.513595 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:40.513539 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/320f1fb7-deb8-4a82-b468-f92066eca8a1-workload-certs" pod "320f1fb7-deb8-4a82-b468-f92066eca8a1" (UID: "320f1fb7-deb8-4a82-b468-f92066eca8a1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:49:40.517861 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:40.513804 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/320f1fb7-deb8-4a82-b468-f92066eca8a1-istio-data" pod "320f1fb7-deb8-4a82-b468-f92066eca8a1" (UID: "320f1fb7-deb8-4a82-b468-f92066eca8a1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:49:40.517861 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:40.514114 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/320f1fb7-deb8-4a82-b468-f92066eca8a1-credential-socket" pod "320f1fb7-deb8-4a82-b468-f92066eca8a1" (UID: "320f1fb7-deb8-4a82-b468-f92066eca8a1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:49:40.517861 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:40.514173 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/320f1fb7-deb8-4a82-b468-f92066eca8a1-workload-socket" pod "320f1fb7-deb8-4a82-b468-f92066eca8a1" (UID: "320f1fb7-deb8-4a82-b468-f92066eca8a1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:49:40.517861 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:40.514204 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/320f1fb7-deb8-4a82-b468-f92066eca8a1-istiod-ca-cert" pod "320f1fb7-deb8-4a82-b468-f92066eca8a1" (UID: "320f1fb7-deb8-4a82-b468-f92066eca8a1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:49:40.517861 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:40.514209 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/320f1fb7-deb8-4a82-b468-f92066eca8a1-istio-data\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:49:40.517861 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:40.514229 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/320f1fb7-deb8-4a82-b468-f92066eca8a1-workload-certs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:49:40.517861 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:40.517366 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/320f1fb7-deb8-4a82-b468-f92066eca8a1-istio-token" pod "320f1fb7-deb8-4a82-b468-f92066eca8a1" (UID: "320f1fb7-deb8-4a82-b468-f92066eca8a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:49:40.517861 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:40.517468 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/320f1fb7-deb8-4a82-b468-f92066eca8a1-istio-envoy" pod "320f1fb7-deb8-4a82-b468-f92066eca8a1" (UID: "320f1fb7-deb8-4a82-b468-f92066eca8a1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:49:40.518376 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:40.518150 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/320f1fb7-deb8-4a82-b468-f92066eca8a1-istio-podinfo" pod "320f1fb7-deb8-4a82-b468-f92066eca8a1" (UID: "320f1fb7-deb8-4a82-b468-f92066eca8a1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:49:40.518450 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:40.518428 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/320f1fb7-deb8-4a82-b468-f92066eca8a1-kube-api-access-mc8rj" pod "320f1fb7-deb8-4a82-b468-f92066eca8a1" (UID: "320f1fb7-deb8-4a82-b468-f92066eca8a1"). InnerVolumeSpecName "kube-api-access-mc8rj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:49:40.616106 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:40.616068 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/320f1fb7-deb8-4a82-b468-f92066eca8a1-credential-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:49:40.616106 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:40.616107 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/320f1fb7-deb8-4a82-b468-f92066eca8a1-istio-token\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:49:40.616313 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:40.616124 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/320f1fb7-deb8-4a82-b468-f92066eca8a1-istio-podinfo\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:49:40.616313 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:40.616138 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mc8rj\" (UniqueName: \"kubernetes.io/projected/320f1fb7-deb8-4a82-b468-f92066eca8a1-kube-api-access-mc8rj\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:49:40.616313 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:40.616152 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/320f1fb7-deb8-4a82-b468-f92066eca8a1-istiod-ca-cert\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:49:40.616313 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:40.616165 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/320f1fb7-deb8-4a82-b468-f92066eca8a1-istio-envoy\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:49:40.616313 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:40.616177 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/320f1fb7-deb8-4a82-b468-f92066eca8a1-workload-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:49:40.864057 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:40.864020 4493 generic.go:356] "Generic (PLEG): container finished" podID="320f1fb7-deb8-4a82-b468-f92066eca8a1" containerID="24a87beb129a10f854bcadc0f61d76be963d76dbd791cb56e7a80316215ae361" exitCode=0 Oct 30 16:49:40.864584 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:40.864072 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk" Oct 30 16:49:40.864584 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:40.864111 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk" event={"ID":"320f1fb7-deb8-4a82-b468-f92066eca8a1","Type":"ContainerDied","Data":"24a87beb129a10f854bcadc0f61d76be963d76dbd791cb56e7a80316215ae361"} Oct 30 16:49:40.864584 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:40.864161 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk" event={"ID":"320f1fb7-deb8-4a82-b468-f92066eca8a1","Type":"ContainerDied","Data":"e42145e35cd0f23618ba288b6817f69341fa221ac529d72a6ae4c59e0bef9b4e"} Oct 30 16:49:40.864584 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:40.864189 4493 scope.go:122] "RemoveContainer" containerID="24a87beb129a10f854bcadc0f61d76be963d76dbd791cb56e7a80316215ae361" Oct 30 16:49:40.885485 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:40.885459 4493 scope.go:122] "RemoveContainer" containerID="24a87beb129a10f854bcadc0f61d76be963d76dbd791cb56e7a80316215ae361" Oct 30 16:49:40.886031 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:49:40.886000 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24a87beb129a10f854bcadc0f61d76be963d76dbd791cb56e7a80316215ae361\": container with ID starting with 24a87beb129a10f854bcadc0f61d76be963d76dbd791cb56e7a80316215ae361 not found: ID does not exist" containerID="24a87beb129a10f854bcadc0f61d76be963d76dbd791cb56e7a80316215ae361" Oct 30 16:49:40.886131 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:40.886047 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24a87beb129a10f854bcadc0f61d76be963d76dbd791cb56e7a80316215ae361"} err="failed to get container status \"24a87beb129a10f854bcadc0f61d76be963d76dbd791cb56e7a80316215ae361\": rpc error: code = NotFound desc = could not find container \"24a87beb129a10f854bcadc0f61d76be963d76dbd791cb56e7a80316215ae361\": container with ID starting with 24a87beb129a10f854bcadc0f61d76be963d76dbd791cb56e7a80316215ae361 not found: ID does not exist" Oct 30 16:49:40.915809 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:40.915560 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk"] Oct 30 16:49:40.925871 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:40.925838 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-r9tbk"] Oct 30 16:49:40.935501 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:40.935459 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="320f1fb7-deb8-4a82-b468-f92066eca8a1" path="/var/lib/kubelet/pods/320f1fb7-deb8-4a82-b468-f92066eca8a1/volumes" Oct 30 16:49:50.866451 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:50.866418 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-cfe9f0-69b4bdf948-tvs7d" Oct 30 16:49:50.867401 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:50.867376 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cfe9f0-76f945bc5f-wt6jt" Oct 30 16:49:50.940354 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:50.940323 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-76cccb-m4q75"] Oct 30 16:49:50.940354 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:50.940360 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-76cccb-m4q75"] Oct 30 16:49:52.934299 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:52.934269 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="232a337f-6b37-4c14-8d9b-fabda3c1f93b" path="/var/lib/kubelet/pods/232a337f-6b37-4c14-8d9b-fabda3c1f93b/volumes" Oct 30 16:49:54.447225 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:54.447191 4493 scope.go:122] "RemoveContainer" containerID="b12fa3950870d86d128c1f730edfe23374c8412866ffecbe1a22d578c4617ced" Oct 30 16:49:58.975467 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:58.975433 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-504442-fq8px"] Oct 30 16:49:58.976413 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:58.976383 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-504442-fq8px"] Oct 30 16:49:59.722012 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:59.721975 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-569b62-2mzzp"] Oct 30 16:49:59.730730 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:59.730703 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-569b62-2mzzp"] Oct 30 16:49:59.942256 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:59.942223 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ea8dca-fbl8d"] Oct 30 16:49:59.943027 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:59.943008 4493 cpu_manager.go:414] "containerMap: removing container" podUID="320f1fb7-deb8-4a82-b468-f92066eca8a1" containerName="istio-proxy" Oct 30 16:49:59.943082 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:59.943031 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="320f1fb7-deb8-4a82-b468-f92066eca8a1" containerName="istio-proxy" Oct 30 16:49:59.943082 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:59.943062 4493 cpu_manager.go:414] "containerMap: removing container" podUID="232a337f-6b37-4c14-8d9b-fabda3c1f93b" containerName="revoke-keys" Oct 30 16:49:59.943082 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:59.943070 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="232a337f-6b37-4c14-8d9b-fabda3c1f93b" containerName="revoke-keys" Oct 30 16:49:59.943249 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:59.943236 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="320f1fb7-deb8-4a82-b468-f92066eca8a1" containerName="istio-proxy" Oct 30 16:49:59.943283 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:59.943256 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="232a337f-6b37-4c14-8d9b-fabda3c1f93b" containerName="revoke-keys" Oct 30 16:49:59.949036 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:59.949015 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ea8dca-fbl8d" Oct 30 16:49:59.963818 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:59.963796 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:49:59.963969 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:59.963855 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:49:59.966920 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:59.966893 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ea8dca-fbl8d"] Oct 30 16:49:59.969976 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:59.969958 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:49:59.973106 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:49:59.973047 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 16:50:00.024056 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:00.024026 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab3ebc3c-e101-4eca-a8e2-67745190a05f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-ea8dca-fbl8d\" (UID: \"ab3ebc3c-e101-4eca-a8e2-67745190a05f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ea8dca-fbl8d" Oct 30 16:50:00.024056 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:00.024057 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbvbx\" (UniqueName: \"kubernetes.io/projected/ab3ebc3c-e101-4eca-a8e2-67745190a05f-kube-api-access-zbvbx\") pod \"maas-api-revoke-keys-e2e-praxis-b-ea8dca-fbl8d\" (UID: \"ab3ebc3c-e101-4eca-a8e2-67745190a05f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ea8dca-fbl8d" Oct 30 16:50:00.024528 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:00.024276 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab3ebc3c-e101-4eca-a8e2-67745190a05f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-ea8dca-fbl8d\" (UID: \"ab3ebc3c-e101-4eca-a8e2-67745190a05f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ea8dca-fbl8d" Oct 30 16:50:00.125567 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:00.125524 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab3ebc3c-e101-4eca-a8e2-67745190a05f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-ea8dca-fbl8d\" (UID: \"ab3ebc3c-e101-4eca-a8e2-67745190a05f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ea8dca-fbl8d" Oct 30 16:50:00.125776 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:00.125575 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbvbx\" (UniqueName: \"kubernetes.io/projected/ab3ebc3c-e101-4eca-a8e2-67745190a05f-kube-api-access-zbvbx\") pod \"maas-api-revoke-keys-e2e-praxis-b-ea8dca-fbl8d\" (UID: \"ab3ebc3c-e101-4eca-a8e2-67745190a05f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ea8dca-fbl8d" Oct 30 16:50:00.125776 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:00.125721 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab3ebc3c-e101-4eca-a8e2-67745190a05f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-ea8dca-fbl8d\" (UID: \"ab3ebc3c-e101-4eca-a8e2-67745190a05f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ea8dca-fbl8d" Oct 30 16:50:00.126554 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:00.126529 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab3ebc3c-e101-4eca-a8e2-67745190a05f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-ea8dca-fbl8d\" (UID: \"ab3ebc3c-e101-4eca-a8e2-67745190a05f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ea8dca-fbl8d" Oct 30 16:50:00.128152 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:00.128129 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab3ebc3c-e101-4eca-a8e2-67745190a05f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-ea8dca-fbl8d\" (UID: \"ab3ebc3c-e101-4eca-a8e2-67745190a05f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ea8dca-fbl8d" Oct 30 16:50:00.147041 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:00.147017 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbvbx\" (UniqueName: \"kubernetes.io/projected/ab3ebc3c-e101-4eca-a8e2-67745190a05f-kube-api-access-zbvbx\") pod \"maas-api-revoke-keys-e2e-praxis-b-ea8dca-fbl8d\" (UID: \"ab3ebc3c-e101-4eca-a8e2-67745190a05f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ea8dca-fbl8d" Oct 30 16:50:00.264331 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:00.264247 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ea8dca-fbl8d" Oct 30 16:50:00.361867 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:50:00.361830 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 16:50:00.576513 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:00.576443 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ea8dca-fbl8d"] Oct 30 16:50:00.579984 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:50:00.579957 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab3ebc3c_e101_4eca_a8e2_67745190a05f.slice/crio-a703985c6b8cf6b16c77332939ca64bf148ccdf1e92a9c28c96f1cf226035e0d WatchSource:0}: Error finding container a703985c6b8cf6b16c77332939ca64bf148ccdf1e92a9c28c96f1cf226035e0d: Status 404 returned error can't find the container with id a703985c6b8cf6b16c77332939ca64bf148ccdf1e92a9c28c96f1cf226035e0d Oct 30 16:50:00.933981 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:00.933945 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ad5370d-702b-410e-967b-9728ac70dba5" path="/var/lib/kubelet/pods/4ad5370d-702b-410e-967b-9728ac70dba5/volumes" Oct 30 16:50:00.934374 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:00.934358 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b43de74a-bb00-40ab-b99d-7ffec5b50bc7" path="/var/lib/kubelet/pods/b43de74a-bb00-40ab-b99d-7ffec5b50bc7/volumes" Oct 30 16:50:01.057150 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:01.057115 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ea8dca-fbl8d" event={"ID":"ab3ebc3c-e101-4eca-a8e2-67745190a05f","Type":"ContainerStarted","Data":"e82892c68529dec976a7682a0721abe2c9c446cbaf46725b596803b04e3704e3"} Oct 30 16:50:01.057651 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:01.057177 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ea8dca-fbl8d" event={"ID":"ab3ebc3c-e101-4eca-a8e2-67745190a05f","Type":"ContainerStarted","Data":"a703985c6b8cf6b16c77332939ca64bf148ccdf1e92a9c28c96f1cf226035e0d"} Oct 30 16:50:01.095633 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:01.095584 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ea8dca-fbl8d" podStartSLOduration=2.095569449 podStartE2EDuration="2.095569449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:49:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:50:01.093148397 +0000 UTC m=+1628.320323497" watchObservedRunningTime="2026-10-30 16:50:01.095569449 +0000 UTC m=+1628.322744548" Oct 30 16:50:02.065700 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:02.065665 4493 generic.go:356] "Generic (PLEG): container finished" podID="ab3ebc3c-e101-4eca-a8e2-67745190a05f" containerID="e82892c68529dec976a7682a0721abe2c9c446cbaf46725b596803b04e3704e3" exitCode=0 Oct 30 16:50:02.066152 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:02.065752 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ea8dca-fbl8d" event={"ID":"ab3ebc3c-e101-4eca-a8e2-67745190a05f","Type":"ContainerDied","Data":"e82892c68529dec976a7682a0721abe2c9c446cbaf46725b596803b04e3704e3"} Oct 30 16:50:02.693816 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:02.690708 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 16:50:03.384807 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:03.380967 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ea8dca-fbl8d" Oct 30 16:50:03.492845 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:03.486550 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab3ebc3c-e101-4eca-a8e2-67745190a05f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab3ebc3c-e101-4eca-a8e2-67745190a05f-maas-api-xks-ca\") pod \"ab3ebc3c-e101-4eca-a8e2-67745190a05f\" (UID: \"ab3ebc3c-e101-4eca-a8e2-67745190a05f\") " Oct 30 16:50:03.492845 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:03.486604 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab3ebc3c-e101-4eca-a8e2-67745190a05f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab3ebc3c-e101-4eca-a8e2-67745190a05f-maas-api-openshift-ca\") pod \"ab3ebc3c-e101-4eca-a8e2-67745190a05f\" (UID: \"ab3ebc3c-e101-4eca-a8e2-67745190a05f\") " Oct 30 16:50:03.492845 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:03.486682 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab3ebc3c-e101-4eca-a8e2-67745190a05f-kube-api-access-zbvbx\" (UniqueName: \"kubernetes.io/projected/ab3ebc3c-e101-4eca-a8e2-67745190a05f-kube-api-access-zbvbx\") pod \"ab3ebc3c-e101-4eca-a8e2-67745190a05f\" (UID: \"ab3ebc3c-e101-4eca-a8e2-67745190a05f\") " Oct 30 16:50:03.492845 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:03.490185 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab3ebc3c-e101-4eca-a8e2-67745190a05f-maas-api-openshift-ca" pod "ab3ebc3c-e101-4eca-a8e2-67745190a05f" (UID: "ab3ebc3c-e101-4eca-a8e2-67745190a05f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:50:03.500015 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:03.494994 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab3ebc3c-e101-4eca-a8e2-67745190a05f-kube-api-access-zbvbx" pod "ab3ebc3c-e101-4eca-a8e2-67745190a05f" (UID: "ab3ebc3c-e101-4eca-a8e2-67745190a05f"). InnerVolumeSpecName "kube-api-access-zbvbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:50:03.500015 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:03.496178 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab3ebc3c-e101-4eca-a8e2-67745190a05f-maas-api-xks-ca" pod "ab3ebc3c-e101-4eca-a8e2-67745190a05f" (UID: "ab3ebc3c-e101-4eca-a8e2-67745190a05f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:50:03.589899 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:03.589856 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab3ebc3c-e101-4eca-a8e2-67745190a05f-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:50:03.589899 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:03.589896 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab3ebc3c-e101-4eca-a8e2-67745190a05f-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:50:03.590109 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:03.589910 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbvbx\" (UniqueName: \"kubernetes.io/projected/ab3ebc3c-e101-4eca-a8e2-67745190a05f-kube-api-access-zbvbx\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:50:04.084658 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:04.084626 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ea8dca-fbl8d" Oct 30 16:50:04.084882 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:04.084620 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ea8dca-fbl8d" event={"ID":"ab3ebc3c-e101-4eca-a8e2-67745190a05f","Type":"ContainerDied","Data":"a703985c6b8cf6b16c77332939ca64bf148ccdf1e92a9c28c96f1cf226035e0d"} Oct 30 16:50:04.084882 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:04.084762 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a703985c6b8cf6b16c77332939ca64bf148ccdf1e92a9c28c96f1cf226035e0d" Oct 30 16:50:10.018540 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:10.018456 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-72b379-wgcmr"] Oct 30 16:50:10.019369 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:10.019348 4493 cpu_manager.go:414] "containerMap: removing container" podUID="ab3ebc3c-e101-4eca-a8e2-67745190a05f" containerName="revoke-keys" Oct 30 16:50:10.019463 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:10.019370 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab3ebc3c-e101-4eca-a8e2-67745190a05f" containerName="revoke-keys" Oct 30 16:50:10.019582 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:10.019569 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab3ebc3c-e101-4eca-a8e2-67745190a05f" containerName="revoke-keys" Oct 30 16:50:10.024070 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:10.024049 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-72b379-wgcmr" Oct 30 16:50:10.038561 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:10.038537 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:50:10.050869 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:10.048279 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:50:10.050869 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:10.048375 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:50:10.050869 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:10.048452 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 16:50:10.053706 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:10.053681 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-72b379-wgcmr"] Oct 30 16:50:10.121891 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:10.121855 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wggmg\" (UniqueName: \"kubernetes.io/projected/f2e7e6ba-4cac-437a-88bc-0b3d3edab047-kube-api-access-wggmg\") pod \"maas-api-revoke-keys-e2e-praxis-a-72b379-wgcmr\" (UID: \"f2e7e6ba-4cac-437a-88bc-0b3d3edab047\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-72b379-wgcmr" Oct 30 16:50:10.122132 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:10.122107 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2e7e6ba-4cac-437a-88bc-0b3d3edab047-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-72b379-wgcmr\" (UID: \"f2e7e6ba-4cac-437a-88bc-0b3d3edab047\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-72b379-wgcmr" Oct 30 16:50:10.122207 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:10.122143 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2e7e6ba-4cac-437a-88bc-0b3d3edab047-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-72b379-wgcmr\" (UID: \"f2e7e6ba-4cac-437a-88bc-0b3d3edab047\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-72b379-wgcmr" Oct 30 16:50:10.223735 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:10.223689 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wggmg\" (UniqueName: \"kubernetes.io/projected/f2e7e6ba-4cac-437a-88bc-0b3d3edab047-kube-api-access-wggmg\") pod \"maas-api-revoke-keys-e2e-praxis-a-72b379-wgcmr\" (UID: \"f2e7e6ba-4cac-437a-88bc-0b3d3edab047\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-72b379-wgcmr" Oct 30 16:50:10.223937 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:10.223919 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2e7e6ba-4cac-437a-88bc-0b3d3edab047-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-72b379-wgcmr\" (UID: \"f2e7e6ba-4cac-437a-88bc-0b3d3edab047\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-72b379-wgcmr" Oct 30 16:50:10.224037 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:10.223955 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2e7e6ba-4cac-437a-88bc-0b3d3edab047-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-72b379-wgcmr\" (UID: \"f2e7e6ba-4cac-437a-88bc-0b3d3edab047\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-72b379-wgcmr" Oct 30 16:50:10.224742 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:10.224710 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2e7e6ba-4cac-437a-88bc-0b3d3edab047-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-72b379-wgcmr\" (UID: \"f2e7e6ba-4cac-437a-88bc-0b3d3edab047\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-72b379-wgcmr" Oct 30 16:50:10.226608 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:10.226583 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2e7e6ba-4cac-437a-88bc-0b3d3edab047-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-72b379-wgcmr\" (UID: \"f2e7e6ba-4cac-437a-88bc-0b3d3edab047\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-72b379-wgcmr" Oct 30 16:50:10.245473 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:10.245446 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wggmg\" (UniqueName: \"kubernetes.io/projected/f2e7e6ba-4cac-437a-88bc-0b3d3edab047-kube-api-access-wggmg\") pod \"maas-api-revoke-keys-e2e-praxis-a-72b379-wgcmr\" (UID: \"f2e7e6ba-4cac-437a-88bc-0b3d3edab047\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-72b379-wgcmr" Oct 30 16:50:10.340061 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:10.339981 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-72b379-wgcmr" Oct 30 16:50:10.656102 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:10.655895 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-72b379-wgcmr"] Oct 30 16:50:11.168556 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:11.168505 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-72b379-wgcmr" event={"ID":"f2e7e6ba-4cac-437a-88bc-0b3d3edab047","Type":"ContainerStarted","Data":"113eac77e74f169721b5659bf4dc1a0962caeb41bfed1eb9375ae6fa2fa7b8e6"} Oct 30 16:50:11.169092 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:11.168578 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-72b379-wgcmr" event={"ID":"f2e7e6ba-4cac-437a-88bc-0b3d3edab047","Type":"ContainerStarted","Data":"8fae43d36ddfd0dd0641de06fe26c84aee66e689c563da1ed2259ee698caad76"} Oct 30 16:50:11.207771 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:11.207654 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-72b379-wgcmr" podStartSLOduration=2.207634222 podStartE2EDuration="2.207634222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:50:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:50:11.203924391 +0000 UTC m=+1638.431099494" watchObservedRunningTime="2026-10-30 16:50:11.207634222 +0000 UTC m=+1638.434809323" Oct 30 16:50:12.178504 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:12.178471 4493 generic.go:356] "Generic (PLEG): container finished" podID="f2e7e6ba-4cac-437a-88bc-0b3d3edab047" containerID="113eac77e74f169721b5659bf4dc1a0962caeb41bfed1eb9375ae6fa2fa7b8e6" exitCode=0 Oct 30 16:50:12.178952 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:12.178552 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-72b379-wgcmr" event={"ID":"f2e7e6ba-4cac-437a-88bc-0b3d3edab047","Type":"ContainerDied","Data":"113eac77e74f169721b5659bf4dc1a0962caeb41bfed1eb9375ae6fa2fa7b8e6"} Oct 30 16:50:12.369017 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:12.368984 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2e9431-9pcmq"] Oct 30 16:50:12.373228 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:12.373209 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2e9431-9pcmq" Oct 30 16:50:12.393696 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:12.393666 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2e9431-9pcmq"] Oct 30 16:50:12.466441 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:12.466352 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/592a8f58-adc8-4a3a-96dd-3d5277d5477b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-2e9431-9pcmq\" (UID: \"592a8f58-adc8-4a3a-96dd-3d5277d5477b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2e9431-9pcmq" Oct 30 16:50:12.466598 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:12.466565 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhl2x\" (UniqueName: \"kubernetes.io/projected/592a8f58-adc8-4a3a-96dd-3d5277d5477b-kube-api-access-rhl2x\") pod \"maas-api-revoke-keys-e2e-praxis-b-2e9431-9pcmq\" (UID: \"592a8f58-adc8-4a3a-96dd-3d5277d5477b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2e9431-9pcmq" Oct 30 16:50:12.466706 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:12.466689 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/592a8f58-adc8-4a3a-96dd-3d5277d5477b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-2e9431-9pcmq\" (UID: \"592a8f58-adc8-4a3a-96dd-3d5277d5477b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2e9431-9pcmq" Oct 30 16:50:12.568485 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:12.568447 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhl2x\" (UniqueName: \"kubernetes.io/projected/592a8f58-adc8-4a3a-96dd-3d5277d5477b-kube-api-access-rhl2x\") pod \"maas-api-revoke-keys-e2e-praxis-b-2e9431-9pcmq\" (UID: \"592a8f58-adc8-4a3a-96dd-3d5277d5477b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2e9431-9pcmq" Oct 30 16:50:12.568678 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:12.568555 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/592a8f58-adc8-4a3a-96dd-3d5277d5477b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-2e9431-9pcmq\" (UID: \"592a8f58-adc8-4a3a-96dd-3d5277d5477b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2e9431-9pcmq" Oct 30 16:50:12.568678 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:12.568614 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/592a8f58-adc8-4a3a-96dd-3d5277d5477b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-2e9431-9pcmq\" (UID: \"592a8f58-adc8-4a3a-96dd-3d5277d5477b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2e9431-9pcmq" Oct 30 16:50:12.569380 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:12.569325 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/592a8f58-adc8-4a3a-96dd-3d5277d5477b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-2e9431-9pcmq\" (UID: \"592a8f58-adc8-4a3a-96dd-3d5277d5477b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2e9431-9pcmq" Oct 30 16:50:12.571082 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:12.571064 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/592a8f58-adc8-4a3a-96dd-3d5277d5477b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-2e9431-9pcmq\" (UID: \"592a8f58-adc8-4a3a-96dd-3d5277d5477b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2e9431-9pcmq" Oct 30 16:50:12.589984 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:12.589954 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhl2x\" (UniqueName: \"kubernetes.io/projected/592a8f58-adc8-4a3a-96dd-3d5277d5477b-kube-api-access-rhl2x\") pod \"maas-api-revoke-keys-e2e-praxis-b-2e9431-9pcmq\" (UID: \"592a8f58-adc8-4a3a-96dd-3d5277d5477b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2e9431-9pcmq" Oct 30 16:50:12.686973 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:12.686937 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2e9431-9pcmq" Oct 30 16:50:13.014392 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:13.014366 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2e9431-9pcmq"] Oct 30 16:50:13.016857 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:50:13.016825 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod592a8f58_adc8_4a3a_96dd_3d5277d5477b.slice/crio-093cad550ced0f2f9800e82b8d2de508351f92e20fb3e64398944e532caff3c5 WatchSource:0}: Error finding container 093cad550ced0f2f9800e82b8d2de508351f92e20fb3e64398944e532caff3c5: Status 404 returned error can't find the container with id 093cad550ced0f2f9800e82b8d2de508351f92e20fb3e64398944e532caff3c5 Oct 30 16:50:13.186980 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:13.186942 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2e9431-9pcmq" event={"ID":"592a8f58-adc8-4a3a-96dd-3d5277d5477b","Type":"ContainerStarted","Data":"093cad550ced0f2f9800e82b8d2de508351f92e20fb3e64398944e532caff3c5"} Oct 30 16:50:13.333306 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:13.333285 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-72b379-wgcmr" Oct 30 16:50:13.389684 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:50:13.389653 4493 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/system.slice/system-systemd\\\\x2dcoredump.slice/systemd-coredump@4-25715-0.service\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86b5095e_c77a_478b_b5ba_ff477800a066.slice/crio-conmon-7621daac5dad9e6f7afc2c9eb3238930f657b8d9e1ff1d6cdf16adf54de856b0.scope\": RecentStats: unable to find data in memory cache]" Oct 30 16:50:13.481246 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:13.481158 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2e7e6ba-4cac-437a-88bc-0b3d3edab047-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2e7e6ba-4cac-437a-88bc-0b3d3edab047-maas-api-xks-ca\") pod \"f2e7e6ba-4cac-437a-88bc-0b3d3edab047\" (UID: \"f2e7e6ba-4cac-437a-88bc-0b3d3edab047\") " Oct 30 16:50:13.481420 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:13.481280 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2e7e6ba-4cac-437a-88bc-0b3d3edab047-kube-api-access-wggmg\" (UniqueName: \"kubernetes.io/projected/f2e7e6ba-4cac-437a-88bc-0b3d3edab047-kube-api-access-wggmg\") pod \"f2e7e6ba-4cac-437a-88bc-0b3d3edab047\" (UID: \"f2e7e6ba-4cac-437a-88bc-0b3d3edab047\") " Oct 30 16:50:13.481420 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:13.481310 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2e7e6ba-4cac-437a-88bc-0b3d3edab047-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2e7e6ba-4cac-437a-88bc-0b3d3edab047-maas-api-openshift-ca\") pod \"f2e7e6ba-4cac-437a-88bc-0b3d3edab047\" (UID: \"f2e7e6ba-4cac-437a-88bc-0b3d3edab047\") " Oct 30 16:50:13.481691 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:13.481668 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2e7e6ba-4cac-437a-88bc-0b3d3edab047-maas-api-openshift-ca" pod "f2e7e6ba-4cac-437a-88bc-0b3d3edab047" (UID: "f2e7e6ba-4cac-437a-88bc-0b3d3edab047"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:50:13.481872 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:13.481857 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2e7e6ba-4cac-437a-88bc-0b3d3edab047-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:50:13.483292 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:13.483273 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2e7e6ba-4cac-437a-88bc-0b3d3edab047-maas-api-xks-ca" pod "f2e7e6ba-4cac-437a-88bc-0b3d3edab047" (UID: "f2e7e6ba-4cac-437a-88bc-0b3d3edab047"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:50:13.483414 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:13.483394 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2e7e6ba-4cac-437a-88bc-0b3d3edab047-kube-api-access-wggmg" pod "f2e7e6ba-4cac-437a-88bc-0b3d3edab047" (UID: "f2e7e6ba-4cac-437a-88bc-0b3d3edab047"). InnerVolumeSpecName "kube-api-access-wggmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:50:13.583168 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:13.583139 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wggmg\" (UniqueName: \"kubernetes.io/projected/f2e7e6ba-4cac-437a-88bc-0b3d3edab047-kube-api-access-wggmg\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:50:13.583168 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:13.583168 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2e7e6ba-4cac-437a-88bc-0b3d3edab047-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:50:14.196763 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:14.196674 4493 generic.go:356] "Generic (PLEG): container finished" podID="592a8f58-adc8-4a3a-96dd-3d5277d5477b" containerID="ffd4fea2bc5ca6f425eb0ff20e92deff8a8e2ee9bf96e851e5de334134fbb592" exitCode=0 Oct 30 16:50:14.197215 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:14.196756 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2e9431-9pcmq" event={"ID":"592a8f58-adc8-4a3a-96dd-3d5277d5477b","Type":"ContainerDied","Data":"ffd4fea2bc5ca6f425eb0ff20e92deff8a8e2ee9bf96e851e5de334134fbb592"} Oct 30 16:50:14.198324 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:14.198300 4493 generic.go:356] "Generic (PLEG): container finished" podID="86b5095e-c77a-478b-b5ba-ff477800a066" containerID="7621daac5dad9e6f7afc2c9eb3238930f657b8d9e1ff1d6cdf16adf54de856b0" exitCode=0 Oct 30 16:50:14.198478 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:14.198372 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7" event={"ID":"86b5095e-c77a-478b-b5ba-ff477800a066","Type":"ContainerDied","Data":"7621daac5dad9e6f7afc2c9eb3238930f657b8d9e1ff1d6cdf16adf54de856b0"} Oct 30 16:50:14.198797 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:14.198766 4493 scope.go:122] "RemoveContainer" containerID="7621daac5dad9e6f7afc2c9eb3238930f657b8d9e1ff1d6cdf16adf54de856b0" Oct 30 16:50:14.199767 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:14.199754 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-72b379-wgcmr" Oct 30 16:50:14.199849 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:14.199761 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-72b379-wgcmr" event={"ID":"f2e7e6ba-4cac-437a-88bc-0b3d3edab047","Type":"ContainerDied","Data":"8fae43d36ddfd0dd0641de06fe26c84aee66e689c563da1ed2259ee698caad76"} Oct 30 16:50:14.199849 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:14.199811 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8fae43d36ddfd0dd0641de06fe26c84aee66e689c563da1ed2259ee698caad76" Oct 30 16:50:14.200399 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:14.200373 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:50:14.200462 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:14.200441 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:50:14.200514 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:14.200471 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:50:14.415872 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:14.415817 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7" Oct 30 16:50:15.231172 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:15.231134 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7" event={"ID":"86b5095e-c77a-478b-b5ba-ff477800a066","Type":"ContainerStarted","Data":"24dfa31fcbb87f3507893c73b0dbeb29b060684ccb851ab123b0b14d7613edb0"} Oct 30 16:50:15.271442 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:15.271404 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9"] Oct 30 16:50:15.271632 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:15.271609 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9" podUID="b08f59e7-615a-4d20-ba53-ad4bcf0d5728" containerName="istio-proxy" containerID="cri-o://e4801a8260760009ece4694c2a4c932f5949126a2ef461d9a34735bd6a2367cd" gracePeriod=30 Oct 30 16:50:15.283318 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:15.283287 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7" Oct 30 16:50:15.284450 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:15.284427 4493 patch_prober.go:29] interesting pod/e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 16:50:15.284561 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:15.284477 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7" podUID="86b5095e-c77a-478b-b5ba-ff477800a066" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 16:50:15.467192 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:15.467173 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2e9431-9pcmq" Oct 30 16:50:15.628870 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:15.628162 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/592a8f58-adc8-4a3a-96dd-3d5277d5477b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/592a8f58-adc8-4a3a-96dd-3d5277d5477b-maas-api-openshift-ca\") pod \"592a8f58-adc8-4a3a-96dd-3d5277d5477b\" (UID: \"592a8f58-adc8-4a3a-96dd-3d5277d5477b\") " Oct 30 16:50:15.628870 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:15.628357 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/592a8f58-adc8-4a3a-96dd-3d5277d5477b-kube-api-access-rhl2x\" (UniqueName: \"kubernetes.io/projected/592a8f58-adc8-4a3a-96dd-3d5277d5477b-kube-api-access-rhl2x\") pod \"592a8f58-adc8-4a3a-96dd-3d5277d5477b\" (UID: \"592a8f58-adc8-4a3a-96dd-3d5277d5477b\") " Oct 30 16:50:15.628870 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:15.628418 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/592a8f58-adc8-4a3a-96dd-3d5277d5477b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/592a8f58-adc8-4a3a-96dd-3d5277d5477b-maas-api-xks-ca\") pod \"592a8f58-adc8-4a3a-96dd-3d5277d5477b\" (UID: \"592a8f58-adc8-4a3a-96dd-3d5277d5477b\") " Oct 30 16:50:15.629301 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:15.629265 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/592a8f58-adc8-4a3a-96dd-3d5277d5477b-maas-api-openshift-ca" pod "592a8f58-adc8-4a3a-96dd-3d5277d5477b" (UID: "592a8f58-adc8-4a3a-96dd-3d5277d5477b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:50:15.631688 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:15.631657 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/592a8f58-adc8-4a3a-96dd-3d5277d5477b-maas-api-xks-ca" pod "592a8f58-adc8-4a3a-96dd-3d5277d5477b" (UID: "592a8f58-adc8-4a3a-96dd-3d5277d5477b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:50:15.631878 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:15.631856 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/592a8f58-adc8-4a3a-96dd-3d5277d5477b-kube-api-access-rhl2x" pod "592a8f58-adc8-4a3a-96dd-3d5277d5477b" (UID: "592a8f58-adc8-4a3a-96dd-3d5277d5477b"). InnerVolumeSpecName "kube-api-access-rhl2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:50:15.733860 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:15.732816 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/592a8f58-adc8-4a3a-96dd-3d5277d5477b-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:50:15.733860 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:15.732857 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhl2x\" (UniqueName: \"kubernetes.io/projected/592a8f58-adc8-4a3a-96dd-3d5277d5477b-kube-api-access-rhl2x\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:50:15.733860 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:15.732872 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/592a8f58-adc8-4a3a-96dd-3d5277d5477b-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:50:16.244761 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:16.244725 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2e9431-9pcmq" Oct 30 16:50:16.245289 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:16.244774 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2e9431-9pcmq" event={"ID":"592a8f58-adc8-4a3a-96dd-3d5277d5477b","Type":"ContainerDied","Data":"093cad550ced0f2f9800e82b8d2de508351f92e20fb3e64398944e532caff3c5"} Oct 30 16:50:16.245289 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:16.244838 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="093cad550ced0f2f9800e82b8d2de508351f92e20fb3e64398944e532caff3c5" Oct 30 16:50:16.283001 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:16.282968 4493 patch_prober.go:29] interesting pod/e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 16:50:16.283159 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:16.283022 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7" podUID="86b5095e-c77a-478b-b5ba-ff477800a066" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 16:50:17.283463 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:17.283103 4493 patch_prober.go:29] interesting pod/e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 16:50:17.283463 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:17.283177 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7" podUID="86b5095e-c77a-478b-b5ba-ff477800a066" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 16:50:18.283739 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:18.283706 4493 patch_prober.go:29] interesting pod/e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 16:50:18.284197 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:18.283769 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7" podUID="86b5095e-c77a-478b-b5ba-ff477800a066" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 16:50:19.283550 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:19.283516 4493 patch_prober.go:29] interesting pod/e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 16:50:19.283806 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:19.283571 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7" podUID="86b5095e-c77a-478b-b5ba-ff477800a066" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 16:50:20.283162 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:20.283125 4493 patch_prober.go:29] interesting pod/e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 16:50:20.283369 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:20.283185 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7" podUID="86b5095e-c77a-478b-b5ba-ff477800a066" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 16:50:20.568478 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:20.568178 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9" Oct 30 16:50:20.701297 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:20.701260 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b08f59e7-615a-4d20-ba53-ad4bcf0d5728-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b08f59e7-615a-4d20-ba53-ad4bcf0d5728-istio-data\") pod \"b08f59e7-615a-4d20-ba53-ad4bcf0d5728\" (UID: \"b08f59e7-615a-4d20-ba53-ad4bcf0d5728\") " Oct 30 16:50:20.701501 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:20.701340 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b08f59e7-615a-4d20-ba53-ad4bcf0d5728-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b08f59e7-615a-4d20-ba53-ad4bcf0d5728-istiod-ca-cert\") pod \"b08f59e7-615a-4d20-ba53-ad4bcf0d5728\" (UID: \"b08f59e7-615a-4d20-ba53-ad4bcf0d5728\") " Oct 30 16:50:20.701501 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:20.701422 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b08f59e7-615a-4d20-ba53-ad4bcf0d5728-kube-api-access-hhsn2\" (UniqueName: \"kubernetes.io/projected/b08f59e7-615a-4d20-ba53-ad4bcf0d5728-kube-api-access-hhsn2\") pod \"b08f59e7-615a-4d20-ba53-ad4bcf0d5728\" (UID: \"b08f59e7-615a-4d20-ba53-ad4bcf0d5728\") " Oct 30 16:50:20.701501 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:20.701466 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b08f59e7-615a-4d20-ba53-ad4bcf0d5728-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b08f59e7-615a-4d20-ba53-ad4bcf0d5728-workload-socket\") pod \"b08f59e7-615a-4d20-ba53-ad4bcf0d5728\" (UID: \"b08f59e7-615a-4d20-ba53-ad4bcf0d5728\") " Oct 30 16:50:20.701684 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:20.701567 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b08f59e7-615a-4d20-ba53-ad4bcf0d5728-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b08f59e7-615a-4d20-ba53-ad4bcf0d5728-istio-podinfo\") pod \"b08f59e7-615a-4d20-ba53-ad4bcf0d5728\" (UID: \"b08f59e7-615a-4d20-ba53-ad4bcf0d5728\") " Oct 30 16:50:20.701684 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:20.701584 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b08f59e7-615a-4d20-ba53-ad4bcf0d5728-istio-data" pod "b08f59e7-615a-4d20-ba53-ad4bcf0d5728" (UID: "b08f59e7-615a-4d20-ba53-ad4bcf0d5728"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:20.701684 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:20.701596 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b08f59e7-615a-4d20-ba53-ad4bcf0d5728-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b08f59e7-615a-4d20-ba53-ad4bcf0d5728-istio-envoy\") pod \"b08f59e7-615a-4d20-ba53-ad4bcf0d5728\" (UID: \"b08f59e7-615a-4d20-ba53-ad4bcf0d5728\") " Oct 30 16:50:20.701684 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:20.701646 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b08f59e7-615a-4d20-ba53-ad4bcf0d5728-istio-token\" (UniqueName: \"kubernetes.io/projected/b08f59e7-615a-4d20-ba53-ad4bcf0d5728-istio-token\") pod \"b08f59e7-615a-4d20-ba53-ad4bcf0d5728\" (UID: \"b08f59e7-615a-4d20-ba53-ad4bcf0d5728\") " Oct 30 16:50:20.701935 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:20.701710 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b08f59e7-615a-4d20-ba53-ad4bcf0d5728-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b08f59e7-615a-4d20-ba53-ad4bcf0d5728-credential-socket\") pod \"b08f59e7-615a-4d20-ba53-ad4bcf0d5728\" (UID: \"b08f59e7-615a-4d20-ba53-ad4bcf0d5728\") " Oct 30 16:50:20.701935 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:20.701796 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b08f59e7-615a-4d20-ba53-ad4bcf0d5728-workload-socket" pod "b08f59e7-615a-4d20-ba53-ad4bcf0d5728" (UID: "b08f59e7-615a-4d20-ba53-ad4bcf0d5728"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:20.701935 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:20.701889 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b08f59e7-615a-4d20-ba53-ad4bcf0d5728-istiod-ca-cert" pod "b08f59e7-615a-4d20-ba53-ad4bcf0d5728" (UID: "b08f59e7-615a-4d20-ba53-ad4bcf0d5728"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:50:20.702103 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:20.701938 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b08f59e7-615a-4d20-ba53-ad4bcf0d5728-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b08f59e7-615a-4d20-ba53-ad4bcf0d5728-workload-certs\") pod \"b08f59e7-615a-4d20-ba53-ad4bcf0d5728\" (UID: \"b08f59e7-615a-4d20-ba53-ad4bcf0d5728\") " Oct 30 16:50:20.702103 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:20.702062 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b08f59e7-615a-4d20-ba53-ad4bcf0d5728-credential-socket" pod "b08f59e7-615a-4d20-ba53-ad4bcf0d5728" (UID: "b08f59e7-615a-4d20-ba53-ad4bcf0d5728"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:20.702680 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:20.702652 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b08f59e7-615a-4d20-ba53-ad4bcf0d5728-istio-data\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:50:20.702827 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:20.702685 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b08f59e7-615a-4d20-ba53-ad4bcf0d5728-istiod-ca-cert\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:50:20.702827 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:20.702701 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b08f59e7-615a-4d20-ba53-ad4bcf0d5728-workload-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:50:20.702827 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:20.702714 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b08f59e7-615a-4d20-ba53-ad4bcf0d5728-credential-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:50:20.703080 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:20.703042 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b08f59e7-615a-4d20-ba53-ad4bcf0d5728-workload-certs" pod "b08f59e7-615a-4d20-ba53-ad4bcf0d5728" (UID: "b08f59e7-615a-4d20-ba53-ad4bcf0d5728"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:20.705033 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:20.705004 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b08f59e7-615a-4d20-ba53-ad4bcf0d5728-istio-podinfo" pod "b08f59e7-615a-4d20-ba53-ad4bcf0d5728" (UID: "b08f59e7-615a-4d20-ba53-ad4bcf0d5728"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:50:20.705033 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:20.705011 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b08f59e7-615a-4d20-ba53-ad4bcf0d5728-istio-envoy" pod "b08f59e7-615a-4d20-ba53-ad4bcf0d5728" (UID: "b08f59e7-615a-4d20-ba53-ad4bcf0d5728"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:20.705535 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:20.705509 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b08f59e7-615a-4d20-ba53-ad4bcf0d5728-kube-api-access-hhsn2" pod "b08f59e7-615a-4d20-ba53-ad4bcf0d5728" (UID: "b08f59e7-615a-4d20-ba53-ad4bcf0d5728"). InnerVolumeSpecName "kube-api-access-hhsn2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:50:20.706434 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:20.706408 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b08f59e7-615a-4d20-ba53-ad4bcf0d5728-istio-token" pod "b08f59e7-615a-4d20-ba53-ad4bcf0d5728" (UID: "b08f59e7-615a-4d20-ba53-ad4bcf0d5728"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:50:20.803737 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:20.803657 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhsn2\" (UniqueName: \"kubernetes.io/projected/b08f59e7-615a-4d20-ba53-ad4bcf0d5728-kube-api-access-hhsn2\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:50:20.803737 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:20.803689 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b08f59e7-615a-4d20-ba53-ad4bcf0d5728-istio-podinfo\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:50:20.803737 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:20.803699 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b08f59e7-615a-4d20-ba53-ad4bcf0d5728-istio-envoy\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:50:20.803737 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:20.803709 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b08f59e7-615a-4d20-ba53-ad4bcf0d5728-istio-token\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:50:20.803737 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:20.803717 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b08f59e7-615a-4d20-ba53-ad4bcf0d5728-workload-certs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:50:21.286546 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:21.286509 4493 generic.go:356] "Generic (PLEG): container finished" podID="b08f59e7-615a-4d20-ba53-ad4bcf0d5728" containerID="e4801a8260760009ece4694c2a4c932f5949126a2ef461d9a34735bd6a2367cd" exitCode=0 Oct 30 16:50:21.286546 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:21.286549 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9" event={"ID":"b08f59e7-615a-4d20-ba53-ad4bcf0d5728","Type":"ContainerDied","Data":"e4801a8260760009ece4694c2a4c932f5949126a2ef461d9a34735bd6a2367cd"} Oct 30 16:50:21.286749 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:21.286588 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9" Oct 30 16:50:21.286749 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:21.286618 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9" event={"ID":"b08f59e7-615a-4d20-ba53-ad4bcf0d5728","Type":"ContainerDied","Data":"0ba42f6b293d42ad617f1f10e623ddaa2d56fe672d99999bae17b576a305ad62"} Oct 30 16:50:21.286749 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:21.286641 4493 scope.go:122] "RemoveContainer" containerID="e4801a8260760009ece4694c2a4c932f5949126a2ef461d9a34735bd6a2367cd" Oct 30 16:50:21.288064 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:21.288042 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7" Oct 30 16:50:21.288407 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:21.288389 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7" Oct 30 16:50:21.289446 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:21.289351 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7" Oct 30 16:50:21.297143 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:21.297123 4493 scope.go:122] "RemoveContainer" containerID="e4801a8260760009ece4694c2a4c932f5949126a2ef461d9a34735bd6a2367cd" Oct 30 16:50:21.297452 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:50:21.297434 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4801a8260760009ece4694c2a4c932f5949126a2ef461d9a34735bd6a2367cd\": container with ID starting with e4801a8260760009ece4694c2a4c932f5949126a2ef461d9a34735bd6a2367cd not found: ID does not exist" containerID="e4801a8260760009ece4694c2a4c932f5949126a2ef461d9a34735bd6a2367cd" Oct 30 16:50:21.297505 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:21.297462 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4801a8260760009ece4694c2a4c932f5949126a2ef461d9a34735bd6a2367cd"} err="failed to get container status \"e4801a8260760009ece4694c2a4c932f5949126a2ef461d9a34735bd6a2367cd\": rpc error: code = NotFound desc = could not find container \"e4801a8260760009ece4694c2a4c932f5949126a2ef461d9a34735bd6a2367cd\": container with ID starting with e4801a8260760009ece4694c2a4c932f5949126a2ef461d9a34735bd6a2367cd not found: ID does not exist" Oct 30 16:50:21.342730 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:21.342698 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9"] Oct 30 16:50:21.352414 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:21.352389 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf2x9"] Oct 30 16:50:22.937962 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:22.937929 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b08f59e7-615a-4d20-ba53-ad4bcf0d5728" path="/var/lib/kubelet/pods/b08f59e7-615a-4d20-ba53-ad4bcf0d5728/volumes" Oct 30 16:50:24.548435 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:24.548405 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ea8dca-fbl8d"] Oct 30 16:50:24.557858 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:24.557830 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ea8dca-fbl8d"] Oct 30 16:50:24.935146 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:24.935112 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab3ebc3c-e101-4eca-a8e2-67745190a05f" path="/var/lib/kubelet/pods/ab3ebc3c-e101-4eca-a8e2-67745190a05f/volumes" Oct 30 16:50:29.148526 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:29.148477 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 16:50:30.342421 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:50:30.342382 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 16:50:36.832955 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:36.832859 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3a5a675-zdph5"] Oct 30 16:50:36.833661 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:36.833643 4493 cpu_manager.go:414] "containerMap: removing container" podUID="f2e7e6ba-4cac-437a-88bc-0b3d3edab047" containerName="revoke-keys" Oct 30 16:50:36.833705 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:36.833664 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2e7e6ba-4cac-437a-88bc-0b3d3edab047" containerName="revoke-keys" Oct 30 16:50:36.833740 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:36.833722 4493 cpu_manager.go:414] "containerMap: removing container" podUID="592a8f58-adc8-4a3a-96dd-3d5277d5477b" containerName="revoke-keys" Oct 30 16:50:36.833740 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:36.833730 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="592a8f58-adc8-4a3a-96dd-3d5277d5477b" containerName="revoke-keys" Oct 30 16:50:36.833838 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:36.833753 4493 cpu_manager.go:414] "containerMap: removing container" podUID="b08f59e7-615a-4d20-ba53-ad4bcf0d5728" containerName="istio-proxy" Oct 30 16:50:36.833838 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:36.833761 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b08f59e7-615a-4d20-ba53-ad4bcf0d5728" containerName="istio-proxy" Oct 30 16:50:36.833920 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:36.833908 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2e7e6ba-4cac-437a-88bc-0b3d3edab047" containerName="revoke-keys" Oct 30 16:50:36.833952 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:36.833929 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="592a8f58-adc8-4a3a-96dd-3d5277d5477b" containerName="revoke-keys" Oct 30 16:50:36.833952 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:36.833941 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="b08f59e7-615a-4d20-ba53-ad4bcf0d5728" containerName="istio-proxy" Oct 30 16:50:36.839284 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:36.839265 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a5a675-zdph5" Oct 30 16:50:36.857777 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:36.857755 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3a5a675-zdph5"] Oct 30 16:50:36.863377 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:36.863352 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:50:36.863377 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:36.863361 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 16:50:36.863544 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:36.863362 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:50:36.863544 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:36.863428 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:50:36.931976 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:36.931946 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpvpc\" (UniqueName: \"kubernetes.io/projected/ad2f82e5-833f-483c-99e9-a460840d8b5a-kube-api-access-hpvpc\") pod \"maas-api-revoke-sub-f3a5a675-zdph5\" (UID: \"ad2f82e5-833f-483c-99e9-a460840d8b5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a5a675-zdph5" Oct 30 16:50:36.932188 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:36.932091 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad2f82e5-833f-483c-99e9-a460840d8b5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3a5a675-zdph5\" (UID: \"ad2f82e5-833f-483c-99e9-a460840d8b5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a5a675-zdph5" Oct 30 16:50:36.932188 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:36.932124 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad2f82e5-833f-483c-99e9-a460840d8b5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3a5a675-zdph5\" (UID: \"ad2f82e5-833f-483c-99e9-a460840d8b5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a5a675-zdph5" Oct 30 16:50:37.033795 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:37.033742 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpvpc\" (UniqueName: \"kubernetes.io/projected/ad2f82e5-833f-483c-99e9-a460840d8b5a-kube-api-access-hpvpc\") pod \"maas-api-revoke-sub-f3a5a675-zdph5\" (UID: \"ad2f82e5-833f-483c-99e9-a460840d8b5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a5a675-zdph5" Oct 30 16:50:37.033980 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:37.033929 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad2f82e5-833f-483c-99e9-a460840d8b5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3a5a675-zdph5\" (UID: \"ad2f82e5-833f-483c-99e9-a460840d8b5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a5a675-zdph5" Oct 30 16:50:37.033980 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:37.033958 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad2f82e5-833f-483c-99e9-a460840d8b5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3a5a675-zdph5\" (UID: \"ad2f82e5-833f-483c-99e9-a460840d8b5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a5a675-zdph5" Oct 30 16:50:37.034584 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:37.034562 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad2f82e5-833f-483c-99e9-a460840d8b5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3a5a675-zdph5\" (UID: \"ad2f82e5-833f-483c-99e9-a460840d8b5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a5a675-zdph5" Oct 30 16:50:37.036609 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:37.036582 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad2f82e5-833f-483c-99e9-a460840d8b5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3a5a675-zdph5\" (UID: \"ad2f82e5-833f-483c-99e9-a460840d8b5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a5a675-zdph5" Oct 30 16:50:37.055255 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:37.055227 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpvpc\" (UniqueName: \"kubernetes.io/projected/ad2f82e5-833f-483c-99e9-a460840d8b5a-kube-api-access-hpvpc\") pod \"maas-api-revoke-sub-f3a5a675-zdph5\" (UID: \"ad2f82e5-833f-483c-99e9-a460840d8b5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a5a675-zdph5" Oct 30 16:50:37.152922 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:37.152887 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a5a675-zdph5" Oct 30 16:50:37.454455 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:37.454380 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3a5a675-zdph5"] Oct 30 16:50:37.457980 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:50:37.457953 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad2f82e5_833f_483c_99e9_a460840d8b5a.slice/crio-fc7d766c13a9aa0c073df2b4c7b20737402637e9c6043c92dc89829030642ec7 WatchSource:0}: Error finding container fc7d766c13a9aa0c073df2b4c7b20737402637e9c6043c92dc89829030642ec7: Status 404 returned error can't find the container with id fc7d766c13a9aa0c073df2b4c7b20737402637e9c6043c92dc89829030642ec7 Oct 30 16:50:38.429676 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:38.429640 4493 generic.go:356] "Generic (PLEG): container finished" podID="ad2f82e5-833f-483c-99e9-a460840d8b5a" containerID="b5442b0183aad780769d7e32378bdcb1422081aec4a9ccc853564d6d174ffa77" exitCode=0 Oct 30 16:50:38.430103 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:38.429722 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a5a675-zdph5" event={"ID":"ad2f82e5-833f-483c-99e9-a460840d8b5a","Type":"ContainerDied","Data":"b5442b0183aad780769d7e32378bdcb1422081aec4a9ccc853564d6d174ffa77"} Oct 30 16:50:38.430103 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:38.429825 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a5a675-zdph5" event={"ID":"ad2f82e5-833f-483c-99e9-a460840d8b5a","Type":"ContainerStarted","Data":"fc7d766c13a9aa0c073df2b4c7b20737402637e9c6043c92dc89829030642ec7"} Oct 30 16:50:39.600069 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:39.600046 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a5a675-zdph5" Oct 30 16:50:39.669818 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:39.669771 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad2f82e5-833f-483c-99e9-a460840d8b5a-kube-api-access-hpvpc\" (UniqueName: \"kubernetes.io/projected/ad2f82e5-833f-483c-99e9-a460840d8b5a-kube-api-access-hpvpc\") pod \"ad2f82e5-833f-483c-99e9-a460840d8b5a\" (UID: \"ad2f82e5-833f-483c-99e9-a460840d8b5a\") " Oct 30 16:50:39.670010 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:39.669841 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad2f82e5-833f-483c-99e9-a460840d8b5a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad2f82e5-833f-483c-99e9-a460840d8b5a-maas-api-xks-ca\") pod \"ad2f82e5-833f-483c-99e9-a460840d8b5a\" (UID: \"ad2f82e5-833f-483c-99e9-a460840d8b5a\") " Oct 30 16:50:39.670010 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:39.669872 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad2f82e5-833f-483c-99e9-a460840d8b5a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad2f82e5-833f-483c-99e9-a460840d8b5a-maas-api-openshift-ca\") pod \"ad2f82e5-833f-483c-99e9-a460840d8b5a\" (UID: \"ad2f82e5-833f-483c-99e9-a460840d8b5a\") " Oct 30 16:50:39.670350 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:39.670320 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad2f82e5-833f-483c-99e9-a460840d8b5a-maas-api-openshift-ca" pod "ad2f82e5-833f-483c-99e9-a460840d8b5a" (UID: "ad2f82e5-833f-483c-99e9-a460840d8b5a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:50:39.672343 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:39.672314 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad2f82e5-833f-483c-99e9-a460840d8b5a-kube-api-access-hpvpc" pod "ad2f82e5-833f-483c-99e9-a460840d8b5a" (UID: "ad2f82e5-833f-483c-99e9-a460840d8b5a"). InnerVolumeSpecName "kube-api-access-hpvpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:50:39.672510 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:39.672490 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad2f82e5-833f-483c-99e9-a460840d8b5a-maas-api-xks-ca" pod "ad2f82e5-833f-483c-99e9-a460840d8b5a" (UID: "ad2f82e5-833f-483c-99e9-a460840d8b5a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:50:39.771983 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:39.771908 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpvpc\" (UniqueName: \"kubernetes.io/projected/ad2f82e5-833f-483c-99e9-a460840d8b5a-kube-api-access-hpvpc\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:50:39.771983 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:39.771934 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad2f82e5-833f-483c-99e9-a460840d8b5a-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:50:39.771983 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:39.771945 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad2f82e5-833f-483c-99e9-a460840d8b5a-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:50:40.447695 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:40.447661 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a5a675-zdph5" event={"ID":"ad2f82e5-833f-483c-99e9-a460840d8b5a","Type":"ContainerDied","Data":"fc7d766c13a9aa0c073df2b4c7b20737402637e9c6043c92dc89829030642ec7"} Oct 30 16:50:40.447923 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:40.447698 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3a5a675-zdph5" Oct 30 16:50:40.447923 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:40.447717 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc7d766c13a9aa0c073df2b4c7b20737402637e9c6043c92dc89829030642ec7" Oct 30 16:50:41.465601 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:41.465569 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2e9431-9pcmq"] Oct 30 16:50:41.475294 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:41.475268 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2e9431-9pcmq"] Oct 30 16:50:42.933502 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:42.933469 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="592a8f58-adc8-4a3a-96dd-3d5277d5477b" path="/var/lib/kubelet/pods/592a8f58-adc8-4a3a-96dd-3d5277d5477b/volumes" Oct 30 16:50:43.968192 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:43.968160 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-72b379-wgcmr"] Oct 30 16:50:43.973342 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:43.973310 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-72b379-wgcmr"] Oct 30 16:50:44.975963 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:44.975924 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2e7e6ba-4cac-437a-88bc-0b3d3edab047" path="/var/lib/kubelet/pods/f2e7e6ba-4cac-437a-88bc-0b3d3edab047/volumes" Oct 30 16:50:50.661169 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:50.661135 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-06ac18-dcldd"] Oct 30 16:50:50.661942 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:50.661921 4493 cpu_manager.go:414] "containerMap: removing container" podUID="ad2f82e5-833f-483c-99e9-a460840d8b5a" containerName="revoke-keys" Oct 30 16:50:50.661942 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:50.661942 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad2f82e5-833f-483c-99e9-a460840d8b5a" containerName="revoke-keys" Oct 30 16:50:50.662133 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:50.662065 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad2f82e5-833f-483c-99e9-a460840d8b5a" containerName="revoke-keys" Oct 30 16:50:50.665553 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:50.665531 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-06ac18-dcldd" Oct 30 16:50:50.678248 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:50.678228 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:50:50.678357 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:50.678252 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:50:50.688352 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:50.688314 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:50:50.688462 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:50.688359 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 16:50:50.693689 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:50.693666 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-06ac18-dcldd"] Oct 30 16:50:50.761752 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:50.761707 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kl2q6\" (UniqueName: \"kubernetes.io/projected/191759dd-4684-48b4-92a1-7890937c34fe-kube-api-access-kl2q6\") pod \"maas-api-revoke-keys-e2e-ipp-b-06ac18-dcldd\" (UID: \"191759dd-4684-48b4-92a1-7890937c34fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-06ac18-dcldd" Oct 30 16:50:50.761974 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:50.761949 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/191759dd-4684-48b4-92a1-7890937c34fe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-06ac18-dcldd\" (UID: \"191759dd-4684-48b4-92a1-7890937c34fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-06ac18-dcldd" Oct 30 16:50:50.762052 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:50.762007 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/191759dd-4684-48b4-92a1-7890937c34fe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-06ac18-dcldd\" (UID: \"191759dd-4684-48b4-92a1-7890937c34fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-06ac18-dcldd" Oct 30 16:50:50.863321 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:50.863284 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl2q6\" (UniqueName: \"kubernetes.io/projected/191759dd-4684-48b4-92a1-7890937c34fe-kube-api-access-kl2q6\") pod \"maas-api-revoke-keys-e2e-ipp-b-06ac18-dcldd\" (UID: \"191759dd-4684-48b4-92a1-7890937c34fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-06ac18-dcldd" Oct 30 16:50:50.863479 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:50.863391 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/191759dd-4684-48b4-92a1-7890937c34fe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-06ac18-dcldd\" (UID: \"191759dd-4684-48b4-92a1-7890937c34fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-06ac18-dcldd" Oct 30 16:50:50.863479 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:50.863421 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/191759dd-4684-48b4-92a1-7890937c34fe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-06ac18-dcldd\" (UID: \"191759dd-4684-48b4-92a1-7890937c34fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-06ac18-dcldd" Oct 30 16:50:50.864034 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:50.864016 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/191759dd-4684-48b4-92a1-7890937c34fe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-06ac18-dcldd\" (UID: \"191759dd-4684-48b4-92a1-7890937c34fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-06ac18-dcldd" Oct 30 16:50:50.866049 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:50.866025 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/191759dd-4684-48b4-92a1-7890937c34fe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-06ac18-dcldd\" (UID: \"191759dd-4684-48b4-92a1-7890937c34fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-06ac18-dcldd" Oct 30 16:50:50.885653 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:50.885626 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl2q6\" (UniqueName: \"kubernetes.io/projected/191759dd-4684-48b4-92a1-7890937c34fe-kube-api-access-kl2q6\") pod \"maas-api-revoke-keys-e2e-ipp-b-06ac18-dcldd\" (UID: \"191759dd-4684-48b4-92a1-7890937c34fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-06ac18-dcldd" Oct 30 16:50:50.979160 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:50.979084 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-06ac18-dcldd" Oct 30 16:50:51.295947 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:51.295871 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-06ac18-dcldd"] Oct 30 16:50:51.299243 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:50:51.299213 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod191759dd_4684_48b4_92a1_7890937c34fe.slice/crio-3428b0c5dd9cf1b112761ee677bc6dc34293bc2b4481a88be645e5cc604d2e54 WatchSource:0}: Error finding container 3428b0c5dd9cf1b112761ee677bc6dc34293bc2b4481a88be645e5cc604d2e54: Status 404 returned error can't find the container with id 3428b0c5dd9cf1b112761ee677bc6dc34293bc2b4481a88be645e5cc604d2e54 Oct 30 16:50:51.746890 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:51.746849 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-06ac18-dcldd" event={"ID":"191759dd-4684-48b4-92a1-7890937c34fe","Type":"ContainerStarted","Data":"a8b5ec593fe2f5819965453b8d83849adf92e05712cc8ec0494e3dca6a5de649"} Oct 30 16:50:51.747291 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:51.746910 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-06ac18-dcldd" event={"ID":"191759dd-4684-48b4-92a1-7890937c34fe","Type":"ContainerStarted","Data":"3428b0c5dd9cf1b112761ee677bc6dc34293bc2b4481a88be645e5cc604d2e54"} Oct 30 16:50:51.783080 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:51.783037 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-06ac18-dcldd" podStartSLOduration=1.783024543 podStartE2EDuration="1.783024543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:50:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:50:51.781824834 +0000 UTC m=+1679.008999933" watchObservedRunningTime="2026-10-30 16:50:51.783024543 +0000 UTC m=+1679.010199643" Oct 30 16:50:52.055097 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:52.055066 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60c9faa7-ch855"] Oct 30 16:50:52.059263 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:52.059238 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c9faa7-ch855" Oct 30 16:50:52.084971 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:52.082140 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60c9faa7-ch855"] Oct 30 16:50:52.180545 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:52.180505 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e57cb61-fe2f-4a2c-930b-eee27bb86327-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60c9faa7-ch855\" (UID: \"5e57cb61-fe2f-4a2c-930b-eee27bb86327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c9faa7-ch855" Oct 30 16:50:52.180734 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:52.180715 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82bm2\" (UniqueName: \"kubernetes.io/projected/5e57cb61-fe2f-4a2c-930b-eee27bb86327-kube-api-access-82bm2\") pod \"maas-api-revoke-sub-60c9faa7-ch855\" (UID: \"5e57cb61-fe2f-4a2c-930b-eee27bb86327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c9faa7-ch855" Oct 30 16:50:52.180803 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:52.180751 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e57cb61-fe2f-4a2c-930b-eee27bb86327-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60c9faa7-ch855\" (UID: \"5e57cb61-fe2f-4a2c-930b-eee27bb86327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c9faa7-ch855" Oct 30 16:50:52.283938 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:52.283905 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e57cb61-fe2f-4a2c-930b-eee27bb86327-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60c9faa7-ch855\" (UID: \"5e57cb61-fe2f-4a2c-930b-eee27bb86327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c9faa7-ch855" Oct 30 16:50:52.284296 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:52.284269 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82bm2\" (UniqueName: \"kubernetes.io/projected/5e57cb61-fe2f-4a2c-930b-eee27bb86327-kube-api-access-82bm2\") pod \"maas-api-revoke-sub-60c9faa7-ch855\" (UID: \"5e57cb61-fe2f-4a2c-930b-eee27bb86327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c9faa7-ch855" Oct 30 16:50:52.284427 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:52.284321 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e57cb61-fe2f-4a2c-930b-eee27bb86327-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60c9faa7-ch855\" (UID: \"5e57cb61-fe2f-4a2c-930b-eee27bb86327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c9faa7-ch855" Oct 30 16:50:52.285174 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:52.285152 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e57cb61-fe2f-4a2c-930b-eee27bb86327-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60c9faa7-ch855\" (UID: \"5e57cb61-fe2f-4a2c-930b-eee27bb86327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c9faa7-ch855" Oct 30 16:50:52.291605 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:52.291260 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e57cb61-fe2f-4a2c-930b-eee27bb86327-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60c9faa7-ch855\" (UID: \"5e57cb61-fe2f-4a2c-930b-eee27bb86327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c9faa7-ch855" Oct 30 16:50:52.323852 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:52.319153 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82bm2\" (UniqueName: \"kubernetes.io/projected/5e57cb61-fe2f-4a2c-930b-eee27bb86327-kube-api-access-82bm2\") pod \"maas-api-revoke-sub-60c9faa7-ch855\" (UID: \"5e57cb61-fe2f-4a2c-930b-eee27bb86327\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c9faa7-ch855" Oct 30 16:50:52.373974 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:52.373945 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c9faa7-ch855" Oct 30 16:50:52.678213 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:52.678179 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60c9faa7-ch855"] Oct 30 16:50:52.683819 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:50:52.683750 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e57cb61_fe2f_4a2c_930b_eee27bb86327.slice/crio-b77d91cb9bdad3b349d4cc1734f38123af3464301e27fca2e8707e3f64432407 WatchSource:0}: Error finding container b77d91cb9bdad3b349d4cc1734f38123af3464301e27fca2e8707e3f64432407: Status 404 returned error can't find the container with id b77d91cb9bdad3b349d4cc1734f38123af3464301e27fca2e8707e3f64432407 Oct 30 16:50:52.685931 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:52.685909 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c408f1-pwkxz"] Oct 30 16:50:52.691404 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:52.691386 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c408f1-pwkxz" Oct 30 16:50:52.714620 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:52.714584 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c408f1-pwkxz"] Oct 30 16:50:52.758411 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:52.758375 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c9faa7-ch855" event={"ID":"5e57cb61-fe2f-4a2c-930b-eee27bb86327","Type":"ContainerStarted","Data":"b77d91cb9bdad3b349d4cc1734f38123af3464301e27fca2e8707e3f64432407"} Oct 30 16:50:52.760510 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:52.760478 4493 generic.go:356] "Generic (PLEG): container finished" podID="191759dd-4684-48b4-92a1-7890937c34fe" containerID="a8b5ec593fe2f5819965453b8d83849adf92e05712cc8ec0494e3dca6a5de649" exitCode=0 Oct 30 16:50:52.760642 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:52.760566 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-06ac18-dcldd" event={"ID":"191759dd-4684-48b4-92a1-7890937c34fe","Type":"ContainerDied","Data":"a8b5ec593fe2f5819965453b8d83849adf92e05712cc8ec0494e3dca6a5de649"} Oct 30 16:50:52.791547 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:52.791523 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15880d82-3211-4955-99ee-b84ca67eff09-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-c408f1-pwkxz\" (UID: \"15880d82-3211-4955-99ee-b84ca67eff09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c408f1-pwkxz" Oct 30 16:50:52.791641 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:52.791580 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j756\" (UniqueName: \"kubernetes.io/projected/15880d82-3211-4955-99ee-b84ca67eff09-kube-api-access-7j756\") pod \"maas-api-revoke-keys-e2e-praxis-b-c408f1-pwkxz\" (UID: \"15880d82-3211-4955-99ee-b84ca67eff09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c408f1-pwkxz" Oct 30 16:50:52.791745 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:52.791728 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15880d82-3211-4955-99ee-b84ca67eff09-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-c408f1-pwkxz\" (UID: \"15880d82-3211-4955-99ee-b84ca67eff09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c408f1-pwkxz" Oct 30 16:50:52.892897 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:52.892863 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15880d82-3211-4955-99ee-b84ca67eff09-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-c408f1-pwkxz\" (UID: \"15880d82-3211-4955-99ee-b84ca67eff09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c408f1-pwkxz" Oct 30 16:50:52.893054 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:52.892961 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j756\" (UniqueName: \"kubernetes.io/projected/15880d82-3211-4955-99ee-b84ca67eff09-kube-api-access-7j756\") pod \"maas-api-revoke-keys-e2e-praxis-b-c408f1-pwkxz\" (UID: \"15880d82-3211-4955-99ee-b84ca67eff09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c408f1-pwkxz" Oct 30 16:50:52.893125 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:52.893111 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15880d82-3211-4955-99ee-b84ca67eff09-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-c408f1-pwkxz\" (UID: \"15880d82-3211-4955-99ee-b84ca67eff09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c408f1-pwkxz" Oct 30 16:50:52.893481 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:52.893462 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15880d82-3211-4955-99ee-b84ca67eff09-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-c408f1-pwkxz\" (UID: \"15880d82-3211-4955-99ee-b84ca67eff09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c408f1-pwkxz" Oct 30 16:50:52.905112 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:52.905086 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15880d82-3211-4955-99ee-b84ca67eff09-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-c408f1-pwkxz\" (UID: \"15880d82-3211-4955-99ee-b84ca67eff09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c408f1-pwkxz" Oct 30 16:50:52.913764 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:52.913741 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j756\" (UniqueName: \"kubernetes.io/projected/15880d82-3211-4955-99ee-b84ca67eff09-kube-api-access-7j756\") pod \"maas-api-revoke-keys-e2e-praxis-b-c408f1-pwkxz\" (UID: \"15880d82-3211-4955-99ee-b84ca67eff09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c408f1-pwkxz" Oct 30 16:50:53.007101 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:53.007071 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c408f1-pwkxz" Oct 30 16:50:53.314524 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:53.314496 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c408f1-pwkxz"] Oct 30 16:50:53.316465 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:50:53.316436 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15880d82_3211_4955_99ee_b84ca67eff09.slice/crio-bc605de376f384e6646813535fa6ed28ec9fb5739c04e5c102ab2a5dee8d2a46 WatchSource:0}: Error finding container bc605de376f384e6646813535fa6ed28ec9fb5739c04e5c102ab2a5dee8d2a46: Status 404 returned error can't find the container with id bc605de376f384e6646813535fa6ed28ec9fb5739c04e5c102ab2a5dee8d2a46 Oct 30 16:50:53.768011 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:53.767977 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8aedbba2-cqbbp"] Oct 30 16:50:53.770931 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:53.770906 4493 generic.go:356] "Generic (PLEG): container finished" podID="5e57cb61-fe2f-4a2c-930b-eee27bb86327" containerID="0beef2c614343479971df943571b5bc6e4098feb43b2dfe9d7458863f85ddb2b" exitCode=0 Oct 30 16:50:53.772157 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:53.772136 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c9faa7-ch855" event={"ID":"5e57cb61-fe2f-4a2c-930b-eee27bb86327","Type":"ContainerDied","Data":"0beef2c614343479971df943571b5bc6e4098feb43b2dfe9d7458863f85ddb2b"} Oct 30 16:50:53.772244 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:53.772225 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8aedbba2-cqbbp" Oct 30 16:50:53.772828 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:53.772807 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c408f1-pwkxz" event={"ID":"15880d82-3211-4955-99ee-b84ca67eff09","Type":"ContainerStarted","Data":"1ca614629d6706ef833fd26e9049bc86994a369bc04fd06e314541917323eeb5"} Oct 30 16:50:53.772933 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:53.772849 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c408f1-pwkxz" event={"ID":"15880d82-3211-4955-99ee-b84ca67eff09","Type":"ContainerStarted","Data":"bc605de376f384e6646813535fa6ed28ec9fb5739c04e5c102ab2a5dee8d2a46"} Oct 30 16:50:53.798119 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:53.798080 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8aedbba2-cqbbp"] Oct 30 16:50:53.847210 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:53.846897 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c408f1-pwkxz" podStartSLOduration=1.846884328 podStartE2EDuration="1.846884328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:50:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:50:53.84552075 +0000 UTC m=+1681.072696037" watchObservedRunningTime="2026-10-30 16:50:53.846884328 +0000 UTC m=+1681.074059429" Oct 30 16:50:53.905608 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:53.905571 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb05f1aa-442f-4999-b8e2-bb01d42e9d7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8aedbba2-cqbbp\" (UID: \"cb05f1aa-442f-4999-b8e2-bb01d42e9d7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8aedbba2-cqbbp" Oct 30 16:50:53.905844 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:53.905825 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtvb8\" (UniqueName: \"kubernetes.io/projected/cb05f1aa-442f-4999-b8e2-bb01d42e9d7a-kube-api-access-vtvb8\") pod \"maas-api-revoke-sub-8aedbba2-cqbbp\" (UID: \"cb05f1aa-442f-4999-b8e2-bb01d42e9d7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8aedbba2-cqbbp" Oct 30 16:50:53.905924 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:53.905866 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb05f1aa-442f-4999-b8e2-bb01d42e9d7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8aedbba2-cqbbp\" (UID: \"cb05f1aa-442f-4999-b8e2-bb01d42e9d7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8aedbba2-cqbbp" Oct 30 16:50:53.974612 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:53.974586 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-06ac18-dcldd" Oct 30 16:50:54.006922 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:54.006899 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb05f1aa-442f-4999-b8e2-bb01d42e9d7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8aedbba2-cqbbp\" (UID: \"cb05f1aa-442f-4999-b8e2-bb01d42e9d7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8aedbba2-cqbbp" Oct 30 16:50:54.007102 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:54.007080 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtvb8\" (UniqueName: \"kubernetes.io/projected/cb05f1aa-442f-4999-b8e2-bb01d42e9d7a-kube-api-access-vtvb8\") pod \"maas-api-revoke-sub-8aedbba2-cqbbp\" (UID: \"cb05f1aa-442f-4999-b8e2-bb01d42e9d7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8aedbba2-cqbbp" Oct 30 16:50:54.007167 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:54.007124 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb05f1aa-442f-4999-b8e2-bb01d42e9d7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8aedbba2-cqbbp\" (UID: \"cb05f1aa-442f-4999-b8e2-bb01d42e9d7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8aedbba2-cqbbp" Oct 30 16:50:54.007875 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:54.007842 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb05f1aa-442f-4999-b8e2-bb01d42e9d7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8aedbba2-cqbbp\" (UID: \"cb05f1aa-442f-4999-b8e2-bb01d42e9d7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8aedbba2-cqbbp" Oct 30 16:50:54.009486 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:54.009457 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb05f1aa-442f-4999-b8e2-bb01d42e9d7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8aedbba2-cqbbp\" (UID: \"cb05f1aa-442f-4999-b8e2-bb01d42e9d7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8aedbba2-cqbbp" Oct 30 16:50:54.033658 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:54.033620 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtvb8\" (UniqueName: \"kubernetes.io/projected/cb05f1aa-442f-4999-b8e2-bb01d42e9d7a-kube-api-access-vtvb8\") pod \"maas-api-revoke-sub-8aedbba2-cqbbp\" (UID: \"cb05f1aa-442f-4999-b8e2-bb01d42e9d7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8aedbba2-cqbbp" Oct 30 16:50:54.089087 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:54.089054 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8aedbba2-cqbbp" Oct 30 16:50:54.113932 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:54.113643 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/191759dd-4684-48b4-92a1-7890937c34fe-kube-api-access-kl2q6\" (UniqueName: \"kubernetes.io/projected/191759dd-4684-48b4-92a1-7890937c34fe-kube-api-access-kl2q6\") pod \"191759dd-4684-48b4-92a1-7890937c34fe\" (UID: \"191759dd-4684-48b4-92a1-7890937c34fe\") " Oct 30 16:50:54.113932 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:54.113744 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/191759dd-4684-48b4-92a1-7890937c34fe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/191759dd-4684-48b4-92a1-7890937c34fe-maas-api-xks-ca\") pod \"191759dd-4684-48b4-92a1-7890937c34fe\" (UID: \"191759dd-4684-48b4-92a1-7890937c34fe\") " Oct 30 16:50:54.114686 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:54.114609 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/191759dd-4684-48b4-92a1-7890937c34fe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/191759dd-4684-48b4-92a1-7890937c34fe-maas-api-openshift-ca\") pod \"191759dd-4684-48b4-92a1-7890937c34fe\" (UID: \"191759dd-4684-48b4-92a1-7890937c34fe\") " Oct 30 16:50:54.115022 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:54.114994 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/191759dd-4684-48b4-92a1-7890937c34fe-maas-api-openshift-ca" pod "191759dd-4684-48b4-92a1-7890937c34fe" (UID: "191759dd-4684-48b4-92a1-7890937c34fe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:50:54.115527 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:54.115505 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/191759dd-4684-48b4-92a1-7890937c34fe-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:50:54.117571 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:54.117170 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/191759dd-4684-48b4-92a1-7890937c34fe-maas-api-xks-ca" pod "191759dd-4684-48b4-92a1-7890937c34fe" (UID: "191759dd-4684-48b4-92a1-7890937c34fe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:50:54.118893 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:54.118871 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/191759dd-4684-48b4-92a1-7890937c34fe-kube-api-access-kl2q6" pod "191759dd-4684-48b4-92a1-7890937c34fe" (UID: "191759dd-4684-48b4-92a1-7890937c34fe"). InnerVolumeSpecName "kube-api-access-kl2q6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:50:54.216953 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:54.216921 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/191759dd-4684-48b4-92a1-7890937c34fe-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:50:54.216953 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:54.216955 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kl2q6\" (UniqueName: \"kubernetes.io/projected/191759dd-4684-48b4-92a1-7890937c34fe-kube-api-access-kl2q6\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:50:54.406409 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:54.406297 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8aedbba2-cqbbp"] Oct 30 16:50:54.409280 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:50:54.409223 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb05f1aa_442f_4999_b8e2_bb01d42e9d7a.slice/crio-7fb51659dd1a95bb7eea9b7ba4f4371cd54c4765af2e0adc58655d1bccb35da5 WatchSource:0}: Error finding container 7fb51659dd1a95bb7eea9b7ba4f4371cd54c4765af2e0adc58655d1bccb35da5: Status 404 returned error can't find the container with id 7fb51659dd1a95bb7eea9b7ba4f4371cd54c4765af2e0adc58655d1bccb35da5 Oct 30 16:50:54.444876 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:54.444852 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cfe9f0-btls9"] Oct 30 16:50:54.445628 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:54.445610 4493 cpu_manager.go:414] "containerMap: removing container" podUID="191759dd-4684-48b4-92a1-7890937c34fe" containerName="revoke-keys" Oct 30 16:50:54.445696 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:54.445633 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="191759dd-4684-48b4-92a1-7890937c34fe" containerName="revoke-keys" Oct 30 16:50:54.445847 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:54.445835 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="191759dd-4684-48b4-92a1-7890937c34fe" containerName="revoke-keys" Oct 30 16:50:54.449664 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:54.449647 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cfe9f0-btls9" Oct 30 16:50:54.470969 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:54.470946 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cfe9f0-btls9"] Oct 30 16:50:54.520072 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:54.520037 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ddfcb4d-50ab-48e3-8591-b27f3105c61a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-cfe9f0-btls9\" (UID: \"6ddfcb4d-50ab-48e3-8591-b27f3105c61a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cfe9f0-btls9" Oct 30 16:50:54.520214 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:54.520197 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tm6h\" (UniqueName: \"kubernetes.io/projected/6ddfcb4d-50ab-48e3-8591-b27f3105c61a-kube-api-access-9tm6h\") pod \"maas-api-revoke-keys-e2e-ipp-a-cfe9f0-btls9\" (UID: \"6ddfcb4d-50ab-48e3-8591-b27f3105c61a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cfe9f0-btls9" Oct 30 16:50:54.520283 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:54.520227 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ddfcb4d-50ab-48e3-8591-b27f3105c61a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-cfe9f0-btls9\" (UID: \"6ddfcb4d-50ab-48e3-8591-b27f3105c61a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cfe9f0-btls9" Oct 30 16:50:54.621693 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:54.621663 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ddfcb4d-50ab-48e3-8591-b27f3105c61a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-cfe9f0-btls9\" (UID: \"6ddfcb4d-50ab-48e3-8591-b27f3105c61a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cfe9f0-btls9" Oct 30 16:50:54.621911 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:54.621755 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tm6h\" (UniqueName: \"kubernetes.io/projected/6ddfcb4d-50ab-48e3-8591-b27f3105c61a-kube-api-access-9tm6h\") pod \"maas-api-revoke-keys-e2e-ipp-a-cfe9f0-btls9\" (UID: \"6ddfcb4d-50ab-48e3-8591-b27f3105c61a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cfe9f0-btls9" Oct 30 16:50:54.621911 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:54.621779 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ddfcb4d-50ab-48e3-8591-b27f3105c61a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-cfe9f0-btls9\" (UID: \"6ddfcb4d-50ab-48e3-8591-b27f3105c61a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cfe9f0-btls9" Oct 30 16:50:54.622535 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:54.622441 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ddfcb4d-50ab-48e3-8591-b27f3105c61a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-cfe9f0-btls9\" (UID: \"6ddfcb4d-50ab-48e3-8591-b27f3105c61a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cfe9f0-btls9" Oct 30 16:50:54.624220 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:54.624198 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ddfcb4d-50ab-48e3-8591-b27f3105c61a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-cfe9f0-btls9\" (UID: \"6ddfcb4d-50ab-48e3-8591-b27f3105c61a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cfe9f0-btls9" Oct 30 16:50:54.648413 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:54.648386 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tm6h\" (UniqueName: \"kubernetes.io/projected/6ddfcb4d-50ab-48e3-8591-b27f3105c61a-kube-api-access-9tm6h\") pod \"maas-api-revoke-keys-e2e-ipp-a-cfe9f0-btls9\" (UID: \"6ddfcb4d-50ab-48e3-8591-b27f3105c61a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cfe9f0-btls9" Oct 30 16:50:54.781656 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:54.781625 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-06ac18-dcldd" Oct 30 16:50:54.782204 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:54.781661 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-06ac18-dcldd" event={"ID":"191759dd-4684-48b4-92a1-7890937c34fe","Type":"ContainerDied","Data":"3428b0c5dd9cf1b112761ee677bc6dc34293bc2b4481a88be645e5cc604d2e54"} Oct 30 16:50:54.782204 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:54.781706 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3428b0c5dd9cf1b112761ee677bc6dc34293bc2b4481a88be645e5cc604d2e54" Oct 30 16:50:54.783361 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:54.783333 4493 generic.go:356] "Generic (PLEG): container finished" podID="15880d82-3211-4955-99ee-b84ca67eff09" containerID="1ca614629d6706ef833fd26e9049bc86994a369bc04fd06e314541917323eeb5" exitCode=0 Oct 30 16:50:54.783516 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:54.783407 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c408f1-pwkxz" event={"ID":"15880d82-3211-4955-99ee-b84ca67eff09","Type":"ContainerDied","Data":"1ca614629d6706ef833fd26e9049bc86994a369bc04fd06e314541917323eeb5"} Oct 30 16:50:54.784902 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:54.784880 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8aedbba2-cqbbp" event={"ID":"cb05f1aa-442f-4999-b8e2-bb01d42e9d7a","Type":"ContainerStarted","Data":"7fb51659dd1a95bb7eea9b7ba4f4371cd54c4765af2e0adc58655d1bccb35da5"} Oct 30 16:50:54.799912 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:54.799887 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cfe9f0-btls9" Oct 30 16:50:55.002157 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:55.002134 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c9faa7-ch855" Oct 30 16:50:55.143502 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:55.143471 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cfe9f0-btls9"] Oct 30 16:50:55.146159 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:50:55.146130 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ddfcb4d_50ab_48e3_8591_b27f3105c61a.slice/crio-009fe436513a1e0a623db4d2cb27f49d99fd695dbbd3996319c69161a71e7f0e WatchSource:0}: Error finding container 009fe436513a1e0a623db4d2cb27f49d99fd695dbbd3996319c69161a71e7f0e: Status 404 returned error can't find the container with id 009fe436513a1e0a623db4d2cb27f49d99fd695dbbd3996319c69161a71e7f0e Oct 30 16:50:55.147282 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:55.147205 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e57cb61-fe2f-4a2c-930b-eee27bb86327-kube-api-access-82bm2\" (UniqueName: \"kubernetes.io/projected/5e57cb61-fe2f-4a2c-930b-eee27bb86327-kube-api-access-82bm2\") pod \"5e57cb61-fe2f-4a2c-930b-eee27bb86327\" (UID: \"5e57cb61-fe2f-4a2c-930b-eee27bb86327\") " Oct 30 16:50:55.147282 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:55.147248 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e57cb61-fe2f-4a2c-930b-eee27bb86327-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e57cb61-fe2f-4a2c-930b-eee27bb86327-maas-api-openshift-ca\") pod \"5e57cb61-fe2f-4a2c-930b-eee27bb86327\" (UID: \"5e57cb61-fe2f-4a2c-930b-eee27bb86327\") " Oct 30 16:50:55.147444 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:55.147386 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e57cb61-fe2f-4a2c-930b-eee27bb86327-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e57cb61-fe2f-4a2c-930b-eee27bb86327-maas-api-xks-ca\") pod \"5e57cb61-fe2f-4a2c-930b-eee27bb86327\" (UID: \"5e57cb61-fe2f-4a2c-930b-eee27bb86327\") " Oct 30 16:50:55.147999 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:55.147653 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e57cb61-fe2f-4a2c-930b-eee27bb86327-maas-api-openshift-ca" pod "5e57cb61-fe2f-4a2c-930b-eee27bb86327" (UID: "5e57cb61-fe2f-4a2c-930b-eee27bb86327"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:50:55.147999 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:55.147972 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e57cb61-fe2f-4a2c-930b-eee27bb86327-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:50:55.149907 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:55.149862 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e57cb61-fe2f-4a2c-930b-eee27bb86327-kube-api-access-82bm2" pod "5e57cb61-fe2f-4a2c-930b-eee27bb86327" (UID: "5e57cb61-fe2f-4a2c-930b-eee27bb86327"). InnerVolumeSpecName "kube-api-access-82bm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:50:55.150406 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:55.150362 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e57cb61-fe2f-4a2c-930b-eee27bb86327-maas-api-xks-ca" pod "5e57cb61-fe2f-4a2c-930b-eee27bb86327" (UID: "5e57cb61-fe2f-4a2c-930b-eee27bb86327"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:50:55.249540 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:55.249474 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e57cb61-fe2f-4a2c-930b-eee27bb86327-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:50:55.249540 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:55.249498 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82bm2\" (UniqueName: \"kubernetes.io/projected/5e57cb61-fe2f-4a2c-930b-eee27bb86327-kube-api-access-82bm2\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:50:55.793419 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:55.793386 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cfe9f0-btls9" event={"ID":"6ddfcb4d-50ab-48e3-8591-b27f3105c61a","Type":"ContainerStarted","Data":"233dbd8fcc7ae5adc593d69826773139334afb62532787f1527fa21b4d81c75c"} Oct 30 16:50:55.793898 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:55.793454 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cfe9f0-btls9" event={"ID":"6ddfcb4d-50ab-48e3-8591-b27f3105c61a","Type":"ContainerStarted","Data":"009fe436513a1e0a623db4d2cb27f49d99fd695dbbd3996319c69161a71e7f0e"} Oct 30 16:50:55.794914 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:55.794893 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c9faa7-ch855" Oct 30 16:50:55.794914 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:55.794899 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c9faa7-ch855" event={"ID":"5e57cb61-fe2f-4a2c-930b-eee27bb86327","Type":"ContainerDied","Data":"b77d91cb9bdad3b349d4cc1734f38123af3464301e27fca2e8707e3f64432407"} Oct 30 16:50:55.795113 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:55.794942 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b77d91cb9bdad3b349d4cc1734f38123af3464301e27fca2e8707e3f64432407" Oct 30 16:50:55.796758 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:55.796735 4493 generic.go:356] "Generic (PLEG): container finished" podID="cb05f1aa-442f-4999-b8e2-bb01d42e9d7a" containerID="58bf3ff75e86d78ed28aef17b44a3f28b2dca7f85a25669d2bdddc2b8aa53d33" exitCode=0 Oct 30 16:50:55.796914 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:55.796827 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8aedbba2-cqbbp" event={"ID":"cb05f1aa-442f-4999-b8e2-bb01d42e9d7a","Type":"ContainerDied","Data":"58bf3ff75e86d78ed28aef17b44a3f28b2dca7f85a25669d2bdddc2b8aa53d33"} Oct 30 16:50:55.833207 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:55.833156 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cfe9f0-btls9" podStartSLOduration=1.833142145 podStartE2EDuration="1.833142145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:50:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:50:55.82933454 +0000 UTC m=+1683.056509635" watchObservedRunningTime="2026-10-30 16:50:55.833142145 +0000 UTC m=+1683.060317244" Oct 30 16:50:55.978346 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:55.978321 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c408f1-pwkxz" Oct 30 16:50:56.060202 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:56.060114 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15880d82-3211-4955-99ee-b84ca67eff09-kube-api-access-7j756\" (UniqueName: \"kubernetes.io/projected/15880d82-3211-4955-99ee-b84ca67eff09-kube-api-access-7j756\") pod \"15880d82-3211-4955-99ee-b84ca67eff09\" (UID: \"15880d82-3211-4955-99ee-b84ca67eff09\") " Oct 30 16:50:56.060202 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:56.060181 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15880d82-3211-4955-99ee-b84ca67eff09-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15880d82-3211-4955-99ee-b84ca67eff09-maas-api-xks-ca\") pod \"15880d82-3211-4955-99ee-b84ca67eff09\" (UID: \"15880d82-3211-4955-99ee-b84ca67eff09\") " Oct 30 16:50:56.060434 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:56.060274 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15880d82-3211-4955-99ee-b84ca67eff09-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15880d82-3211-4955-99ee-b84ca67eff09-maas-api-openshift-ca\") pod \"15880d82-3211-4955-99ee-b84ca67eff09\" (UID: \"15880d82-3211-4955-99ee-b84ca67eff09\") " Oct 30 16:50:56.060756 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:56.060726 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15880d82-3211-4955-99ee-b84ca67eff09-maas-api-openshift-ca" pod "15880d82-3211-4955-99ee-b84ca67eff09" (UID: "15880d82-3211-4955-99ee-b84ca67eff09"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:50:56.062987 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:56.062959 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15880d82-3211-4955-99ee-b84ca67eff09-kube-api-access-7j756" pod "15880d82-3211-4955-99ee-b84ca67eff09" (UID: "15880d82-3211-4955-99ee-b84ca67eff09"). InnerVolumeSpecName "kube-api-access-7j756". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:50:56.063400 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:56.063377 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15880d82-3211-4955-99ee-b84ca67eff09-maas-api-xks-ca" pod "15880d82-3211-4955-99ee-b84ca67eff09" (UID: "15880d82-3211-4955-99ee-b84ca67eff09"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:50:56.162605 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:56.162570 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7j756\" (UniqueName: \"kubernetes.io/projected/15880d82-3211-4955-99ee-b84ca67eff09-kube-api-access-7j756\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:50:56.162605 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:56.162603 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15880d82-3211-4955-99ee-b84ca67eff09-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:50:56.162820 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:56.162617 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15880d82-3211-4955-99ee-b84ca67eff09-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:50:56.806389 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:56.806348 4493 generic.go:356] "Generic (PLEG): container finished" podID="6ddfcb4d-50ab-48e3-8591-b27f3105c61a" containerID="233dbd8fcc7ae5adc593d69826773139334afb62532787f1527fa21b4d81c75c" exitCode=0 Oct 30 16:50:56.806897 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:56.806419 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cfe9f0-btls9" event={"ID":"6ddfcb4d-50ab-48e3-8591-b27f3105c61a","Type":"ContainerDied","Data":"233dbd8fcc7ae5adc593d69826773139334afb62532787f1527fa21b4d81c75c"} Oct 30 16:50:56.809727 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:56.809707 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c408f1-pwkxz" Oct 30 16:50:56.809897 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:56.809742 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c408f1-pwkxz" event={"ID":"15880d82-3211-4955-99ee-b84ca67eff09","Type":"ContainerDied","Data":"bc605de376f384e6646813535fa6ed28ec9fb5739c04e5c102ab2a5dee8d2a46"} Oct 30 16:50:56.809897 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:56.809809 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc605de376f384e6646813535fa6ed28ec9fb5739c04e5c102ab2a5dee8d2a46" Oct 30 16:50:56.985921 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:56.985897 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8aedbba2-cqbbp" Oct 30 16:50:57.073549 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:57.073460 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb05f1aa-442f-4999-b8e2-bb01d42e9d7a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb05f1aa-442f-4999-b8e2-bb01d42e9d7a-maas-api-xks-ca\") pod \"cb05f1aa-442f-4999-b8e2-bb01d42e9d7a\" (UID: \"cb05f1aa-442f-4999-b8e2-bb01d42e9d7a\") " Oct 30 16:50:57.073549 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:57.073524 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb05f1aa-442f-4999-b8e2-bb01d42e9d7a-kube-api-access-vtvb8\" (UniqueName: \"kubernetes.io/projected/cb05f1aa-442f-4999-b8e2-bb01d42e9d7a-kube-api-access-vtvb8\") pod \"cb05f1aa-442f-4999-b8e2-bb01d42e9d7a\" (UID: \"cb05f1aa-442f-4999-b8e2-bb01d42e9d7a\") " Oct 30 16:50:57.073774 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:57.073561 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb05f1aa-442f-4999-b8e2-bb01d42e9d7a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb05f1aa-442f-4999-b8e2-bb01d42e9d7a-maas-api-openshift-ca\") pod \"cb05f1aa-442f-4999-b8e2-bb01d42e9d7a\" (UID: \"cb05f1aa-442f-4999-b8e2-bb01d42e9d7a\") " Oct 30 16:50:57.074076 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:57.074051 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb05f1aa-442f-4999-b8e2-bb01d42e9d7a-maas-api-openshift-ca" pod "cb05f1aa-442f-4999-b8e2-bb01d42e9d7a" (UID: "cb05f1aa-442f-4999-b8e2-bb01d42e9d7a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:50:57.074283 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:57.074264 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb05f1aa-442f-4999-b8e2-bb01d42e9d7a-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:50:57.076022 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:57.075998 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb05f1aa-442f-4999-b8e2-bb01d42e9d7a-kube-api-access-vtvb8" pod "cb05f1aa-442f-4999-b8e2-bb01d42e9d7a" (UID: "cb05f1aa-442f-4999-b8e2-bb01d42e9d7a"). InnerVolumeSpecName "kube-api-access-vtvb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:50:57.076242 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:57.076226 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb05f1aa-442f-4999-b8e2-bb01d42e9d7a-maas-api-xks-ca" pod "cb05f1aa-442f-4999-b8e2-bb01d42e9d7a" (UID: "cb05f1aa-442f-4999-b8e2-bb01d42e9d7a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:50:57.175692 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:57.175659 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb05f1aa-442f-4999-b8e2-bb01d42e9d7a-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:50:57.175692 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:57.175688 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtvb8\" (UniqueName: \"kubernetes.io/projected/cb05f1aa-442f-4999-b8e2-bb01d42e9d7a-kube-api-access-vtvb8\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:50:57.821891 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:57.821855 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8aedbba2-cqbbp" event={"ID":"cb05f1aa-442f-4999-b8e2-bb01d42e9d7a","Type":"ContainerDied","Data":"7fb51659dd1a95bb7eea9b7ba4f4371cd54c4765af2e0adc58655d1bccb35da5"} Oct 30 16:50:57.822327 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:57.821918 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7fb51659dd1a95bb7eea9b7ba4f4371cd54c4765af2e0adc58655d1bccb35da5" Oct 30 16:50:57.822327 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:57.822052 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8aedbba2-cqbbp" Oct 30 16:50:57.993046 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:57.993020 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cfe9f0-btls9" Oct 30 16:50:58.087048 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:58.086970 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ddfcb4d-50ab-48e3-8591-b27f3105c61a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ddfcb4d-50ab-48e3-8591-b27f3105c61a-maas-api-openshift-ca\") pod \"6ddfcb4d-50ab-48e3-8591-b27f3105c61a\" (UID: \"6ddfcb4d-50ab-48e3-8591-b27f3105c61a\") " Oct 30 16:50:58.087048 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:58.087044 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ddfcb4d-50ab-48e3-8591-b27f3105c61a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ddfcb4d-50ab-48e3-8591-b27f3105c61a-maas-api-xks-ca\") pod \"6ddfcb4d-50ab-48e3-8591-b27f3105c61a\" (UID: \"6ddfcb4d-50ab-48e3-8591-b27f3105c61a\") " Oct 30 16:50:58.087237 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:58.087115 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ddfcb4d-50ab-48e3-8591-b27f3105c61a-kube-api-access-9tm6h\" (UniqueName: \"kubernetes.io/projected/6ddfcb4d-50ab-48e3-8591-b27f3105c61a-kube-api-access-9tm6h\") pod \"6ddfcb4d-50ab-48e3-8591-b27f3105c61a\" (UID: \"6ddfcb4d-50ab-48e3-8591-b27f3105c61a\") " Oct 30 16:50:58.087362 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:58.087339 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ddfcb4d-50ab-48e3-8591-b27f3105c61a-maas-api-openshift-ca" pod "6ddfcb4d-50ab-48e3-8591-b27f3105c61a" (UID: "6ddfcb4d-50ab-48e3-8591-b27f3105c61a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:50:58.087562 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:58.087541 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ddfcb4d-50ab-48e3-8591-b27f3105c61a-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:50:58.089294 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:58.089268 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ddfcb4d-50ab-48e3-8591-b27f3105c61a-kube-api-access-9tm6h" pod "6ddfcb4d-50ab-48e3-8591-b27f3105c61a" (UID: "6ddfcb4d-50ab-48e3-8591-b27f3105c61a"). InnerVolumeSpecName "kube-api-access-9tm6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:50:58.089414 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:58.089397 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ddfcb4d-50ab-48e3-8591-b27f3105c61a-maas-api-xks-ca" pod "6ddfcb4d-50ab-48e3-8591-b27f3105c61a" (UID: "6ddfcb4d-50ab-48e3-8591-b27f3105c61a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:50:58.188688 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:58.188656 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ddfcb4d-50ab-48e3-8591-b27f3105c61a-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:50:58.188688 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:58.188685 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tm6h\" (UniqueName: \"kubernetes.io/projected/6ddfcb4d-50ab-48e3-8591-b27f3105c61a-kube-api-access-9tm6h\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:50:58.833338 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:58.833306 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cfe9f0-btls9" Oct 30 16:50:58.833848 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:58.833348 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cfe9f0-btls9" event={"ID":"6ddfcb4d-50ab-48e3-8591-b27f3105c61a","Type":"ContainerDied","Data":"009fe436513a1e0a623db4d2cb27f49d99fd695dbbd3996319c69161a71e7f0e"} Oct 30 16:50:58.833848 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:50:58.833396 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="009fe436513a1e0a623db4d2cb27f49d99fd695dbbd3996319c69161a71e7f0e" Oct 30 16:51:00.342343 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:51:00.342300 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 16:51:02.590800 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:02.590759 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 16:51:05.976422 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:05.976385 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d21c86c8-w4d65"] Oct 30 16:51:05.976949 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:05.976934 4493 cpu_manager.go:414] "containerMap: removing container" podUID="6ddfcb4d-50ab-48e3-8591-b27f3105c61a" containerName="revoke-keys" Oct 30 16:51:05.977001 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:05.976950 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ddfcb4d-50ab-48e3-8591-b27f3105c61a" containerName="revoke-keys" Oct 30 16:51:05.977001 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:05.976966 4493 cpu_manager.go:414] "containerMap: removing container" podUID="15880d82-3211-4955-99ee-b84ca67eff09" containerName="revoke-keys" Oct 30 16:51:05.977001 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:05.976971 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15880d82-3211-4955-99ee-b84ca67eff09" containerName="revoke-keys" Oct 30 16:51:05.977001 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:05.976979 4493 cpu_manager.go:414] "containerMap: removing container" podUID="5e57cb61-fe2f-4a2c-930b-eee27bb86327" containerName="revoke-keys" Oct 30 16:51:05.977001 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:05.976984 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e57cb61-fe2f-4a2c-930b-eee27bb86327" containerName="revoke-keys" Oct 30 16:51:05.977001 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:05.977002 4493 cpu_manager.go:414] "containerMap: removing container" podUID="cb05f1aa-442f-4999-b8e2-bb01d42e9d7a" containerName="revoke-keys" Oct 30 16:51:05.977186 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:05.977007 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb05f1aa-442f-4999-b8e2-bb01d42e9d7a" containerName="revoke-keys" Oct 30 16:51:05.977186 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:05.977110 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb05f1aa-442f-4999-b8e2-bb01d42e9d7a" containerName="revoke-keys" Oct 30 16:51:05.977186 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:05.977130 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ddfcb4d-50ab-48e3-8591-b27f3105c61a" containerName="revoke-keys" Oct 30 16:51:05.977186 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:05.977137 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="15880d82-3211-4955-99ee-b84ca67eff09" containerName="revoke-keys" Oct 30 16:51:05.977186 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:05.977143 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e57cb61-fe2f-4a2c-930b-eee27bb86327" containerName="revoke-keys" Oct 30 16:51:05.980758 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:05.980740 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d21c86c8-w4d65" Oct 30 16:51:05.993387 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:05.993365 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:51:06.003434 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:06.003410 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 16:51:06.003775 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:06.003754 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:51:06.003910 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:06.003845 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:51:06.008161 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:06.008138 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d21c86c8-w4d65"] Oct 30 16:51:06.097315 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:06.097280 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fb4b393-7056-4e36-87bf-e3c9f0c7bfd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d21c86c8-w4d65\" (UID: \"1fb4b393-7056-4e36-87bf-e3c9f0c7bfd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d21c86c8-w4d65" Oct 30 16:51:06.097486 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:06.097338 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fb4b393-7056-4e36-87bf-e3c9f0c7bfd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d21c86c8-w4d65\" (UID: \"1fb4b393-7056-4e36-87bf-e3c9f0c7bfd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d21c86c8-w4d65" Oct 30 16:51:06.097574 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:06.097554 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwzc4\" (UniqueName: \"kubernetes.io/projected/1fb4b393-7056-4e36-87bf-e3c9f0c7bfd3-kube-api-access-dwzc4\") pod \"maas-api-revoke-sub-d21c86c8-w4d65\" (UID: \"1fb4b393-7056-4e36-87bf-e3c9f0c7bfd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d21c86c8-w4d65" Oct 30 16:51:06.199174 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:06.199142 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwzc4\" (UniqueName: \"kubernetes.io/projected/1fb4b393-7056-4e36-87bf-e3c9f0c7bfd3-kube-api-access-dwzc4\") pod \"maas-api-revoke-sub-d21c86c8-w4d65\" (UID: \"1fb4b393-7056-4e36-87bf-e3c9f0c7bfd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d21c86c8-w4d65" Oct 30 16:51:06.199363 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:06.199215 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fb4b393-7056-4e36-87bf-e3c9f0c7bfd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d21c86c8-w4d65\" (UID: \"1fb4b393-7056-4e36-87bf-e3c9f0c7bfd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d21c86c8-w4d65" Oct 30 16:51:06.199363 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:06.199264 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fb4b393-7056-4e36-87bf-e3c9f0c7bfd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d21c86c8-w4d65\" (UID: \"1fb4b393-7056-4e36-87bf-e3c9f0c7bfd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d21c86c8-w4d65" Oct 30 16:51:06.200105 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:06.200020 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fb4b393-7056-4e36-87bf-e3c9f0c7bfd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d21c86c8-w4d65\" (UID: \"1fb4b393-7056-4e36-87bf-e3c9f0c7bfd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d21c86c8-w4d65" Oct 30 16:51:06.201827 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:06.201804 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fb4b393-7056-4e36-87bf-e3c9f0c7bfd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d21c86c8-w4d65\" (UID: \"1fb4b393-7056-4e36-87bf-e3c9f0c7bfd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d21c86c8-w4d65" Oct 30 16:51:06.220290 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:06.220259 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwzc4\" (UniqueName: \"kubernetes.io/projected/1fb4b393-7056-4e36-87bf-e3c9f0c7bfd3-kube-api-access-dwzc4\") pod \"maas-api-revoke-sub-d21c86c8-w4d65\" (UID: \"1fb4b393-7056-4e36-87bf-e3c9f0c7bfd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d21c86c8-w4d65" Oct 30 16:51:06.294709 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:06.294636 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d21c86c8-w4d65" Oct 30 16:51:06.621451 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:51:06.618721 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fb4b393_7056_4e36_87bf_e3c9f0c7bfd3.slice/crio-f4ded0f075ef312e46f94c8f7e39afbd5b085843f9d4295f88299c18aec693c6 WatchSource:0}: Error finding container f4ded0f075ef312e46f94c8f7e39afbd5b085843f9d4295f88299c18aec693c6: Status 404 returned error can't find the container with id f4ded0f075ef312e46f94c8f7e39afbd5b085843f9d4295f88299c18aec693c6 Oct 30 16:51:06.621716 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:06.621672 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d21c86c8-w4d65"] Oct 30 16:51:06.898338 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:06.898300 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-793f1b59-dww6v"] Oct 30 16:51:06.914421 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:06.913223 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-793f1b59-dww6v" Oct 30 16:51:06.968725 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:06.968697 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-793f1b59-dww6v"] Oct 30 16:51:06.968906 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:06.968750 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d21c86c8-w4d65" event={"ID":"1fb4b393-7056-4e36-87bf-e3c9f0c7bfd3","Type":"ContainerStarted","Data":"f4ded0f075ef312e46f94c8f7e39afbd5b085843f9d4295f88299c18aec693c6"} Oct 30 16:51:07.011126 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:07.011089 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0959750-e09d-44cb-8a15-86408ff8de6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-793f1b59-dww6v\" (UID: \"d0959750-e09d-44cb-8a15-86408ff8de6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-793f1b59-dww6v" Oct 30 16:51:07.011757 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:07.011735 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dssl9\" (UniqueName: \"kubernetes.io/projected/d0959750-e09d-44cb-8a15-86408ff8de6e-kube-api-access-dssl9\") pod \"maas-api-revoke-sub-793f1b59-dww6v\" (UID: \"d0959750-e09d-44cb-8a15-86408ff8de6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-793f1b59-dww6v" Oct 30 16:51:07.011960 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:07.011935 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0959750-e09d-44cb-8a15-86408ff8de6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-793f1b59-dww6v\" (UID: \"d0959750-e09d-44cb-8a15-86408ff8de6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-793f1b59-dww6v" Oct 30 16:51:07.113585 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:07.113546 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0959750-e09d-44cb-8a15-86408ff8de6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-793f1b59-dww6v\" (UID: \"d0959750-e09d-44cb-8a15-86408ff8de6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-793f1b59-dww6v" Oct 30 16:51:07.113949 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:07.113926 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dssl9\" (UniqueName: \"kubernetes.io/projected/d0959750-e09d-44cb-8a15-86408ff8de6e-kube-api-access-dssl9\") pod \"maas-api-revoke-sub-793f1b59-dww6v\" (UID: \"d0959750-e09d-44cb-8a15-86408ff8de6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-793f1b59-dww6v" Oct 30 16:51:07.114083 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:07.114070 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0959750-e09d-44cb-8a15-86408ff8de6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-793f1b59-dww6v\" (UID: \"d0959750-e09d-44cb-8a15-86408ff8de6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-793f1b59-dww6v" Oct 30 16:51:07.115025 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:07.115002 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0959750-e09d-44cb-8a15-86408ff8de6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-793f1b59-dww6v\" (UID: \"d0959750-e09d-44cb-8a15-86408ff8de6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-793f1b59-dww6v" Oct 30 16:51:07.117907 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:07.117872 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0959750-e09d-44cb-8a15-86408ff8de6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-793f1b59-dww6v\" (UID: \"d0959750-e09d-44cb-8a15-86408ff8de6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-793f1b59-dww6v" Oct 30 16:51:07.136499 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:07.136471 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dssl9\" (UniqueName: \"kubernetes.io/projected/d0959750-e09d-44cb-8a15-86408ff8de6e-kube-api-access-dssl9\") pod \"maas-api-revoke-sub-793f1b59-dww6v\" (UID: \"d0959750-e09d-44cb-8a15-86408ff8de6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-793f1b59-dww6v" Oct 30 16:51:07.273070 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:07.272995 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-793f1b59-dww6v" Oct 30 16:51:07.597993 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:07.597967 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-793f1b59-dww6v"] Oct 30 16:51:07.600577 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:51:07.600550 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0959750_e09d_44cb_8a15_86408ff8de6e.slice/crio-d2bf39d951cf8e80d65594d8475e257f5ea3823752fb1e79d7654d2a8a8bf9a4 WatchSource:0}: Error finding container d2bf39d951cf8e80d65594d8475e257f5ea3823752fb1e79d7654d2a8a8bf9a4: Status 404 returned error can't find the container with id d2bf39d951cf8e80d65594d8475e257f5ea3823752fb1e79d7654d2a8a8bf9a4 Oct 30 16:51:07.972253 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:07.972216 4493 generic.go:356] "Generic (PLEG): container finished" podID="1fb4b393-7056-4e36-87bf-e3c9f0c7bfd3" containerID="debe275a684c902752a54cea4f438ce72b77ea3792559f99fc73716907729cc0" exitCode=0 Oct 30 16:51:07.972403 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:07.972257 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d21c86c8-w4d65" event={"ID":"1fb4b393-7056-4e36-87bf-e3c9f0c7bfd3","Type":"ContainerDied","Data":"debe275a684c902752a54cea4f438ce72b77ea3792559f99fc73716907729cc0"} Oct 30 16:51:07.973739 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:07.973715 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-793f1b59-dww6v" event={"ID":"d0959750-e09d-44cb-8a15-86408ff8de6e","Type":"ContainerStarted","Data":"d2bf39d951cf8e80d65594d8475e257f5ea3823752fb1e79d7654d2a8a8bf9a4"} Oct 30 16:51:08.986557 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:08.986517 4493 generic.go:356] "Generic (PLEG): container finished" podID="d0959750-e09d-44cb-8a15-86408ff8de6e" containerID="fc3dfd491653aa8fd428f419d8d36ff4893675b5d9a0a9e11a87e0d84b808f46" exitCode=0 Oct 30 16:51:08.987071 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:08.986645 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-793f1b59-dww6v" event={"ID":"d0959750-e09d-44cb-8a15-86408ff8de6e","Type":"ContainerDied","Data":"fc3dfd491653aa8fd428f419d8d36ff4893675b5d9a0a9e11a87e0d84b808f46"} Oct 30 16:51:09.000255 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:09.000224 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-cfe9f0-69b4bdf948-tvs7d"] Oct 30 16:51:09.000451 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:09.000420 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-cfe9f0-69b4bdf948-tvs7d" podUID="13cca9a3-0b99-497e-9abd-22b739f7a1b3" containerName="payload-processing" containerID="cri-o://ace9d19c2f6fa5ffc2356c5e256c4783fa4285087ddc41cb1a530996e1465244" gracePeriod=30 Oct 30 16:51:09.050599 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:09.050551 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-cfe9f0-76f945bc5f-wt6jt"] Oct 30 16:51:09.051187 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:09.050833 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cfe9f0-76f945bc5f-wt6jt" podUID="5943c777-6afd-4066-b448-1f273de792f0" containerName="payload-pre-processing" containerID="cri-o://ca433fe9b7cbefa4da621d37fe9ef7a771a81bd2217c1d8a95cb52e381f2e10f" gracePeriod=30 Oct 30 16:51:09.207914 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:09.207882 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c408f1-pwkxz"] Oct 30 16:51:09.218271 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:09.218196 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c408f1-pwkxz"] Oct 30 16:51:09.283335 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:09.283309 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d21c86c8-w4d65" Oct 30 16:51:09.361160 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:09.360585 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fb4b393-7056-4e36-87bf-e3c9f0c7bfd3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fb4b393-7056-4e36-87bf-e3c9f0c7bfd3-maas-api-xks-ca\") pod \"1fb4b393-7056-4e36-87bf-e3c9f0c7bfd3\" (UID: \"1fb4b393-7056-4e36-87bf-e3c9f0c7bfd3\") " Oct 30 16:51:09.361160 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:09.360765 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fb4b393-7056-4e36-87bf-e3c9f0c7bfd3-kube-api-access-dwzc4\" (UniqueName: \"kubernetes.io/projected/1fb4b393-7056-4e36-87bf-e3c9f0c7bfd3-kube-api-access-dwzc4\") pod \"1fb4b393-7056-4e36-87bf-e3c9f0c7bfd3\" (UID: \"1fb4b393-7056-4e36-87bf-e3c9f0c7bfd3\") " Oct 30 16:51:09.361160 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:09.360849 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fb4b393-7056-4e36-87bf-e3c9f0c7bfd3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fb4b393-7056-4e36-87bf-e3c9f0c7bfd3-maas-api-openshift-ca\") pod \"1fb4b393-7056-4e36-87bf-e3c9f0c7bfd3\" (UID: \"1fb4b393-7056-4e36-87bf-e3c9f0c7bfd3\") " Oct 30 16:51:09.362312 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:09.362003 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fb4b393-7056-4e36-87bf-e3c9f0c7bfd3-maas-api-openshift-ca" pod "1fb4b393-7056-4e36-87bf-e3c9f0c7bfd3" (UID: "1fb4b393-7056-4e36-87bf-e3c9f0c7bfd3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:51:09.387992 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:09.374933 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fb4b393-7056-4e36-87bf-e3c9f0c7bfd3-kube-api-access-dwzc4" pod "1fb4b393-7056-4e36-87bf-e3c9f0c7bfd3" (UID: "1fb4b393-7056-4e36-87bf-e3c9f0c7bfd3"). InnerVolumeSpecName "kube-api-access-dwzc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:51:09.394269 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:09.391414 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fb4b393-7056-4e36-87bf-e3c9f0c7bfd3-maas-api-xks-ca" pod "1fb4b393-7056-4e36-87bf-e3c9f0c7bfd3" (UID: "1fb4b393-7056-4e36-87bf-e3c9f0c7bfd3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:51:09.464459 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:09.463379 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fb4b393-7056-4e36-87bf-e3c9f0c7bfd3-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:51:09.465110 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:09.464737 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-cfe9f0-69b4bdf948-tvs7d" Oct 30 16:51:09.465258 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:09.465243 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwzc4\" (UniqueName: \"kubernetes.io/projected/1fb4b393-7056-4e36-87bf-e3c9f0c7bfd3-kube-api-access-dwzc4\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:51:09.465364 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:09.465352 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fb4b393-7056-4e36-87bf-e3c9f0c7bfd3-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:51:09.525864 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:09.525756 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cfe9f0-76f945bc5f-wt6jt" Oct 30 16:51:09.566296 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:09.566252 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5943c777-6afd-4066-b448-1f273de792f0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5943c777-6afd-4066-b448-1f273de792f0-plugins-config-volume\") pod \"5943c777-6afd-4066-b448-1f273de792f0\" (UID: \"5943c777-6afd-4066-b448-1f273de792f0\") " Oct 30 16:51:09.566505 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:09.566476 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13cca9a3-0b99-497e-9abd-22b739f7a1b3-kube-api-access-bn9gd\" (UniqueName: \"kubernetes.io/projected/13cca9a3-0b99-497e-9abd-22b739f7a1b3-kube-api-access-bn9gd\") pod \"13cca9a3-0b99-497e-9abd-22b739f7a1b3\" (UID: \"13cca9a3-0b99-497e-9abd-22b739f7a1b3\") " Oct 30 16:51:09.566616 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:09.566583 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5943c777-6afd-4066-b448-1f273de792f0-plugins-config-volume" pod "5943c777-6afd-4066-b448-1f273de792f0" (UID: "5943c777-6afd-4066-b448-1f273de792f0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:51:09.567382 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:09.566954 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5943c777-6afd-4066-b448-1f273de792f0-kube-api-access-5gh97\" (UniqueName: \"kubernetes.io/projected/5943c777-6afd-4066-b448-1f273de792f0-kube-api-access-5gh97\") pod \"5943c777-6afd-4066-b448-1f273de792f0\" (UID: \"5943c777-6afd-4066-b448-1f273de792f0\") " Oct 30 16:51:09.567382 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:09.567049 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13cca9a3-0b99-497e-9abd-22b739f7a1b3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13cca9a3-0b99-497e-9abd-22b739f7a1b3-plugins-config-volume\") pod \"13cca9a3-0b99-497e-9abd-22b739f7a1b3\" (UID: \"13cca9a3-0b99-497e-9abd-22b739f7a1b3\") " Oct 30 16:51:09.567630 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:09.567506 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13cca9a3-0b99-497e-9abd-22b739f7a1b3-plugins-config-volume" pod "13cca9a3-0b99-497e-9abd-22b739f7a1b3" (UID: "13cca9a3-0b99-497e-9abd-22b739f7a1b3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:51:09.572620 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:09.568336 4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13cca9a3-0b99-497e-9abd-22b739f7a1b3-plugins-config-volume\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:51:09.572620 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:09.568364 4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5943c777-6afd-4066-b448-1f273de792f0-plugins-config-volume\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:51:09.572620 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:09.569696 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13cca9a3-0b99-497e-9abd-22b739f7a1b3-kube-api-access-bn9gd" pod "13cca9a3-0b99-497e-9abd-22b739f7a1b3" (UID: "13cca9a3-0b99-497e-9abd-22b739f7a1b3"). InnerVolumeSpecName "kube-api-access-bn9gd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:51:09.572620 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:09.571473 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5943c777-6afd-4066-b448-1f273de792f0-kube-api-access-5gh97" pod "5943c777-6afd-4066-b448-1f273de792f0" (UID: "5943c777-6afd-4066-b448-1f273de792f0"). InnerVolumeSpecName "kube-api-access-5gh97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:51:09.669483 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:09.669449 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bn9gd\" (UniqueName: \"kubernetes.io/projected/13cca9a3-0b99-497e-9abd-22b739f7a1b3-kube-api-access-bn9gd\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:51:09.669483 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:09.669477 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gh97\" (UniqueName: \"kubernetes.io/projected/5943c777-6afd-4066-b448-1f273de792f0-kube-api-access-5gh97\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:51:09.864481 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:09.864401 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cfe9f0-btls9"] Oct 30 16:51:09.867754 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:09.867730 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cfe9f0-btls9"] Oct 30 16:51:09.996732 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:09.996702 4493 generic.go:356] "Generic (PLEG): container finished" podID="5943c777-6afd-4066-b448-1f273de792f0" containerID="ca433fe9b7cbefa4da621d37fe9ef7a771a81bd2217c1d8a95cb52e381f2e10f" exitCode=0 Oct 30 16:51:09.997211 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:09.996765 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cfe9f0-76f945bc5f-wt6jt" Oct 30 16:51:09.997211 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:09.996815 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cfe9f0-76f945bc5f-wt6jt" event={"ID":"5943c777-6afd-4066-b448-1f273de792f0","Type":"ContainerDied","Data":"ca433fe9b7cbefa4da621d37fe9ef7a771a81bd2217c1d8a95cb52e381f2e10f"} Oct 30 16:51:09.997211 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:09.996867 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cfe9f0-76f945bc5f-wt6jt" event={"ID":"5943c777-6afd-4066-b448-1f273de792f0","Type":"ContainerDied","Data":"bc53880f3679df1b2f3e9cc1774f6333db97a0d085a5a6ff941a589d05b2d63d"} Oct 30 16:51:09.997211 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:09.996895 4493 scope.go:122] "RemoveContainer" containerID="ca433fe9b7cbefa4da621d37fe9ef7a771a81bd2217c1d8a95cb52e381f2e10f" Oct 30 16:51:09.998615 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:09.998590 4493 generic.go:356] "Generic (PLEG): container finished" podID="13cca9a3-0b99-497e-9abd-22b739f7a1b3" containerID="ace9d19c2f6fa5ffc2356c5e256c4783fa4285087ddc41cb1a530996e1465244" exitCode=0 Oct 30 16:51:09.998739 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:09.998650 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-cfe9f0-69b4bdf948-tvs7d" Oct 30 16:51:09.998739 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:09.998669 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-cfe9f0-69b4bdf948-tvs7d" event={"ID":"13cca9a3-0b99-497e-9abd-22b739f7a1b3","Type":"ContainerDied","Data":"ace9d19c2f6fa5ffc2356c5e256c4783fa4285087ddc41cb1a530996e1465244"} Oct 30 16:51:09.998739 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:09.998730 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-cfe9f0-69b4bdf948-tvs7d" event={"ID":"13cca9a3-0b99-497e-9abd-22b739f7a1b3","Type":"ContainerDied","Data":"40f96042024e35ef25c06c6021563c7c95c4f35e41ab00b29edef5b8c977cc86"} Oct 30 16:51:10.000314 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:10.000288 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d21c86c8-w4d65" Oct 30 16:51:10.000431 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:10.000333 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d21c86c8-w4d65" event={"ID":"1fb4b393-7056-4e36-87bf-e3c9f0c7bfd3","Type":"ContainerDied","Data":"f4ded0f075ef312e46f94c8f7e39afbd5b085843f9d4295f88299c18aec693c6"} Oct 30 16:51:10.000431 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:10.000371 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4ded0f075ef312e46f94c8f7e39afbd5b085843f9d4295f88299c18aec693c6" Oct 30 16:51:10.011947 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:10.011908 4493 scope.go:122] "RemoveContainer" containerID="ca433fe9b7cbefa4da621d37fe9ef7a771a81bd2217c1d8a95cb52e381f2e10f" Oct 30 16:51:10.012341 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:51:10.012318 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca433fe9b7cbefa4da621d37fe9ef7a771a81bd2217c1d8a95cb52e381f2e10f\": container with ID starting with ca433fe9b7cbefa4da621d37fe9ef7a771a81bd2217c1d8a95cb52e381f2e10f not found: ID does not exist" containerID="ca433fe9b7cbefa4da621d37fe9ef7a771a81bd2217c1d8a95cb52e381f2e10f" Oct 30 16:51:10.012490 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:10.012467 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca433fe9b7cbefa4da621d37fe9ef7a771a81bd2217c1d8a95cb52e381f2e10f"} err="failed to get container status \"ca433fe9b7cbefa4da621d37fe9ef7a771a81bd2217c1d8a95cb52e381f2e10f\": rpc error: code = NotFound desc = could not find container \"ca433fe9b7cbefa4da621d37fe9ef7a771a81bd2217c1d8a95cb52e381f2e10f\": container with ID starting with ca433fe9b7cbefa4da621d37fe9ef7a771a81bd2217c1d8a95cb52e381f2e10f not found: ID does not exist" Oct 30 16:51:10.012490 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:10.012490 4493 scope.go:122] "RemoveContainer" containerID="ace9d19c2f6fa5ffc2356c5e256c4783fa4285087ddc41cb1a530996e1465244" Oct 30 16:51:10.023900 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:10.023877 4493 scope.go:122] "RemoveContainer" containerID="ace9d19c2f6fa5ffc2356c5e256c4783fa4285087ddc41cb1a530996e1465244" Oct 30 16:51:10.024242 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:51:10.024215 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ace9d19c2f6fa5ffc2356c5e256c4783fa4285087ddc41cb1a530996e1465244\": container with ID starting with ace9d19c2f6fa5ffc2356c5e256c4783fa4285087ddc41cb1a530996e1465244 not found: ID does not exist" containerID="ace9d19c2f6fa5ffc2356c5e256c4783fa4285087ddc41cb1a530996e1465244" Oct 30 16:51:10.024322 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:10.024253 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ace9d19c2f6fa5ffc2356c5e256c4783fa4285087ddc41cb1a530996e1465244"} err="failed to get container status \"ace9d19c2f6fa5ffc2356c5e256c4783fa4285087ddc41cb1a530996e1465244\": rpc error: code = NotFound desc = could not find container \"ace9d19c2f6fa5ffc2356c5e256c4783fa4285087ddc41cb1a530996e1465244\": container with ID starting with ace9d19c2f6fa5ffc2356c5e256c4783fa4285087ddc41cb1a530996e1465244 not found: ID does not exist" Oct 30 16:51:10.060777 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:10.060747 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-cfe9f0-76f945bc5f-wt6jt"] Oct 30 16:51:10.065573 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:10.065547 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-cfe9f0-76f945bc5f-wt6jt"] Oct 30 16:51:10.093460 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:10.093432 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-cfe9f0-69b4bdf948-tvs7d"] Oct 30 16:51:10.098094 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:10.098070 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-cfe9f0-69b4bdf948-tvs7d"] Oct 30 16:51:10.162138 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:10.162115 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-793f1b59-dww6v" Oct 30 16:51:10.277196 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:10.277171 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0959750-e09d-44cb-8a15-86408ff8de6e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0959750-e09d-44cb-8a15-86408ff8de6e-maas-api-openshift-ca\") pod \"d0959750-e09d-44cb-8a15-86408ff8de6e\" (UID: \"d0959750-e09d-44cb-8a15-86408ff8de6e\") " Oct 30 16:51:10.277346 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:10.277333 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0959750-e09d-44cb-8a15-86408ff8de6e-kube-api-access-dssl9\" (UniqueName: \"kubernetes.io/projected/d0959750-e09d-44cb-8a15-86408ff8de6e-kube-api-access-dssl9\") pod \"d0959750-e09d-44cb-8a15-86408ff8de6e\" (UID: \"d0959750-e09d-44cb-8a15-86408ff8de6e\") " Oct 30 16:51:10.277460 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:10.277444 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0959750-e09d-44cb-8a15-86408ff8de6e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0959750-e09d-44cb-8a15-86408ff8de6e-maas-api-xks-ca\") pod \"d0959750-e09d-44cb-8a15-86408ff8de6e\" (UID: \"d0959750-e09d-44cb-8a15-86408ff8de6e\") " Oct 30 16:51:10.277520 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:10.277485 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0959750-e09d-44cb-8a15-86408ff8de6e-maas-api-openshift-ca" pod "d0959750-e09d-44cb-8a15-86408ff8de6e" (UID: "d0959750-e09d-44cb-8a15-86408ff8de6e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:51:10.277917 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:10.277897 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0959750-e09d-44cb-8a15-86408ff8de6e-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:51:10.279599 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:10.279566 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0959750-e09d-44cb-8a15-86408ff8de6e-maas-api-xks-ca" pod "d0959750-e09d-44cb-8a15-86408ff8de6e" (UID: "d0959750-e09d-44cb-8a15-86408ff8de6e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:51:10.279699 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:10.279601 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0959750-e09d-44cb-8a15-86408ff8de6e-kube-api-access-dssl9" pod "d0959750-e09d-44cb-8a15-86408ff8de6e" (UID: "d0959750-e09d-44cb-8a15-86408ff8de6e"). InnerVolumeSpecName "kube-api-access-dssl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:51:10.378624 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:10.378588 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0959750-e09d-44cb-8a15-86408ff8de6e-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:51:10.378624 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:10.378618 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dssl9\" (UniqueName: \"kubernetes.io/projected/d0959750-e09d-44cb-8a15-86408ff8de6e-kube-api-access-dssl9\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:51:10.935585 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:10.935552 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13cca9a3-0b99-497e-9abd-22b739f7a1b3" path="/var/lib/kubelet/pods/13cca9a3-0b99-497e-9abd-22b739f7a1b3/volumes" Oct 30 16:51:10.936092 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:10.936074 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15880d82-3211-4955-99ee-b84ca67eff09" path="/var/lib/kubelet/pods/15880d82-3211-4955-99ee-b84ca67eff09/volumes" Oct 30 16:51:10.936538 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:10.936513 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5943c777-6afd-4066-b448-1f273de792f0" path="/var/lib/kubelet/pods/5943c777-6afd-4066-b448-1f273de792f0/volumes" Oct 30 16:51:10.937262 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:10.937242 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ddfcb4d-50ab-48e3-8591-b27f3105c61a" path="/var/lib/kubelet/pods/6ddfcb4d-50ab-48e3-8591-b27f3105c61a/volumes" Oct 30 16:51:11.014014 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:11.013974 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-793f1b59-dww6v" event={"ID":"d0959750-e09d-44cb-8a15-86408ff8de6e","Type":"ContainerDied","Data":"d2bf39d951cf8e80d65594d8475e257f5ea3823752fb1e79d7654d2a8a8bf9a4"} Oct 30 16:51:11.014504 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:11.014486 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-793f1b59-dww6v" Oct 30 16:51:11.014650 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:11.014491 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2bf39d951cf8e80d65594d8475e257f5ea3823752fb1e79d7654d2a8a8bf9a4" Oct 30 16:51:12.451655 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:12.451620 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-06ac18-dcldd"] Oct 30 16:51:12.459632 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:12.459606 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-06ac18-dcldd"] Oct 30 16:51:12.933576 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:12.933545 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="191759dd-4684-48b4-92a1-7890937c34fe" path="/var/lib/kubelet/pods/191759dd-4684-48b4-92a1-7890937c34fe/volumes" Oct 30 16:51:16.424666 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:16.424633 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-573e69d7-7d9zk"] Oct 30 16:51:16.425162 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:16.425149 4493 cpu_manager.go:414] "containerMap: removing container" podUID="d0959750-e09d-44cb-8a15-86408ff8de6e" containerName="revoke-keys" Oct 30 16:51:16.425206 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:16.425163 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0959750-e09d-44cb-8a15-86408ff8de6e" containerName="revoke-keys" Oct 30 16:51:16.425206 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:16.425178 4493 cpu_manager.go:414] "containerMap: removing container" podUID="1fb4b393-7056-4e36-87bf-e3c9f0c7bfd3" containerName="revoke-keys" Oct 30 16:51:16.425206 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:16.425183 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fb4b393-7056-4e36-87bf-e3c9f0c7bfd3" containerName="revoke-keys" Oct 30 16:51:16.425290 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:16.425208 4493 cpu_manager.go:414] "containerMap: removing container" podUID="13cca9a3-0b99-497e-9abd-22b739f7a1b3" containerName="payload-processing" Oct 30 16:51:16.425290 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:16.425213 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13cca9a3-0b99-497e-9abd-22b739f7a1b3" containerName="payload-processing" Oct 30 16:51:16.425290 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:16.425229 4493 cpu_manager.go:414] "containerMap: removing container" podUID="5943c777-6afd-4066-b448-1f273de792f0" containerName="payload-pre-processing" Oct 30 16:51:16.425290 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:16.425234 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5943c777-6afd-4066-b448-1f273de792f0" containerName="payload-pre-processing" Oct 30 16:51:16.425423 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:16.425311 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="13cca9a3-0b99-497e-9abd-22b739f7a1b3" containerName="payload-processing" Oct 30 16:51:16.425423 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:16.425323 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="5943c777-6afd-4066-b448-1f273de792f0" containerName="payload-pre-processing" Oct 30 16:51:16.425423 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:16.425332 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0959750-e09d-44cb-8a15-86408ff8de6e" containerName="revoke-keys" Oct 30 16:51:16.425423 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:16.425338 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fb4b393-7056-4e36-87bf-e3c9f0c7bfd3" containerName="revoke-keys" Oct 30 16:51:16.428358 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:16.428341 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-573e69d7-7d9zk" Oct 30 16:51:16.443081 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:16.443059 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:51:16.451511 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:16.451463 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 16:51:16.451511 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:16.451493 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:51:16.451820 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:16.451763 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:51:16.456929 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:16.456908 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-573e69d7-7d9zk"] Oct 30 16:51:16.466242 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:16.465843 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0bb0921-5020-450a-81bf-3f1e6f385939-maas-api-xks-ca\") pod \"maas-api-revoke-sub-573e69d7-7d9zk\" (UID: \"b0bb0921-5020-450a-81bf-3f1e6f385939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-573e69d7-7d9zk" Oct 30 16:51:16.466242 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:16.465905 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfksj\" (UniqueName: \"kubernetes.io/projected/b0bb0921-5020-450a-81bf-3f1e6f385939-kube-api-access-cfksj\") pod \"maas-api-revoke-sub-573e69d7-7d9zk\" (UID: \"b0bb0921-5020-450a-81bf-3f1e6f385939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-573e69d7-7d9zk" Oct 30 16:51:16.466242 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:16.466002 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0bb0921-5020-450a-81bf-3f1e6f385939-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-573e69d7-7d9zk\" (UID: \"b0bb0921-5020-450a-81bf-3f1e6f385939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-573e69d7-7d9zk" Oct 30 16:51:16.567160 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:16.567132 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0bb0921-5020-450a-81bf-3f1e6f385939-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-573e69d7-7d9zk\" (UID: \"b0bb0921-5020-450a-81bf-3f1e6f385939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-573e69d7-7d9zk" Oct 30 16:51:16.567318 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:16.567241 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0bb0921-5020-450a-81bf-3f1e6f385939-maas-api-xks-ca\") pod \"maas-api-revoke-sub-573e69d7-7d9zk\" (UID: \"b0bb0921-5020-450a-81bf-3f1e6f385939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-573e69d7-7d9zk" Oct 30 16:51:16.567318 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:16.567262 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfksj\" (UniqueName: \"kubernetes.io/projected/b0bb0921-5020-450a-81bf-3f1e6f385939-kube-api-access-cfksj\") pod \"maas-api-revoke-sub-573e69d7-7d9zk\" (UID: \"b0bb0921-5020-450a-81bf-3f1e6f385939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-573e69d7-7d9zk" Oct 30 16:51:16.567757 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:16.567731 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0bb0921-5020-450a-81bf-3f1e6f385939-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-573e69d7-7d9zk\" (UID: \"b0bb0921-5020-450a-81bf-3f1e6f385939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-573e69d7-7d9zk" Oct 30 16:51:16.569656 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:16.569638 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0bb0921-5020-450a-81bf-3f1e6f385939-maas-api-xks-ca\") pod \"maas-api-revoke-sub-573e69d7-7d9zk\" (UID: \"b0bb0921-5020-450a-81bf-3f1e6f385939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-573e69d7-7d9zk" Oct 30 16:51:16.593468 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:16.593442 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfksj\" (UniqueName: \"kubernetes.io/projected/b0bb0921-5020-450a-81bf-3f1e6f385939-kube-api-access-cfksj\") pod \"maas-api-revoke-sub-573e69d7-7d9zk\" (UID: \"b0bb0921-5020-450a-81bf-3f1e6f385939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-573e69d7-7d9zk" Oct 30 16:51:16.742730 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:16.742648 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-573e69d7-7d9zk" Oct 30 16:51:17.053047 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:17.053027 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-573e69d7-7d9zk"] Oct 30 16:51:17.056011 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:51:17.055988 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0bb0921_5020_450a_81bf_3f1e6f385939.slice/crio-b37b5545f85ceb65c66ff09c0a16237affe88698d2591f19511c82065813005f WatchSource:0}: Error finding container b37b5545f85ceb65c66ff09c0a16237affe88698d2591f19511c82065813005f: Status 404 returned error can't find the container with id b37b5545f85ceb65c66ff09c0a16237affe88698d2591f19511c82065813005f Oct 30 16:51:17.076643 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:17.076616 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-573e69d7-7d9zk" event={"ID":"b0bb0921-5020-450a-81bf-3f1e6f385939","Type":"ContainerStarted","Data":"b37b5545f85ceb65c66ff09c0a16237affe88698d2591f19511c82065813005f"} Oct 30 16:51:18.121845 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:18.117854 4493 generic.go:356] "Generic (PLEG): container finished" podID="b0bb0921-5020-450a-81bf-3f1e6f385939" containerID="8de68f126bd1b459ed3c43a6310b564e99dc79ed80c91d27b09076b33a7bbb44" exitCode=0 Oct 30 16:51:18.121845 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:18.117917 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-573e69d7-7d9zk" event={"ID":"b0bb0921-5020-450a-81bf-3f1e6f385939","Type":"ContainerDied","Data":"8de68f126bd1b459ed3c43a6310b564e99dc79ed80c91d27b09076b33a7bbb44"} Oct 30 16:51:19.298351 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:19.298329 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-573e69d7-7d9zk" Oct 30 16:51:19.429435 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:19.429404 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0bb0921-5020-450a-81bf-3f1e6f385939-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0bb0921-5020-450a-81bf-3f1e6f385939-maas-api-xks-ca\") pod \"b0bb0921-5020-450a-81bf-3f1e6f385939\" (UID: \"b0bb0921-5020-450a-81bf-3f1e6f385939\") " Oct 30 16:51:19.429625 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:19.429461 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0bb0921-5020-450a-81bf-3f1e6f385939-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0bb0921-5020-450a-81bf-3f1e6f385939-maas-api-openshift-ca\") pod \"b0bb0921-5020-450a-81bf-3f1e6f385939\" (UID: \"b0bb0921-5020-450a-81bf-3f1e6f385939\") " Oct 30 16:51:19.429625 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:19.429480 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0bb0921-5020-450a-81bf-3f1e6f385939-kube-api-access-cfksj\" (UniqueName: \"kubernetes.io/projected/b0bb0921-5020-450a-81bf-3f1e6f385939-kube-api-access-cfksj\") pod \"b0bb0921-5020-450a-81bf-3f1e6f385939\" (UID: \"b0bb0921-5020-450a-81bf-3f1e6f385939\") " Oct 30 16:51:19.429794 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:19.429760 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0bb0921-5020-450a-81bf-3f1e6f385939-maas-api-openshift-ca" pod "b0bb0921-5020-450a-81bf-3f1e6f385939" (UID: "b0bb0921-5020-450a-81bf-3f1e6f385939"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:51:19.430113 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:19.430096 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0bb0921-5020-450a-81bf-3f1e6f385939-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:51:19.432293 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:19.432267 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0bb0921-5020-450a-81bf-3f1e6f385939-maas-api-xks-ca" pod "b0bb0921-5020-450a-81bf-3f1e6f385939" (UID: "b0bb0921-5020-450a-81bf-3f1e6f385939"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:51:19.432413 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:19.432300 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0bb0921-5020-450a-81bf-3f1e6f385939-kube-api-access-cfksj" pod "b0bb0921-5020-450a-81bf-3f1e6f385939" (UID: "b0bb0921-5020-450a-81bf-3f1e6f385939"). InnerVolumeSpecName "kube-api-access-cfksj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:51:19.530974 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:19.530940 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfksj\" (UniqueName: \"kubernetes.io/projected/b0bb0921-5020-450a-81bf-3f1e6f385939-kube-api-access-cfksj\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:51:19.530974 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:19.530964 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0bb0921-5020-450a-81bf-3f1e6f385939-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:51:20.138590 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:20.138558 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-573e69d7-7d9zk" event={"ID":"b0bb0921-5020-450a-81bf-3f1e6f385939","Type":"ContainerDied","Data":"b37b5545f85ceb65c66ff09c0a16237affe88698d2591f19511c82065813005f"} Oct 30 16:51:20.138590 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:20.138589 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-573e69d7-7d9zk" Oct 30 16:51:20.138882 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:20.138608 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b37b5545f85ceb65c66ff09c0a16237affe88698d2591f19511c82065813005f" Oct 30 16:51:21.609291 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:21.609250 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b744ec4f-nttt6"] Oct 30 16:51:21.610033 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:21.610012 4493 cpu_manager.go:414] "containerMap: removing container" podUID="b0bb0921-5020-450a-81bf-3f1e6f385939" containerName="revoke-keys" Oct 30 16:51:21.610033 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:21.610034 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0bb0921-5020-450a-81bf-3f1e6f385939" containerName="revoke-keys" Oct 30 16:51:21.610201 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:21.610161 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0bb0921-5020-450a-81bf-3f1e6f385939" containerName="revoke-keys" Oct 30 16:51:21.612983 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:21.612963 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b744ec4f-nttt6" Oct 30 16:51:21.635375 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:21.635348 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:51:21.635523 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:21.635479 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 16:51:21.635651 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:21.635630 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:51:21.635878 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:21.635862 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:51:21.639639 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:21.639619 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b744ec4f-nttt6"] Oct 30 16:51:21.757618 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:21.757586 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de478b61-79a3-4aae-9bc8-cdda3391855d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b744ec4f-nttt6\" (UID: \"de478b61-79a3-4aae-9bc8-cdda3391855d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b744ec4f-nttt6" Oct 30 16:51:21.757807 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:21.757662 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de478b61-79a3-4aae-9bc8-cdda3391855d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b744ec4f-nttt6\" (UID: \"de478b61-79a3-4aae-9bc8-cdda3391855d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b744ec4f-nttt6" Oct 30 16:51:21.757807 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:21.757735 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrq74\" (UniqueName: \"kubernetes.io/projected/de478b61-79a3-4aae-9bc8-cdda3391855d-kube-api-access-qrq74\") pod \"maas-api-revoke-sub-b744ec4f-nttt6\" (UID: \"de478b61-79a3-4aae-9bc8-cdda3391855d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b744ec4f-nttt6" Oct 30 16:51:21.859048 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:21.859013 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de478b61-79a3-4aae-9bc8-cdda3391855d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b744ec4f-nttt6\" (UID: \"de478b61-79a3-4aae-9bc8-cdda3391855d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b744ec4f-nttt6" Oct 30 16:51:21.859220 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:21.859064 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de478b61-79a3-4aae-9bc8-cdda3391855d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b744ec4f-nttt6\" (UID: \"de478b61-79a3-4aae-9bc8-cdda3391855d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b744ec4f-nttt6" Oct 30 16:51:21.859220 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:21.859124 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrq74\" (UniqueName: \"kubernetes.io/projected/de478b61-79a3-4aae-9bc8-cdda3391855d-kube-api-access-qrq74\") pod \"maas-api-revoke-sub-b744ec4f-nttt6\" (UID: \"de478b61-79a3-4aae-9bc8-cdda3391855d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b744ec4f-nttt6" Oct 30 16:51:21.859832 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:21.859755 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de478b61-79a3-4aae-9bc8-cdda3391855d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b744ec4f-nttt6\" (UID: \"de478b61-79a3-4aae-9bc8-cdda3391855d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b744ec4f-nttt6" Oct 30 16:51:21.861426 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:21.861408 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de478b61-79a3-4aae-9bc8-cdda3391855d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b744ec4f-nttt6\" (UID: \"de478b61-79a3-4aae-9bc8-cdda3391855d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b744ec4f-nttt6" Oct 30 16:51:21.880352 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:21.880326 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrq74\" (UniqueName: \"kubernetes.io/projected/de478b61-79a3-4aae-9bc8-cdda3391855d-kube-api-access-qrq74\") pod \"maas-api-revoke-sub-b744ec4f-nttt6\" (UID: \"de478b61-79a3-4aae-9bc8-cdda3391855d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b744ec4f-nttt6" Oct 30 16:51:21.927185 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:21.927161 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b744ec4f-nttt6" Oct 30 16:51:22.239840 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:22.239807 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b744ec4f-nttt6"] Oct 30 16:51:22.243221 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:51:22.243195 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde478b61_79a3_4aae_9bc8_cdda3391855d.slice/crio-b5917a82aa8c480ba903c0e5531d22d76b03e08f5214ba3468a4ae2c51fad83c WatchSource:0}: Error finding container b5917a82aa8c480ba903c0e5531d22d76b03e08f5214ba3468a4ae2c51fad83c: Status 404 returned error can't find the container with id b5917a82aa8c480ba903c0e5531d22d76b03e08f5214ba3468a4ae2c51fad83c Oct 30 16:51:23.176401 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:23.176359 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b744ec4f-nttt6" event={"ID":"de478b61-79a3-4aae-9bc8-cdda3391855d","Type":"ContainerStarted","Data":"373273ad35cffe31981a317f7edf4c2633f0bff5c7f7c9e0c408db58b9f39a62"} Oct 30 16:51:23.180101 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:23.180057 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b744ec4f-nttt6" event={"ID":"de478b61-79a3-4aae-9bc8-cdda3391855d","Type":"ContainerStarted","Data":"b5917a82aa8c480ba903c0e5531d22d76b03e08f5214ba3468a4ae2c51fad83c"} Oct 30 16:51:23.240811 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:23.237608 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b744ec4f-nttt6" podStartSLOduration=2.23758808 podStartE2EDuration="2.23758808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:51:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:51:23.218745652 +0000 UTC m=+1710.445920757" watchObservedRunningTime="2026-10-30 16:51:23.23758808 +0000 UTC m=+1710.464763182" Oct 30 16:51:24.196819 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:24.196766 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6d5903-vpcxv"] Oct 30 16:51:24.209200 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:24.209161 4493 generic.go:356] "Generic (PLEG): container finished" podID="de478b61-79a3-4aae-9bc8-cdda3391855d" containerID="373273ad35cffe31981a317f7edf4c2633f0bff5c7f7c9e0c408db58b9f39a62" exitCode=0 Oct 30 16:51:24.211331 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:24.209962 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b744ec4f-nttt6" event={"ID":"de478b61-79a3-4aae-9bc8-cdda3391855d","Type":"ContainerDied","Data":"373273ad35cffe31981a317f7edf4c2633f0bff5c7f7c9e0c408db58b9f39a62"} Oct 30 16:51:24.211331 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:24.210062 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6d5903-vpcxv" Oct 30 16:51:24.250379 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:24.250335 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6d5903-vpcxv"] Oct 30 16:51:24.339584 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:24.339546 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlx7x\" (UniqueName: \"kubernetes.io/projected/3cc7c3ae-4732-442b-beb2-636e5568bf8c-kube-api-access-qlx7x\") pod \"maas-api-revoke-keys-e2e-ipp-b-6d5903-vpcxv\" (UID: \"3cc7c3ae-4732-442b-beb2-636e5568bf8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6d5903-vpcxv" Oct 30 16:51:24.356941 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:24.356906 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cc7c3ae-4732-442b-beb2-636e5568bf8c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6d5903-vpcxv\" (UID: \"3cc7c3ae-4732-442b-beb2-636e5568bf8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6d5903-vpcxv" Oct 30 16:51:24.357232 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:24.357208 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cc7c3ae-4732-442b-beb2-636e5568bf8c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6d5903-vpcxv\" (UID: \"3cc7c3ae-4732-442b-beb2-636e5568bf8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6d5903-vpcxv" Oct 30 16:51:24.461621 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:24.461531 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cc7c3ae-4732-442b-beb2-636e5568bf8c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6d5903-vpcxv\" (UID: \"3cc7c3ae-4732-442b-beb2-636e5568bf8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6d5903-vpcxv" Oct 30 16:51:24.462031 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:24.462010 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlx7x\" (UniqueName: \"kubernetes.io/projected/3cc7c3ae-4732-442b-beb2-636e5568bf8c-kube-api-access-qlx7x\") pod \"maas-api-revoke-keys-e2e-ipp-b-6d5903-vpcxv\" (UID: \"3cc7c3ae-4732-442b-beb2-636e5568bf8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6d5903-vpcxv" Oct 30 16:51:24.462199 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:24.462181 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cc7c3ae-4732-442b-beb2-636e5568bf8c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6d5903-vpcxv\" (UID: \"3cc7c3ae-4732-442b-beb2-636e5568bf8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6d5903-vpcxv" Oct 30 16:51:24.475840 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:24.463025 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cc7c3ae-4732-442b-beb2-636e5568bf8c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6d5903-vpcxv\" (UID: \"3cc7c3ae-4732-442b-beb2-636e5568bf8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6d5903-vpcxv" Oct 30 16:51:24.476060 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:24.466322 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cc7c3ae-4732-442b-beb2-636e5568bf8c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-6d5903-vpcxv\" (UID: \"3cc7c3ae-4732-442b-beb2-636e5568bf8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6d5903-vpcxv" Oct 30 16:51:24.492148 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:24.492121 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlx7x\" (UniqueName: \"kubernetes.io/projected/3cc7c3ae-4732-442b-beb2-636e5568bf8c-kube-api-access-qlx7x\") pod \"maas-api-revoke-keys-e2e-ipp-b-6d5903-vpcxv\" (UID: \"3cc7c3ae-4732-442b-beb2-636e5568bf8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6d5903-vpcxv" Oct 30 16:51:24.496495 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:24.496261 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92963943-c2bqh"] Oct 30 16:51:24.501398 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:24.501377 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92963943-c2bqh" Oct 30 16:51:24.536895 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:24.530897 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92963943-c2bqh"] Oct 30 16:51:24.575816 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:24.566223 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6d5903-vpcxv" Oct 30 16:51:24.698917 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:24.698479 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a46c947e-b960-472d-a3b4-859cdd639b83-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92963943-c2bqh\" (UID: \"a46c947e-b960-472d-a3b4-859cdd639b83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92963943-c2bqh" Oct 30 16:51:24.698917 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:24.698545 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w8w6\" (UniqueName: \"kubernetes.io/projected/a46c947e-b960-472d-a3b4-859cdd639b83-kube-api-access-5w8w6\") pod \"maas-api-revoke-sub-92963943-c2bqh\" (UID: \"a46c947e-b960-472d-a3b4-859cdd639b83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92963943-c2bqh" Oct 30 16:51:24.698917 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:24.698572 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a46c947e-b960-472d-a3b4-859cdd639b83-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92963943-c2bqh\" (UID: \"a46c947e-b960-472d-a3b4-859cdd639b83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92963943-c2bqh" Oct 30 16:51:24.800803 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:24.800134 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a46c947e-b960-472d-a3b4-859cdd639b83-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92963943-c2bqh\" (UID: \"a46c947e-b960-472d-a3b4-859cdd639b83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92963943-c2bqh" Oct 30 16:51:24.800803 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:24.800265 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a46c947e-b960-472d-a3b4-859cdd639b83-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92963943-c2bqh\" (UID: \"a46c947e-b960-472d-a3b4-859cdd639b83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92963943-c2bqh" Oct 30 16:51:24.800803 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:24.800312 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w8w6\" (UniqueName: \"kubernetes.io/projected/a46c947e-b960-472d-a3b4-859cdd639b83-kube-api-access-5w8w6\") pod \"maas-api-revoke-sub-92963943-c2bqh\" (UID: \"a46c947e-b960-472d-a3b4-859cdd639b83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92963943-c2bqh" Oct 30 16:51:24.810808 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:24.801306 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a46c947e-b960-472d-a3b4-859cdd639b83-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92963943-c2bqh\" (UID: \"a46c947e-b960-472d-a3b4-859cdd639b83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92963943-c2bqh" Oct 30 16:51:24.810808 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:24.803351 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a46c947e-b960-472d-a3b4-859cdd639b83-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92963943-c2bqh\" (UID: \"a46c947e-b960-472d-a3b4-859cdd639b83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92963943-c2bqh" Oct 30 16:51:24.839818 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:24.837524 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w8w6\" (UniqueName: \"kubernetes.io/projected/a46c947e-b960-472d-a3b4-859cdd639b83-kube-api-access-5w8w6\") pod \"maas-api-revoke-sub-92963943-c2bqh\" (UID: \"a46c947e-b960-472d-a3b4-859cdd639b83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92963943-c2bqh" Oct 30 16:51:24.903973 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:24.903945 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92963943-c2bqh" Oct 30 16:51:24.960850 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:24.960814 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6d5903-vpcxv"] Oct 30 16:51:24.968899 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:51:24.968871 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cc7c3ae_4732_442b_beb2_636e5568bf8c.slice/crio-2871c80b322e018f9bd8040c5e0f7729e8bac294eb77eaf4419dfb57013e1c80 WatchSource:0}: Error finding container 2871c80b322e018f9bd8040c5e0f7729e8bac294eb77eaf4419dfb57013e1c80: Status 404 returned error can't find the container with id 2871c80b322e018f9bd8040c5e0f7729e8bac294eb77eaf4419dfb57013e1c80 Oct 30 16:51:25.214538 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:25.214496 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92963943-c2bqh"] Oct 30 16:51:25.218239 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:51:25.218207 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda46c947e_b960_472d_a3b4_859cdd639b83.slice/crio-e96abe23b33e13b054781a7546393cb1e424fc7527edb2578eea8a45bf06f66b WatchSource:0}: Error finding container e96abe23b33e13b054781a7546393cb1e424fc7527edb2578eea8a45bf06f66b: Status 404 returned error can't find the container with id e96abe23b33e13b054781a7546393cb1e424fc7527edb2578eea8a45bf06f66b Oct 30 16:51:25.222324 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:25.222298 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6d5903-vpcxv" event={"ID":"3cc7c3ae-4732-442b-beb2-636e5568bf8c","Type":"ContainerStarted","Data":"2871c80b322e018f9bd8040c5e0f7729e8bac294eb77eaf4419dfb57013e1c80"} Oct 30 16:51:25.374423 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:25.374400 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b744ec4f-nttt6" Oct 30 16:51:25.429809 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:25.423181 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de478b61-79a3-4aae-9bc8-cdda3391855d-kube-api-access-qrq74\" (UniqueName: \"kubernetes.io/projected/de478b61-79a3-4aae-9bc8-cdda3391855d-kube-api-access-qrq74\") pod \"de478b61-79a3-4aae-9bc8-cdda3391855d\" (UID: \"de478b61-79a3-4aae-9bc8-cdda3391855d\") " Oct 30 16:51:25.429809 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:25.423322 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de478b61-79a3-4aae-9bc8-cdda3391855d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de478b61-79a3-4aae-9bc8-cdda3391855d-maas-api-xks-ca\") pod \"de478b61-79a3-4aae-9bc8-cdda3391855d\" (UID: \"de478b61-79a3-4aae-9bc8-cdda3391855d\") " Oct 30 16:51:25.429809 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:25.423357 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de478b61-79a3-4aae-9bc8-cdda3391855d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de478b61-79a3-4aae-9bc8-cdda3391855d-maas-api-openshift-ca\") pod \"de478b61-79a3-4aae-9bc8-cdda3391855d\" (UID: \"de478b61-79a3-4aae-9bc8-cdda3391855d\") " Oct 30 16:51:25.429809 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:25.424303 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de478b61-79a3-4aae-9bc8-cdda3391855d-maas-api-openshift-ca" pod "de478b61-79a3-4aae-9bc8-cdda3391855d" (UID: "de478b61-79a3-4aae-9bc8-cdda3391855d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:51:25.429809 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:25.426273 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de478b61-79a3-4aae-9bc8-cdda3391855d-maas-api-xks-ca" pod "de478b61-79a3-4aae-9bc8-cdda3391855d" (UID: "de478b61-79a3-4aae-9bc8-cdda3391855d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:51:25.429809 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:25.426405 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de478b61-79a3-4aae-9bc8-cdda3391855d-kube-api-access-qrq74" pod "de478b61-79a3-4aae-9bc8-cdda3391855d" (UID: "de478b61-79a3-4aae-9bc8-cdda3391855d"). InnerVolumeSpecName "kube-api-access-qrq74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:51:25.525315 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:25.525284 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de478b61-79a3-4aae-9bc8-cdda3391855d-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:51:25.525457 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:25.525318 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de478b61-79a3-4aae-9bc8-cdda3391855d-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:51:25.525457 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:25.525336 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrq74\" (UniqueName: \"kubernetes.io/projected/de478b61-79a3-4aae-9bc8-cdda3391855d-kube-api-access-qrq74\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:51:26.243046 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:26.242964 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b744ec4f-nttt6" Oct 30 16:51:26.243621 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:26.242955 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b744ec4f-nttt6" event={"ID":"de478b61-79a3-4aae-9bc8-cdda3391855d","Type":"ContainerDied","Data":"b5917a82aa8c480ba903c0e5531d22d76b03e08f5214ba3468a4ae2c51fad83c"} Oct 30 16:51:26.243765 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:26.243687 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5917a82aa8c480ba903c0e5531d22d76b03e08f5214ba3468a4ae2c51fad83c" Oct 30 16:51:26.246290 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:26.246261 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92963943-c2bqh" event={"ID":"a46c947e-b960-472d-a3b4-859cdd639b83","Type":"ContainerStarted","Data":"e218ffb6a33b42ae4fa3f0251d39d8a776b86fa8ed6078b596cbc38f04d39b45"} Oct 30 16:51:26.246453 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:26.246316 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92963943-c2bqh" event={"ID":"a46c947e-b960-472d-a3b4-859cdd639b83","Type":"ContainerStarted","Data":"e96abe23b33e13b054781a7546393cb1e424fc7527edb2578eea8a45bf06f66b"} Oct 30 16:51:26.254923 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:26.254772 4493 generic.go:356] "Generic (PLEG): container finished" podID="3cc7c3ae-4732-442b-beb2-636e5568bf8c" containerID="b53e6e8bd98fb0f91fa6a2ffb1f61ccafd51cecf7efdc816c15aa65ee26139a4" exitCode=0 Oct 30 16:51:26.255243 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:26.254923 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6d5903-vpcxv" event={"ID":"3cc7c3ae-4732-442b-beb2-636e5568bf8c","Type":"ContainerDied","Data":"b53e6e8bd98fb0f91fa6a2ffb1f61ccafd51cecf7efdc816c15aa65ee26139a4"} Oct 30 16:51:26.283825 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:26.283743 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92963943-c2bqh" podStartSLOduration=2.283730194 podStartE2EDuration="2.283730194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:51:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:51:26.283179714 +0000 UTC m=+1713.510354827" watchObservedRunningTime="2026-10-30 16:51:26.283730194 +0000 UTC m=+1713.510905294" Oct 30 16:51:27.264093 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:27.264057 4493 generic.go:356] "Generic (PLEG): container finished" podID="a46c947e-b960-472d-a3b4-859cdd639b83" containerID="e218ffb6a33b42ae4fa3f0251d39d8a776b86fa8ed6078b596cbc38f04d39b45" exitCode=0 Oct 30 16:51:27.264540 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:27.264135 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92963943-c2bqh" event={"ID":"a46c947e-b960-472d-a3b4-859cdd639b83","Type":"ContainerDied","Data":"e218ffb6a33b42ae4fa3f0251d39d8a776b86fa8ed6078b596cbc38f04d39b45"} Oct 30 16:51:27.428187 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:27.428156 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6d5903-vpcxv" Oct 30 16:51:27.454994 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:27.454956 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3cc7c3ae-4732-442b-beb2-636e5568bf8c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cc7c3ae-4732-442b-beb2-636e5568bf8c-maas-api-xks-ca\") pod \"3cc7c3ae-4732-442b-beb2-636e5568bf8c\" (UID: \"3cc7c3ae-4732-442b-beb2-636e5568bf8c\") " Oct 30 16:51:27.455160 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:27.455146 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cc7c3ae-4732-442b-beb2-636e5568bf8c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cc7c3ae-4732-442b-beb2-636e5568bf8c-maas-api-openshift-ca\") pod \"3cc7c3ae-4732-442b-beb2-636e5568bf8c\" (UID: \"3cc7c3ae-4732-442b-beb2-636e5568bf8c\") " Oct 30 16:51:27.455217 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:27.455188 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cc7c3ae-4732-442b-beb2-636e5568bf8c-kube-api-access-qlx7x\" (UniqueName: \"kubernetes.io/projected/3cc7c3ae-4732-442b-beb2-636e5568bf8c-kube-api-access-qlx7x\") pod \"3cc7c3ae-4732-442b-beb2-636e5568bf8c\" (UID: \"3cc7c3ae-4732-442b-beb2-636e5568bf8c\") " Oct 30 16:51:27.455525 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:27.455501 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cc7c3ae-4732-442b-beb2-636e5568bf8c-maas-api-openshift-ca" pod "3cc7c3ae-4732-442b-beb2-636e5568bf8c" (UID: "3cc7c3ae-4732-442b-beb2-636e5568bf8c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:51:27.457244 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:27.457218 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cc7c3ae-4732-442b-beb2-636e5568bf8c-maas-api-xks-ca" pod "3cc7c3ae-4732-442b-beb2-636e5568bf8c" (UID: "3cc7c3ae-4732-442b-beb2-636e5568bf8c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:51:27.457341 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:27.457278 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cc7c3ae-4732-442b-beb2-636e5568bf8c-kube-api-access-qlx7x" pod "3cc7c3ae-4732-442b-beb2-636e5568bf8c" (UID: "3cc7c3ae-4732-442b-beb2-636e5568bf8c"). InnerVolumeSpecName "kube-api-access-qlx7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:51:27.556456 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:27.556376 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cc7c3ae-4732-442b-beb2-636e5568bf8c-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:51:27.556456 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:27.556404 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlx7x\" (UniqueName: \"kubernetes.io/projected/3cc7c3ae-4732-442b-beb2-636e5568bf8c-kube-api-access-qlx7x\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:51:27.556456 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:27.556418 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cc7c3ae-4732-442b-beb2-636e5568bf8c-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:51:28.275551 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:28.275513 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6d5903-vpcxv" Oct 30 16:51:28.276114 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:28.275507 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6d5903-vpcxv" event={"ID":"3cc7c3ae-4732-442b-beb2-636e5568bf8c","Type":"ContainerDied","Data":"2871c80b322e018f9bd8040c5e0f7729e8bac294eb77eaf4419dfb57013e1c80"} Oct 30 16:51:28.276114 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:28.275640 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2871c80b322e018f9bd8040c5e0f7729e8bac294eb77eaf4419dfb57013e1c80" Oct 30 16:51:28.424346 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:28.424320 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92963943-c2bqh" Oct 30 16:51:28.467609 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:28.467579 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a46c947e-b960-472d-a3b4-859cdd639b83-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a46c947e-b960-472d-a3b4-859cdd639b83-maas-api-openshift-ca\") pod \"a46c947e-b960-472d-a3b4-859cdd639b83\" (UID: \"a46c947e-b960-472d-a3b4-859cdd639b83\") " Oct 30 16:51:28.467751 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:28.467638 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a46c947e-b960-472d-a3b4-859cdd639b83-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a46c947e-b960-472d-a3b4-859cdd639b83-maas-api-xks-ca\") pod \"a46c947e-b960-472d-a3b4-859cdd639b83\" (UID: \"a46c947e-b960-472d-a3b4-859cdd639b83\") " Oct 30 16:51:28.467848 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:28.467812 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a46c947e-b960-472d-a3b4-859cdd639b83-kube-api-access-5w8w6\" (UniqueName: \"kubernetes.io/projected/a46c947e-b960-472d-a3b4-859cdd639b83-kube-api-access-5w8w6\") pod \"a46c947e-b960-472d-a3b4-859cdd639b83\" (UID: \"a46c947e-b960-472d-a3b4-859cdd639b83\") " Oct 30 16:51:28.467994 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:28.467967 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a46c947e-b960-472d-a3b4-859cdd639b83-maas-api-openshift-ca" pod "a46c947e-b960-472d-a3b4-859cdd639b83" (UID: "a46c947e-b960-472d-a3b4-859cdd639b83"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:51:28.468185 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:28.468162 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a46c947e-b960-472d-a3b4-859cdd639b83-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:51:28.469958 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:28.469931 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a46c947e-b960-472d-a3b4-859cdd639b83-maas-api-xks-ca" pod "a46c947e-b960-472d-a3b4-859cdd639b83" (UID: "a46c947e-b960-472d-a3b4-859cdd639b83"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:51:28.470264 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:28.470242 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a46c947e-b960-472d-a3b4-859cdd639b83-kube-api-access-5w8w6" pod "a46c947e-b960-472d-a3b4-859cdd639b83" (UID: "a46c947e-b960-472d-a3b4-859cdd639b83"). InnerVolumeSpecName "kube-api-access-5w8w6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:51:28.569062 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:28.568996 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5w8w6\" (UniqueName: \"kubernetes.io/projected/a46c947e-b960-472d-a3b4-859cdd639b83-kube-api-access-5w8w6\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:51:28.569062 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:28.569021 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a46c947e-b960-472d-a3b4-859cdd639b83-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:51:29.285249 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:29.285175 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92963943-c2bqh" Oct 30 16:51:29.285620 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:29.285180 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92963943-c2bqh" event={"ID":"a46c947e-b960-472d-a3b4-859cdd639b83","Type":"ContainerDied","Data":"e96abe23b33e13b054781a7546393cb1e424fc7527edb2578eea8a45bf06f66b"} Oct 30 16:51:29.285620 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:29.285301 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e96abe23b33e13b054781a7546393cb1e424fc7527edb2578eea8a45bf06f66b" Oct 30 16:51:29.419981 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:29.419950 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52d3111f-gm54m"] Oct 30 16:51:29.420705 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:29.420680 4493 cpu_manager.go:414] "containerMap: removing container" podUID="de478b61-79a3-4aae-9bc8-cdda3391855d" containerName="revoke-keys" Oct 30 16:51:29.420705 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:29.420701 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de478b61-79a3-4aae-9bc8-cdda3391855d" containerName="revoke-keys" Oct 30 16:51:29.420925 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:29.420743 4493 cpu_manager.go:414] "containerMap: removing container" podUID="a46c947e-b960-472d-a3b4-859cdd639b83" containerName="revoke-keys" Oct 30 16:51:29.420925 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:29.420751 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a46c947e-b960-472d-a3b4-859cdd639b83" containerName="revoke-keys" Oct 30 16:51:29.420925 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:29.420803 4493 cpu_manager.go:414] "containerMap: removing container" podUID="3cc7c3ae-4732-442b-beb2-636e5568bf8c" containerName="revoke-keys" Oct 30 16:51:29.420925 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:29.420811 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cc7c3ae-4732-442b-beb2-636e5568bf8c" containerName="revoke-keys" Oct 30 16:51:29.421135 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:29.420947 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="a46c947e-b960-472d-a3b4-859cdd639b83" containerName="revoke-keys" Oct 30 16:51:29.421135 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:29.420968 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cc7c3ae-4732-442b-beb2-636e5568bf8c" containerName="revoke-keys" Oct 30 16:51:29.421135 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:29.420981 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="de478b61-79a3-4aae-9bc8-cdda3391855d" containerName="revoke-keys" Oct 30 16:51:29.428679 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:29.425476 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d3111f-gm54m" Oct 30 16:51:29.444695 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:29.443037 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:51:29.444695 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:29.443113 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:51:29.444695 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:29.443244 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:51:29.452835 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:29.448434 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52d3111f-gm54m"] Oct 30 16:51:29.452946 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:29.452848 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 16:51:29.481173 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:29.481141 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b37484ec-3263-40c3-a268-e39297e58a81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52d3111f-gm54m\" (UID: \"b37484ec-3263-40c3-a268-e39297e58a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d3111f-gm54m" Oct 30 16:51:29.481417 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:29.481393 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n9kf\" (UniqueName: \"kubernetes.io/projected/b37484ec-3263-40c3-a268-e39297e58a81-kube-api-access-8n9kf\") pod \"maas-api-revoke-sub-52d3111f-gm54m\" (UID: \"b37484ec-3263-40c3-a268-e39297e58a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d3111f-gm54m" Oct 30 16:51:29.481484 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:29.481446 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b37484ec-3263-40c3-a268-e39297e58a81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52d3111f-gm54m\" (UID: \"b37484ec-3263-40c3-a268-e39297e58a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d3111f-gm54m" Oct 30 16:51:29.582508 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:29.582430 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n9kf\" (UniqueName: \"kubernetes.io/projected/b37484ec-3263-40c3-a268-e39297e58a81-kube-api-access-8n9kf\") pod \"maas-api-revoke-sub-52d3111f-gm54m\" (UID: \"b37484ec-3263-40c3-a268-e39297e58a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d3111f-gm54m" Oct 30 16:51:29.582508 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:29.582472 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b37484ec-3263-40c3-a268-e39297e58a81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52d3111f-gm54m\" (UID: \"b37484ec-3263-40c3-a268-e39297e58a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d3111f-gm54m" Oct 30 16:51:29.582687 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:29.582540 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b37484ec-3263-40c3-a268-e39297e58a81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52d3111f-gm54m\" (UID: \"b37484ec-3263-40c3-a268-e39297e58a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d3111f-gm54m" Oct 30 16:51:29.583145 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:29.583120 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b37484ec-3263-40c3-a268-e39297e58a81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52d3111f-gm54m\" (UID: \"b37484ec-3263-40c3-a268-e39297e58a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d3111f-gm54m" Oct 30 16:51:29.585051 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:29.585027 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b37484ec-3263-40c3-a268-e39297e58a81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52d3111f-gm54m\" (UID: \"b37484ec-3263-40c3-a268-e39297e58a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d3111f-gm54m" Oct 30 16:51:29.603458 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:29.603434 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n9kf\" (UniqueName: \"kubernetes.io/projected/b37484ec-3263-40c3-a268-e39297e58a81-kube-api-access-8n9kf\") pod \"maas-api-revoke-sub-52d3111f-gm54m\" (UID: \"b37484ec-3263-40c3-a268-e39297e58a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d3111f-gm54m" Oct 30 16:51:29.754622 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:29.754591 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d3111f-gm54m" Oct 30 16:51:30.084846 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:30.075887 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv"] Oct 30 16:51:30.089099 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:30.089062 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv"] Oct 30 16:51:30.089245 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:30.089187 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv" Oct 30 16:51:30.105634 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:30.105601 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ec886a6-5f5d-4057-a72e-b7886477edaa-istiod-ca-cert\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv\" (UID: \"8ec886a6-5f5d-4057-a72e-b7886477edaa\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv" Oct 30 16:51:30.105832 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:30.105683 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fk89\" (UniqueName: \"kubernetes.io/projected/8ec886a6-5f5d-4057-a72e-b7886477edaa-kube-api-access-5fk89\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv\" (UID: \"8ec886a6-5f5d-4057-a72e-b7886477edaa\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv" Oct 30 16:51:30.105832 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:30.105714 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ec886a6-5f5d-4057-a72e-b7886477edaa-istio-envoy\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv\" (UID: \"8ec886a6-5f5d-4057-a72e-b7886477edaa\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv" Oct 30 16:51:30.105832 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:30.105746 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ec886a6-5f5d-4057-a72e-b7886477edaa-istio-data\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv\" (UID: \"8ec886a6-5f5d-4057-a72e-b7886477edaa\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv" Oct 30 16:51:30.106039 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:30.105843 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ec886a6-5f5d-4057-a72e-b7886477edaa-workload-certs\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv\" (UID: \"8ec886a6-5f5d-4057-a72e-b7886477edaa\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv" Oct 30 16:51:30.106039 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:30.105888 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec886a6-5f5d-4057-a72e-b7886477edaa-workload-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv\" (UID: \"8ec886a6-5f5d-4057-a72e-b7886477edaa\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv" Oct 30 16:51:30.106039 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:30.105931 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ec886a6-5f5d-4057-a72e-b7886477edaa-istio-podinfo\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv\" (UID: \"8ec886a6-5f5d-4057-a72e-b7886477edaa\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv" Oct 30 16:51:30.106039 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:30.105988 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ec886a6-5f5d-4057-a72e-b7886477edaa-istio-token\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv\" (UID: \"8ec886a6-5f5d-4057-a72e-b7886477edaa\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv" Oct 30 16:51:30.106039 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:30.106018 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec886a6-5f5d-4057-a72e-b7886477edaa-credential-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv\" (UID: \"8ec886a6-5f5d-4057-a72e-b7886477edaa\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv" Oct 30 16:51:30.123034 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:30.122948 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52d3111f-gm54m"] Oct 30 16:51:30.126287 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:51:30.126261 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb37484ec_3263_40c3_a268_e39297e58a81.slice/crio-7015804e0721ab19d80f87edea04014e61331692eb8bd2a24553d54661b81cde WatchSource:0}: Error finding container 7015804e0721ab19d80f87edea04014e61331692eb8bd2a24553d54661b81cde: Status 404 returned error can't find the container with id 7015804e0721ab19d80f87edea04014e61331692eb8bd2a24553d54661b81cde Oct 30 16:51:30.206920 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:30.206879 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ec886a6-5f5d-4057-a72e-b7886477edaa-istio-podinfo\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv\" (UID: \"8ec886a6-5f5d-4057-a72e-b7886477edaa\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv" Oct 30 16:51:30.207099 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:30.206946 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ec886a6-5f5d-4057-a72e-b7886477edaa-istio-token\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv\" (UID: \"8ec886a6-5f5d-4057-a72e-b7886477edaa\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv" Oct 30 16:51:30.207168 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:30.207106 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec886a6-5f5d-4057-a72e-b7886477edaa-credential-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv\" (UID: \"8ec886a6-5f5d-4057-a72e-b7886477edaa\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv" Oct 30 16:51:30.207232 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:30.207196 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ec886a6-5f5d-4057-a72e-b7886477edaa-istiod-ca-cert\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv\" (UID: \"8ec886a6-5f5d-4057-a72e-b7886477edaa\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv" Oct 30 16:51:30.207284 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:30.207244 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fk89\" (UniqueName: \"kubernetes.io/projected/8ec886a6-5f5d-4057-a72e-b7886477edaa-kube-api-access-5fk89\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv\" (UID: \"8ec886a6-5f5d-4057-a72e-b7886477edaa\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv" Oct 30 16:51:30.207284 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:30.207261 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ec886a6-5f5d-4057-a72e-b7886477edaa-istio-envoy\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv\" (UID: \"8ec886a6-5f5d-4057-a72e-b7886477edaa\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv" Oct 30 16:51:30.207390 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:30.207284 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ec886a6-5f5d-4057-a72e-b7886477edaa-istio-data\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv\" (UID: \"8ec886a6-5f5d-4057-a72e-b7886477edaa\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv" Oct 30 16:51:30.207390 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:30.207350 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ec886a6-5f5d-4057-a72e-b7886477edaa-workload-certs\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv\" (UID: \"8ec886a6-5f5d-4057-a72e-b7886477edaa\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv" Oct 30 16:51:30.207483 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:30.207386 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec886a6-5f5d-4057-a72e-b7886477edaa-workload-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv\" (UID: \"8ec886a6-5f5d-4057-a72e-b7886477edaa\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv" Oct 30 16:51:30.207725 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:30.207704 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec886a6-5f5d-4057-a72e-b7886477edaa-workload-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv\" (UID: \"8ec886a6-5f5d-4057-a72e-b7886477edaa\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv" Oct 30 16:51:30.207843 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:30.207815 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ec886a6-5f5d-4057-a72e-b7886477edaa-istio-data\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv\" (UID: \"8ec886a6-5f5d-4057-a72e-b7886477edaa\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv" Oct 30 16:51:30.207972 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:30.207942 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ec886a6-5f5d-4057-a72e-b7886477edaa-workload-certs\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv\" (UID: \"8ec886a6-5f5d-4057-a72e-b7886477edaa\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv" Oct 30 16:51:30.208141 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:30.208122 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec886a6-5f5d-4057-a72e-b7886477edaa-credential-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv\" (UID: \"8ec886a6-5f5d-4057-a72e-b7886477edaa\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv" Oct 30 16:51:30.208703 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:30.208681 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ec886a6-5f5d-4057-a72e-b7886477edaa-istiod-ca-cert\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv\" (UID: \"8ec886a6-5f5d-4057-a72e-b7886477edaa\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv" Oct 30 16:51:30.210036 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:30.210008 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ec886a6-5f5d-4057-a72e-b7886477edaa-istio-podinfo\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv\" (UID: \"8ec886a6-5f5d-4057-a72e-b7886477edaa\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv" Oct 30 16:51:30.210205 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:30.210188 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ec886a6-5f5d-4057-a72e-b7886477edaa-istio-envoy\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv\" (UID: \"8ec886a6-5f5d-4057-a72e-b7886477edaa\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv" Oct 30 16:51:30.231385 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:30.231359 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fk89\" (UniqueName: \"kubernetes.io/projected/8ec886a6-5f5d-4057-a72e-b7886477edaa-kube-api-access-5fk89\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv\" (UID: \"8ec886a6-5f5d-4057-a72e-b7886477edaa\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv" Oct 30 16:51:30.234332 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:30.234303 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ec886a6-5f5d-4057-a72e-b7886477edaa-istio-token\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv\" (UID: \"8ec886a6-5f5d-4057-a72e-b7886477edaa\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv" Oct 30 16:51:30.294128 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:30.294094 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d3111f-gm54m" event={"ID":"b37484ec-3263-40c3-a268-e39297e58a81","Type":"ContainerStarted","Data":"7015804e0721ab19d80f87edea04014e61331692eb8bd2a24553d54661b81cde"} Oct 30 16:51:30.361552 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:51:30.361445 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 16:51:30.405080 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:30.405035 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv" Oct 30 16:51:30.736608 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:30.736581 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv"] Oct 30 16:51:30.736947 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:51:30.736928 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ec886a6_5f5d_4057_a72e_b7886477edaa.slice/crio-a37151411ea57e5e91817de0d62e06675831478f26df6e27be3a03809614c71e WatchSource:0}: Error finding container a37151411ea57e5e91817de0d62e06675831478f26df6e27be3a03809614c71e: Status 404 returned error can't find the container with id a37151411ea57e5e91817de0d62e06675831478f26df6e27be3a03809614c71e Oct 30 16:51:30.740004 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:30.739974 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:51:30.740064 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:30.740040 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:51:30.740097 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:30.740070 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:51:31.303940 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:31.303817 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv" event={"ID":"8ec886a6-5f5d-4057-a72e-b7886477edaa","Type":"ContainerStarted","Data":"cc518afd361399d7e1b7a3b7ed48b5c937e735941ca52c08db969755dcb1a7cb"} Oct 30 16:51:31.303940 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:31.303899 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv" event={"ID":"8ec886a6-5f5d-4057-a72e-b7886477edaa","Type":"ContainerStarted","Data":"a37151411ea57e5e91817de0d62e06675831478f26df6e27be3a03809614c71e"} Oct 30 16:51:31.306327 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:31.305979 4493 generic.go:356] "Generic (PLEG): container finished" podID="b37484ec-3263-40c3-a268-e39297e58a81" containerID="757cb325a3947328a8b8bbabcd6680aeec959e3f7c82b92b972cd25b8b687a39" exitCode=0 Oct 30 16:51:31.306327 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:31.306183 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d3111f-gm54m" event={"ID":"b37484ec-3263-40c3-a268-e39297e58a81","Type":"ContainerDied","Data":"757cb325a3947328a8b8bbabcd6680aeec959e3f7c82b92b972cd25b8b687a39"} Oct 30 16:51:31.368451 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:31.368390 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv" podStartSLOduration=2.368372393 podStartE2EDuration="2.368372393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:51:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:51:31.365038258 +0000 UTC m=+1718.592213358" watchObservedRunningTime="2026-10-30 16:51:31.368372393 +0000 UTC m=+1718.595547496" Oct 30 16:51:31.410806 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:31.406168 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv" Oct 30 16:51:31.412828 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:31.412293 4493 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Oct 30 16:51:31.412828 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:31.412354 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv" podUID="8ec886a6-5f5d-4057-a72e-b7886477edaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Oct 30 16:51:32.422808 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:32.408172 4493 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Oct 30 16:51:32.422808 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:32.408231 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv" podUID="8ec886a6-5f5d-4057-a72e-b7886477edaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Oct 30 16:51:32.540067 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:32.540036 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d3111f-gm54m" Oct 30 16:51:32.542499 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:32.542475 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b37484ec-3263-40c3-a268-e39297e58a81-kube-api-access-8n9kf\" (UniqueName: \"kubernetes.io/projected/b37484ec-3263-40c3-a268-e39297e58a81-kube-api-access-8n9kf\") pod \"b37484ec-3263-40c3-a268-e39297e58a81\" (UID: \"b37484ec-3263-40c3-a268-e39297e58a81\") " Oct 30 16:51:32.542649 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:32.542586 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b37484ec-3263-40c3-a268-e39297e58a81-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b37484ec-3263-40c3-a268-e39297e58a81-maas-api-openshift-ca\") pod \"b37484ec-3263-40c3-a268-e39297e58a81\" (UID: \"b37484ec-3263-40c3-a268-e39297e58a81\") " Oct 30 16:51:32.542649 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:32.542613 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b37484ec-3263-40c3-a268-e39297e58a81-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b37484ec-3263-40c3-a268-e39297e58a81-maas-api-xks-ca\") pod \"b37484ec-3263-40c3-a268-e39297e58a81\" (UID: \"b37484ec-3263-40c3-a268-e39297e58a81\") " Oct 30 16:51:32.543000 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:32.542974 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b37484ec-3263-40c3-a268-e39297e58a81-maas-api-openshift-ca" pod "b37484ec-3263-40c3-a268-e39297e58a81" (UID: "b37484ec-3263-40c3-a268-e39297e58a81"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:51:32.543117 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:32.543088 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b37484ec-3263-40c3-a268-e39297e58a81-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:51:32.545010 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:32.544981 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b37484ec-3263-40c3-a268-e39297e58a81-maas-api-xks-ca" pod "b37484ec-3263-40c3-a268-e39297e58a81" (UID: "b37484ec-3263-40c3-a268-e39297e58a81"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:51:32.545103 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:32.545033 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b37484ec-3263-40c3-a268-e39297e58a81-kube-api-access-8n9kf" pod "b37484ec-3263-40c3-a268-e39297e58a81" (UID: "b37484ec-3263-40c3-a268-e39297e58a81"). InnerVolumeSpecName "kube-api-access-8n9kf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:51:32.644392 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:32.644359 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8n9kf\" (UniqueName: \"kubernetes.io/projected/b37484ec-3263-40c3-a268-e39297e58a81-kube-api-access-8n9kf\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:51:32.644392 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:32.644392 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b37484ec-3263-40c3-a268-e39297e58a81-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:51:33.343196 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:33.343011 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d3111f-gm54m" Oct 30 16:51:33.343196 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:33.343011 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52d3111f-gm54m" event={"ID":"b37484ec-3263-40c3-a268-e39297e58a81","Type":"ContainerDied","Data":"7015804e0721ab19d80f87edea04014e61331692eb8bd2a24553d54661b81cde"} Oct 30 16:51:33.343196 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:33.343143 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7015804e0721ab19d80f87edea04014e61331692eb8bd2a24553d54661b81cde" Oct 30 16:51:33.408811 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:33.408128 4493 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Oct 30 16:51:33.408811 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:33.408185 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv" podUID="8ec886a6-5f5d-4057-a72e-b7886477edaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Oct 30 16:51:33.932805 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:33.932245 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-293848aa-rzk7x"] Oct 30 16:51:33.933385 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:33.933235 4493 cpu_manager.go:414] "containerMap: removing container" podUID="b37484ec-3263-40c3-a268-e39297e58a81" containerName="revoke-keys" Oct 30 16:51:33.933385 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:33.933269 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b37484ec-3263-40c3-a268-e39297e58a81" containerName="revoke-keys" Oct 30 16:51:33.933495 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:33.933467 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="b37484ec-3263-40c3-a268-e39297e58a81" containerName="revoke-keys" Oct 30 16:51:33.937629 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:33.937603 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-293848aa-rzk7x" Oct 30 16:51:33.942312 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:33.942286 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-293848aa-rzk7x"] Oct 30 16:51:33.952107 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:33.952085 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:51:33.952257 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:33.952101 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:51:33.963991 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:33.963969 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 16:51:33.963991 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:33.963984 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:51:33.968364 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:33.968329 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c384f2a6-168e-423e-9d81-bf21b618621b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-293848aa-rzk7x\" (UID: \"c384f2a6-168e-423e-9d81-bf21b618621b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-293848aa-rzk7x" Oct 30 16:51:33.968488 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:33.968368 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c384f2a6-168e-423e-9d81-bf21b618621b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-293848aa-rzk7x\" (UID: \"c384f2a6-168e-423e-9d81-bf21b618621b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-293848aa-rzk7x" Oct 30 16:51:33.968566 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:33.968480 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9s62\" (UniqueName: \"kubernetes.io/projected/c384f2a6-168e-423e-9d81-bf21b618621b-kube-api-access-g9s62\") pod \"maas-api-revoke-sub-293848aa-rzk7x\" (UID: \"c384f2a6-168e-423e-9d81-bf21b618621b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-293848aa-rzk7x" Oct 30 16:51:34.069645 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:34.069610 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c384f2a6-168e-423e-9d81-bf21b618621b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-293848aa-rzk7x\" (UID: \"c384f2a6-168e-423e-9d81-bf21b618621b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-293848aa-rzk7x" Oct 30 16:51:34.069888 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:34.069658 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c384f2a6-168e-423e-9d81-bf21b618621b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-293848aa-rzk7x\" (UID: \"c384f2a6-168e-423e-9d81-bf21b618621b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-293848aa-rzk7x" Oct 30 16:51:34.069888 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:34.069694 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9s62\" (UniqueName: \"kubernetes.io/projected/c384f2a6-168e-423e-9d81-bf21b618621b-kube-api-access-g9s62\") pod \"maas-api-revoke-sub-293848aa-rzk7x\" (UID: \"c384f2a6-168e-423e-9d81-bf21b618621b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-293848aa-rzk7x" Oct 30 16:51:34.070418 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:34.070393 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c384f2a6-168e-423e-9d81-bf21b618621b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-293848aa-rzk7x\" (UID: \"c384f2a6-168e-423e-9d81-bf21b618621b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-293848aa-rzk7x" Oct 30 16:51:34.072223 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:34.072201 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c384f2a6-168e-423e-9d81-bf21b618621b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-293848aa-rzk7x\" (UID: \"c384f2a6-168e-423e-9d81-bf21b618621b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-293848aa-rzk7x" Oct 30 16:51:34.092828 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:34.092776 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9s62\" (UniqueName: \"kubernetes.io/projected/c384f2a6-168e-423e-9d81-bf21b618621b-kube-api-access-g9s62\") pod \"maas-api-revoke-sub-293848aa-rzk7x\" (UID: \"c384f2a6-168e-423e-9d81-bf21b618621b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-293848aa-rzk7x" Oct 30 16:51:34.258225 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:34.258142 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-293848aa-rzk7x" Oct 30 16:51:34.426807 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:34.419151 4493 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Oct 30 16:51:34.426807 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:34.419212 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv" podUID="8ec886a6-5f5d-4057-a72e-b7886477edaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Oct 30 16:51:34.586554 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:34.586522 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-293848aa-rzk7x"] Oct 30 16:51:34.587088 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:51:34.587056 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc384f2a6_168e_423e_9d81_bf21b618621b.slice/crio-a3f4326dfc0dff56ea016bbb0bb694776f77c1ec81e5b09727af69cb4ce68886 WatchSource:0}: Error finding container a3f4326dfc0dff56ea016bbb0bb694776f77c1ec81e5b09727af69cb4ce68886: Status 404 returned error can't find the container with id a3f4326dfc0dff56ea016bbb0bb694776f77c1ec81e5b09727af69cb4ce68886 Oct 30 16:51:35.363865 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:35.363824 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-293848aa-rzk7x" event={"ID":"c384f2a6-168e-423e-9d81-bf21b618621b","Type":"ContainerStarted","Data":"b84f04e469761baad64b62b8e1097d43d06c06ae27efbf37666062c64de42078"} Oct 30 16:51:35.364371 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:35.363893 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-293848aa-rzk7x" event={"ID":"c384f2a6-168e-423e-9d81-bf21b618621b","Type":"ContainerStarted","Data":"a3f4326dfc0dff56ea016bbb0bb694776f77c1ec81e5b09727af69cb4ce68886"} Oct 30 16:51:35.409326 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:35.409294 4493 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Oct 30 16:51:35.410278 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:35.410248 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv" podUID="8ec886a6-5f5d-4057-a72e-b7886477edaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Oct 30 16:51:35.424887 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:35.423831 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-293848aa-rzk7x" podStartSLOduration=2.423775512 podStartE2EDuration="2.423775512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:51:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:51:35.409005888 +0000 UTC m=+1722.636180992" watchObservedRunningTime="2026-10-30 16:51:35.423775512 +0000 UTC m=+1722.650950616" Oct 30 16:51:36.372882 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:36.372836 4493 generic.go:356] "Generic (PLEG): container finished" podID="c384f2a6-168e-423e-9d81-bf21b618621b" containerID="b84f04e469761baad64b62b8e1097d43d06c06ae27efbf37666062c64de42078" exitCode=0 Oct 30 16:51:36.373374 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:36.372920 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-293848aa-rzk7x" event={"ID":"c384f2a6-168e-423e-9d81-bf21b618621b","Type":"ContainerDied","Data":"b84f04e469761baad64b62b8e1097d43d06c06ae27efbf37666062c64de42078"} Oct 30 16:51:36.408175 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:36.408141 4493 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Oct 30 16:51:36.408424 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:36.408397 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv" podUID="8ec886a6-5f5d-4057-a72e-b7886477edaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Oct 30 16:51:37.406397 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:37.406363 4493 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Oct 30 16:51:37.406889 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:37.406425 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv" podUID="8ec886a6-5f5d-4057-a72e-b7886477edaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Oct 30 16:51:37.551100 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:37.551069 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-293848aa-rzk7x" Oct 30 16:51:37.613203 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:37.613169 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c384f2a6-168e-423e-9d81-bf21b618621b-kube-api-access-g9s62\" (UniqueName: \"kubernetes.io/projected/c384f2a6-168e-423e-9d81-bf21b618621b-kube-api-access-g9s62\") pod \"c384f2a6-168e-423e-9d81-bf21b618621b\" (UID: \"c384f2a6-168e-423e-9d81-bf21b618621b\") " Oct 30 16:51:37.613386 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:37.613248 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c384f2a6-168e-423e-9d81-bf21b618621b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c384f2a6-168e-423e-9d81-bf21b618621b-maas-api-openshift-ca\") pod \"c384f2a6-168e-423e-9d81-bf21b618621b\" (UID: \"c384f2a6-168e-423e-9d81-bf21b618621b\") " Oct 30 16:51:37.613386 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:37.613283 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c384f2a6-168e-423e-9d81-bf21b618621b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c384f2a6-168e-423e-9d81-bf21b618621b-maas-api-xks-ca\") pod \"c384f2a6-168e-423e-9d81-bf21b618621b\" (UID: \"c384f2a6-168e-423e-9d81-bf21b618621b\") " Oct 30 16:51:37.613855 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:37.613824 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c384f2a6-168e-423e-9d81-bf21b618621b-maas-api-openshift-ca" pod "c384f2a6-168e-423e-9d81-bf21b618621b" (UID: "c384f2a6-168e-423e-9d81-bf21b618621b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:51:37.614018 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:37.613963 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c384f2a6-168e-423e-9d81-bf21b618621b-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:51:37.616433 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:37.616399 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c384f2a6-168e-423e-9d81-bf21b618621b-maas-api-xks-ca" pod "c384f2a6-168e-423e-9d81-bf21b618621b" (UID: "c384f2a6-168e-423e-9d81-bf21b618621b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:51:37.616542 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:37.616506 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c384f2a6-168e-423e-9d81-bf21b618621b-kube-api-access-g9s62" pod "c384f2a6-168e-423e-9d81-bf21b618621b" (UID: "c384f2a6-168e-423e-9d81-bf21b618621b"). InnerVolumeSpecName "kube-api-access-g9s62". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:51:37.715254 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:37.715217 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9s62\" (UniqueName: \"kubernetes.io/projected/c384f2a6-168e-423e-9d81-bf21b618621b-kube-api-access-g9s62\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:51:37.715254 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:37.715245 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c384f2a6-168e-423e-9d81-bf21b618621b-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:51:38.277836 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:38.277798 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0ebf400-2vzj4"] Oct 30 16:51:38.278424 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:38.278406 4493 cpu_manager.go:414] "containerMap: removing container" podUID="c384f2a6-168e-423e-9d81-bf21b618621b" containerName="revoke-keys" Oct 30 16:51:38.278542 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:38.278424 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c384f2a6-168e-423e-9d81-bf21b618621b" containerName="revoke-keys" Oct 30 16:51:38.278603 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:38.278591 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="c384f2a6-168e-423e-9d81-bf21b618621b" containerName="revoke-keys" Oct 30 16:51:38.282116 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:38.282094 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0ebf400-2vzj4" Oct 30 16:51:38.303993 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:38.303967 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0ebf400-2vzj4"] Oct 30 16:51:38.322237 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:38.322205 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91635317-235e-44ef-bd57-c71110a989c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0ebf400-2vzj4\" (UID: \"91635317-235e-44ef-bd57-c71110a989c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0ebf400-2vzj4" Oct 30 16:51:38.322569 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:38.322549 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91635317-235e-44ef-bd57-c71110a989c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0ebf400-2vzj4\" (UID: \"91635317-235e-44ef-bd57-c71110a989c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0ebf400-2vzj4" Oct 30 16:51:38.322752 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:38.322737 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2h7p\" (UniqueName: \"kubernetes.io/projected/91635317-235e-44ef-bd57-c71110a989c8-kube-api-access-s2h7p\") pod \"maas-api-revoke-sub-a0ebf400-2vzj4\" (UID: \"91635317-235e-44ef-bd57-c71110a989c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0ebf400-2vzj4" Oct 30 16:51:38.390411 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:38.390381 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-293848aa-rzk7x" Oct 30 16:51:38.390580 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:38.390415 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-293848aa-rzk7x" event={"ID":"c384f2a6-168e-423e-9d81-bf21b618621b","Type":"ContainerDied","Data":"a3f4326dfc0dff56ea016bbb0bb694776f77c1ec81e5b09727af69cb4ce68886"} Oct 30 16:51:38.390580 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:38.390465 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3f4326dfc0dff56ea016bbb0bb694776f77c1ec81e5b09727af69cb4ce68886" Oct 30 16:51:38.410389 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:38.410364 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv" Oct 30 16:51:38.411464 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:38.410908 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv" Oct 30 16:51:38.411592 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:38.411573 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv" Oct 30 16:51:38.424250 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:38.424222 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91635317-235e-44ef-bd57-c71110a989c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0ebf400-2vzj4\" (UID: \"91635317-235e-44ef-bd57-c71110a989c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0ebf400-2vzj4" Oct 30 16:51:38.424417 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:38.424333 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2h7p\" (UniqueName: \"kubernetes.io/projected/91635317-235e-44ef-bd57-c71110a989c8-kube-api-access-s2h7p\") pod \"maas-api-revoke-sub-a0ebf400-2vzj4\" (UID: \"91635317-235e-44ef-bd57-c71110a989c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0ebf400-2vzj4" Oct 30 16:51:38.424479 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:38.424437 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91635317-235e-44ef-bd57-c71110a989c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0ebf400-2vzj4\" (UID: \"91635317-235e-44ef-bd57-c71110a989c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0ebf400-2vzj4" Oct 30 16:51:38.425273 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:38.425247 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91635317-235e-44ef-bd57-c71110a989c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0ebf400-2vzj4\" (UID: \"91635317-235e-44ef-bd57-c71110a989c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0ebf400-2vzj4" Oct 30 16:51:38.427465 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:38.427423 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91635317-235e-44ef-bd57-c71110a989c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0ebf400-2vzj4\" (UID: \"91635317-235e-44ef-bd57-c71110a989c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0ebf400-2vzj4" Oct 30 16:51:38.448841 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:38.448821 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2h7p\" (UniqueName: \"kubernetes.io/projected/91635317-235e-44ef-bd57-c71110a989c8-kube-api-access-s2h7p\") pod \"maas-api-revoke-sub-a0ebf400-2vzj4\" (UID: \"91635317-235e-44ef-bd57-c71110a989c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0ebf400-2vzj4" Oct 30 16:51:38.598493 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:38.598425 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0ebf400-2vzj4" Oct 30 16:51:38.935099 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:51:38.935069 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91635317_235e_44ef_bd57_c71110a989c8.slice/crio-aa423ce453ddccf88342d210d008540ec5344dba2d5fa872e245b120b90a8669 WatchSource:0}: Error finding container aa423ce453ddccf88342d210d008540ec5344dba2d5fa872e245b120b90a8669: Status 404 returned error can't find the container with id aa423ce453ddccf88342d210d008540ec5344dba2d5fa872e245b120b90a8669 Oct 30 16:51:38.936420 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:38.936394 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0ebf400-2vzj4"] Oct 30 16:51:39.399486 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:39.399453 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0ebf400-2vzj4" event={"ID":"91635317-235e-44ef-bd57-c71110a989c8","Type":"ContainerStarted","Data":"3447ab92ffdcd82a6fb992492283fd884cd46f1fe1cecd4bdbb33dd3e251eb46"} Oct 30 16:51:39.399680 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:39.399502 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0ebf400-2vzj4" event={"ID":"91635317-235e-44ef-bd57-c71110a989c8","Type":"ContainerStarted","Data":"aa423ce453ddccf88342d210d008540ec5344dba2d5fa872e245b120b90a8669"} Oct 30 16:51:39.441473 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:39.441424 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0ebf400-2vzj4" podStartSLOduration=1.441410531 podStartE2EDuration="1.441410531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:51:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:51:39.438907781 +0000 UTC m=+1726.666082881" watchObservedRunningTime="2026-10-30 16:51:39.441410531 +0000 UTC m=+1726.668585632" Oct 30 16:51:39.773635 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:39.773605 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad31a642-xrxsz"] Oct 30 16:51:39.777633 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:39.777615 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad31a642-xrxsz" Oct 30 16:51:39.800617 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:39.800591 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad31a642-xrxsz"] Oct 30 16:51:39.842187 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:39.842155 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25h5b\" (UniqueName: \"kubernetes.io/projected/d7cca371-51e1-4544-a846-dc6fd448f6c0-kube-api-access-25h5b\") pod \"maas-api-revoke-sub-ad31a642-xrxsz\" (UID: \"d7cca371-51e1-4544-a846-dc6fd448f6c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad31a642-xrxsz" Oct 30 16:51:39.842339 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:39.842325 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7cca371-51e1-4544-a846-dc6fd448f6c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad31a642-xrxsz\" (UID: \"d7cca371-51e1-4544-a846-dc6fd448f6c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad31a642-xrxsz" Oct 30 16:51:39.842404 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:39.842363 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7cca371-51e1-4544-a846-dc6fd448f6c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad31a642-xrxsz\" (UID: \"d7cca371-51e1-4544-a846-dc6fd448f6c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad31a642-xrxsz" Oct 30 16:51:39.943584 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:39.943553 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7cca371-51e1-4544-a846-dc6fd448f6c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad31a642-xrxsz\" (UID: \"d7cca371-51e1-4544-a846-dc6fd448f6c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad31a642-xrxsz" Oct 30 16:51:39.943757 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:39.943597 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7cca371-51e1-4544-a846-dc6fd448f6c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad31a642-xrxsz\" (UID: \"d7cca371-51e1-4544-a846-dc6fd448f6c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad31a642-xrxsz" Oct 30 16:51:39.943837 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:39.943766 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25h5b\" (UniqueName: \"kubernetes.io/projected/d7cca371-51e1-4544-a846-dc6fd448f6c0-kube-api-access-25h5b\") pod \"maas-api-revoke-sub-ad31a642-xrxsz\" (UID: \"d7cca371-51e1-4544-a846-dc6fd448f6c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad31a642-xrxsz" Oct 30 16:51:39.944438 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:39.944405 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7cca371-51e1-4544-a846-dc6fd448f6c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad31a642-xrxsz\" (UID: \"d7cca371-51e1-4544-a846-dc6fd448f6c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad31a642-xrxsz" Oct 30 16:51:39.946062 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:39.946035 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7cca371-51e1-4544-a846-dc6fd448f6c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad31a642-xrxsz\" (UID: \"d7cca371-51e1-4544-a846-dc6fd448f6c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad31a642-xrxsz" Oct 30 16:51:39.966987 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:39.966963 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25h5b\" (UniqueName: \"kubernetes.io/projected/d7cca371-51e1-4544-a846-dc6fd448f6c0-kube-api-access-25h5b\") pod \"maas-api-revoke-sub-ad31a642-xrxsz\" (UID: \"d7cca371-51e1-4544-a846-dc6fd448f6c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad31a642-xrxsz" Oct 30 16:51:40.114862 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:40.114776 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad31a642-xrxsz" Oct 30 16:51:40.410905 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:40.410869 4493 generic.go:356] "Generic (PLEG): container finished" podID="91635317-235e-44ef-bd57-c71110a989c8" containerID="3447ab92ffdcd82a6fb992492283fd884cd46f1fe1cecd4bdbb33dd3e251eb46" exitCode=0 Oct 30 16:51:40.411065 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:40.410951 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0ebf400-2vzj4" event={"ID":"91635317-235e-44ef-bd57-c71110a989c8","Type":"ContainerDied","Data":"3447ab92ffdcd82a6fb992492283fd884cd46f1fe1cecd4bdbb33dd3e251eb46"} Oct 30 16:51:40.424361 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:51:40.424332 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7cca371_51e1_4544_a846_dc6fd448f6c0.slice/crio-78836526b6135bd53c8836d5ae60a48fa942028f8135e2c9281ba6085e034518 WatchSource:0}: Error finding container 78836526b6135bd53c8836d5ae60a48fa942028f8135e2c9281ba6085e034518: Status 404 returned error can't find the container with id 78836526b6135bd53c8836d5ae60a48fa942028f8135e2c9281ba6085e034518 Oct 30 16:51:40.425117 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:40.425088 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad31a642-xrxsz"] Oct 30 16:51:41.421852 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:41.421818 4493 generic.go:356] "Generic (PLEG): container finished" podID="d7cca371-51e1-4544-a846-dc6fd448f6c0" containerID="edab1a6f61467248a4b6bae760baf4db511f39601bf2fb4fcc9d4955adfebfe3" exitCode=0 Oct 30 16:51:41.422313 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:41.421910 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad31a642-xrxsz" event={"ID":"d7cca371-51e1-4544-a846-dc6fd448f6c0","Type":"ContainerDied","Data":"edab1a6f61467248a4b6bae760baf4db511f39601bf2fb4fcc9d4955adfebfe3"} Oct 30 16:51:41.422313 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:41.421960 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad31a642-xrxsz" event={"ID":"d7cca371-51e1-4544-a846-dc6fd448f6c0","Type":"ContainerStarted","Data":"78836526b6135bd53c8836d5ae60a48fa942028f8135e2c9281ba6085e034518"} Oct 30 16:51:41.581254 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:41.581195 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0ebf400-2vzj4" Oct 30 16:51:41.668274 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:41.668244 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91635317-235e-44ef-bd57-c71110a989c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91635317-235e-44ef-bd57-c71110a989c8-maas-api-openshift-ca\") pod \"91635317-235e-44ef-bd57-c71110a989c8\" (UID: \"91635317-235e-44ef-bd57-c71110a989c8\") " Oct 30 16:51:41.668435 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:41.668296 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91635317-235e-44ef-bd57-c71110a989c8-kube-api-access-s2h7p\" (UniqueName: \"kubernetes.io/projected/91635317-235e-44ef-bd57-c71110a989c8-kube-api-access-s2h7p\") pod \"91635317-235e-44ef-bd57-c71110a989c8\" (UID: \"91635317-235e-44ef-bd57-c71110a989c8\") " Oct 30 16:51:41.668435 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:41.668416 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91635317-235e-44ef-bd57-c71110a989c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91635317-235e-44ef-bd57-c71110a989c8-maas-api-xks-ca\") pod \"91635317-235e-44ef-bd57-c71110a989c8\" (UID: \"91635317-235e-44ef-bd57-c71110a989c8\") " Oct 30 16:51:41.668578 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:41.668560 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91635317-235e-44ef-bd57-c71110a989c8-maas-api-openshift-ca" pod "91635317-235e-44ef-bd57-c71110a989c8" (UID: "91635317-235e-44ef-bd57-c71110a989c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:51:41.669092 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:41.669004 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91635317-235e-44ef-bd57-c71110a989c8-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:51:41.670567 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:41.670548 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91635317-235e-44ef-bd57-c71110a989c8-maas-api-xks-ca" pod "91635317-235e-44ef-bd57-c71110a989c8" (UID: "91635317-235e-44ef-bd57-c71110a989c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:51:41.670719 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:41.670701 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91635317-235e-44ef-bd57-c71110a989c8-kube-api-access-s2h7p" pod "91635317-235e-44ef-bd57-c71110a989c8" (UID: "91635317-235e-44ef-bd57-c71110a989c8"). InnerVolumeSpecName "kube-api-access-s2h7p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:51:41.769863 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:41.769829 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2h7p\" (UniqueName: \"kubernetes.io/projected/91635317-235e-44ef-bd57-c71110a989c8-kube-api-access-s2h7p\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:51:41.769863 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:41.769859 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91635317-235e-44ef-bd57-c71110a989c8-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:51:42.446063 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:42.446028 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0ebf400-2vzj4" Oct 30 16:51:42.447539 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:42.447488 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0ebf400-2vzj4" event={"ID":"91635317-235e-44ef-bd57-c71110a989c8","Type":"ContainerDied","Data":"aa423ce453ddccf88342d210d008540ec5344dba2d5fa872e245b120b90a8669"} Oct 30 16:51:42.447705 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:42.447592 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa423ce453ddccf88342d210d008540ec5344dba2d5fa872e245b120b90a8669" Oct 30 16:51:42.616210 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:42.615896 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad31a642-xrxsz" Oct 30 16:51:42.681450 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:42.681427 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7cca371-51e1-4544-a846-dc6fd448f6c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7cca371-51e1-4544-a846-dc6fd448f6c0-maas-api-openshift-ca\") pod \"d7cca371-51e1-4544-a846-dc6fd448f6c0\" (UID: \"d7cca371-51e1-4544-a846-dc6fd448f6c0\") " Oct 30 16:51:42.681616 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:42.681594 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7cca371-51e1-4544-a846-dc6fd448f6c0-kube-api-access-25h5b\" (UniqueName: \"kubernetes.io/projected/d7cca371-51e1-4544-a846-dc6fd448f6c0-kube-api-access-25h5b\") pod \"d7cca371-51e1-4544-a846-dc6fd448f6c0\" (UID: \"d7cca371-51e1-4544-a846-dc6fd448f6c0\") " Oct 30 16:51:42.681907 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:42.681880 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7cca371-51e1-4544-a846-dc6fd448f6c0-maas-api-openshift-ca" pod "d7cca371-51e1-4544-a846-dc6fd448f6c0" (UID: "d7cca371-51e1-4544-a846-dc6fd448f6c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:51:42.682091 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:42.681666 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7cca371-51e1-4544-a846-dc6fd448f6c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7cca371-51e1-4544-a846-dc6fd448f6c0-maas-api-xks-ca\") pod \"d7cca371-51e1-4544-a846-dc6fd448f6c0\" (UID: \"d7cca371-51e1-4544-a846-dc6fd448f6c0\") " Oct 30 16:51:42.682577 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:42.682518 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7cca371-51e1-4544-a846-dc6fd448f6c0-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:51:42.683949 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:42.683926 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7cca371-51e1-4544-a846-dc6fd448f6c0-maas-api-xks-ca" pod "d7cca371-51e1-4544-a846-dc6fd448f6c0" (UID: "d7cca371-51e1-4544-a846-dc6fd448f6c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:51:42.684047 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:42.684009 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7cca371-51e1-4544-a846-dc6fd448f6c0-kube-api-access-25h5b" pod "d7cca371-51e1-4544-a846-dc6fd448f6c0" (UID: "d7cca371-51e1-4544-a846-dc6fd448f6c0"). InnerVolumeSpecName "kube-api-access-25h5b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:51:42.784297 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:42.784263 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7cca371-51e1-4544-a846-dc6fd448f6c0-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:51:42.784297 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:42.784291 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25h5b\" (UniqueName: \"kubernetes.io/projected/d7cca371-51e1-4544-a846-dc6fd448f6c0-kube-api-access-25h5b\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:51:43.266776 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:43.266681 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 16:51:43.456220 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:43.456183 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad31a642-xrxsz" Oct 30 16:51:43.456740 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:43.456238 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad31a642-xrxsz" event={"ID":"d7cca371-51e1-4544-a846-dc6fd448f6c0","Type":"ContainerDied","Data":"78836526b6135bd53c8836d5ae60a48fa942028f8135e2c9281ba6085e034518"} Oct 30 16:51:43.456740 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:43.456283 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78836526b6135bd53c8836d5ae60a48fa942028f8135e2c9281ba6085e034518" Oct 30 16:51:45.020367 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:45.020331 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw"] Oct 30 16:51:45.030667 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:45.030636 4493 cpu_manager.go:414] "containerMap: removing container" podUID="91635317-235e-44ef-bd57-c71110a989c8" containerName="revoke-keys" Oct 30 16:51:45.030898 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:45.030879 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91635317-235e-44ef-bd57-c71110a989c8" containerName="revoke-keys" Oct 30 16:51:45.031048 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:45.031034 4493 cpu_manager.go:414] "containerMap: removing container" podUID="d7cca371-51e1-4544-a846-dc6fd448f6c0" containerName="revoke-keys" Oct 30 16:51:45.031146 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:45.031135 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7cca371-51e1-4544-a846-dc6fd448f6c0" containerName="revoke-keys" Oct 30 16:51:45.031405 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:45.031392 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="91635317-235e-44ef-bd57-c71110a989c8" containerName="revoke-keys" Oct 30 16:51:45.031495 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:45.031486 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7cca371-51e1-4544-a846-dc6fd448f6c0" containerName="revoke-keys" Oct 30 16:51:45.045099 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:45.045078 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw" Oct 30 16:51:45.082931 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:45.082903 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw"] Oct 30 16:51:45.188540 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:45.188504 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92c1d749-8598-4a7b-b840-042533ac0bea-istio-token\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw\" (UID: \"92c1d749-8598-4a7b-b840-042533ac0bea\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw" Oct 30 16:51:45.188711 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:45.188572 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92c1d749-8598-4a7b-b840-042533ac0bea-istiod-ca-cert\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw\" (UID: \"92c1d749-8598-4a7b-b840-042533ac0bea\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw" Oct 30 16:51:45.188711 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:45.188662 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92c1d749-8598-4a7b-b840-042533ac0bea-workload-certs\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw\" (UID: \"92c1d749-8598-4a7b-b840-042533ac0bea\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw" Oct 30 16:51:45.188711 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:45.188684 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zdqz\" (UniqueName: \"kubernetes.io/projected/92c1d749-8598-4a7b-b840-042533ac0bea-kube-api-access-6zdqz\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw\" (UID: \"92c1d749-8598-4a7b-b840-042533ac0bea\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw" Oct 30 16:51:45.188954 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:45.188743 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92c1d749-8598-4a7b-b840-042533ac0bea-workload-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw\" (UID: \"92c1d749-8598-4a7b-b840-042533ac0bea\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw" Oct 30 16:51:45.188954 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:45.188804 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92c1d749-8598-4a7b-b840-042533ac0bea-istio-envoy\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw\" (UID: \"92c1d749-8598-4a7b-b840-042533ac0bea\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw" Oct 30 16:51:45.188954 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:45.188852 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92c1d749-8598-4a7b-b840-042533ac0bea-credential-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw\" (UID: \"92c1d749-8598-4a7b-b840-042533ac0bea\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw" Oct 30 16:51:45.188954 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:45.188912 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92c1d749-8598-4a7b-b840-042533ac0bea-istio-data\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw\" (UID: \"92c1d749-8598-4a7b-b840-042533ac0bea\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw" Oct 30 16:51:45.189167 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:45.188971 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92c1d749-8598-4a7b-b840-042533ac0bea-istio-podinfo\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw\" (UID: \"92c1d749-8598-4a7b-b840-042533ac0bea\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw" Oct 30 16:51:45.296615 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:45.296519 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92c1d749-8598-4a7b-b840-042533ac0bea-istio-data\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw\" (UID: \"92c1d749-8598-4a7b-b840-042533ac0bea\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw" Oct 30 16:51:45.303519 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:45.303491 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92c1d749-8598-4a7b-b840-042533ac0bea-istio-podinfo\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw\" (UID: \"92c1d749-8598-4a7b-b840-042533ac0bea\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw" Oct 30 16:51:45.312296 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:45.303363 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92c1d749-8598-4a7b-b840-042533ac0bea-istio-data\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw\" (UID: \"92c1d749-8598-4a7b-b840-042533ac0bea\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw" Oct 30 16:51:45.312942 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:45.312499 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92c1d749-8598-4a7b-b840-042533ac0bea-istio-token\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw\" (UID: \"92c1d749-8598-4a7b-b840-042533ac0bea\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw" Oct 30 16:51:45.318975 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:45.318954 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92c1d749-8598-4a7b-b840-042533ac0bea-istiod-ca-cert\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw\" (UID: \"92c1d749-8598-4a7b-b840-042533ac0bea\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw" Oct 30 16:51:45.319295 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:45.319274 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92c1d749-8598-4a7b-b840-042533ac0bea-workload-certs\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw\" (UID: \"92c1d749-8598-4a7b-b840-042533ac0bea\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw" Oct 30 16:51:45.328062 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:45.328032 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zdqz\" (UniqueName: \"kubernetes.io/projected/92c1d749-8598-4a7b-b840-042533ac0bea-kube-api-access-6zdqz\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw\" (UID: \"92c1d749-8598-4a7b-b840-042533ac0bea\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw" Oct 30 16:51:45.328656 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:45.327964 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92c1d749-8598-4a7b-b840-042533ac0bea-workload-certs\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw\" (UID: \"92c1d749-8598-4a7b-b840-042533ac0bea\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw" Oct 30 16:51:45.328891 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:45.328873 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92c1d749-8598-4a7b-b840-042533ac0bea-workload-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw\" (UID: \"92c1d749-8598-4a7b-b840-042533ac0bea\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw" Oct 30 16:51:45.340806 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:45.329056 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92c1d749-8598-4a7b-b840-042533ac0bea-istio-envoy\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw\" (UID: \"92c1d749-8598-4a7b-b840-042533ac0bea\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw" Oct 30 16:51:45.342097 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:45.342075 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92c1d749-8598-4a7b-b840-042533ac0bea-credential-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw\" (UID: \"92c1d749-8598-4a7b-b840-042533ac0bea\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw" Oct 30 16:51:45.342733 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:45.342545 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92c1d749-8598-4a7b-b840-042533ac0bea-credential-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw\" (UID: \"92c1d749-8598-4a7b-b840-042533ac0bea\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw" Oct 30 16:51:45.342897 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:45.330126 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92c1d749-8598-4a7b-b840-042533ac0bea-workload-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw\" (UID: \"92c1d749-8598-4a7b-b840-042533ac0bea\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw" Oct 30 16:51:45.361141 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:45.361118 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92c1d749-8598-4a7b-b840-042533ac0bea-istiod-ca-cert\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw\" (UID: \"92c1d749-8598-4a7b-b840-042533ac0bea\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw" Oct 30 16:51:45.423822 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:45.405251 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92c1d749-8598-4a7b-b840-042533ac0bea-istio-token\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw\" (UID: \"92c1d749-8598-4a7b-b840-042533ac0bea\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw" Oct 30 16:51:45.423822 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:45.417094 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92c1d749-8598-4a7b-b840-042533ac0bea-istio-envoy\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw\" (UID: \"92c1d749-8598-4a7b-b840-042533ac0bea\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw" Oct 30 16:51:45.451200 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:45.451172 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zdqz\" (UniqueName: \"kubernetes.io/projected/92c1d749-8598-4a7b-b840-042533ac0bea-kube-api-access-6zdqz\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw\" (UID: \"92c1d749-8598-4a7b-b840-042533ac0bea\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw" Oct 30 16:51:45.451405 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:45.451286 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92c1d749-8598-4a7b-b840-042533ac0bea-istio-podinfo\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw\" (UID: \"92c1d749-8598-4a7b-b840-042533ac0bea\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw" Oct 30 16:51:45.491963 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:45.491672 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw" Oct 30 16:51:45.914812 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:45.911378 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" podUID="b5abe703-fee0-4185-aa22-88d1db06392b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Oct 30 16:51:45.914812 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:45.911395 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" podUID="14805045-a727-41ba-8bd0-100d41f9f27d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:51:45.989818 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:45.988931 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" podUID="80b9bd9a-30d6-45ff-b318-26fcdc608e5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 16:51:46.468193 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:46.468165 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw"] Oct 30 16:51:46.470641 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:51:46.470591 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92c1d749_8598_4a7b_b840_042533ac0bea.slice/crio-73fd323d337921eb5709b87c19290b7539250f089bea62b493f48d37d677f12d WatchSource:0}: Error finding container 73fd323d337921eb5709b87c19290b7539250f089bea62b493f48d37d677f12d: Status 404 returned error can't find the container with id 73fd323d337921eb5709b87c19290b7539250f089bea62b493f48d37d677f12d Oct 30 16:51:46.473669 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:46.473626 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:51:46.473775 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:46.473719 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:51:46.473775 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:46.473765 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:51:46.521999 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:46.521962 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw" event={"ID":"92c1d749-8598-4a7b-b840-042533ac0bea","Type":"ContainerStarted","Data":"73fd323d337921eb5709b87c19290b7539250f089bea62b493f48d37d677f12d"} Oct 30 16:51:46.796040 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:46.795942 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6d5903-vpcxv"] Oct 30 16:51:46.806529 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:46.802565 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-6d5903-vpcxv"] Oct 30 16:51:46.935000 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:46.934963 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cc7c3ae-4732-442b-beb2-636e5568bf8c" path="/var/lib/kubelet/pods/3cc7c3ae-4732-442b-beb2-636e5568bf8c/volumes" Oct 30 16:51:47.548441 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:47.548408 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw" event={"ID":"92c1d749-8598-4a7b-b840-042533ac0bea","Type":"ContainerStarted","Data":"a3f3b65b8f0a1f539fbe0e280ce6cf5432a16a5f7e4d05b092031dc5a9e00201"} Oct 30 16:51:47.635807 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:47.635295 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw" podStartSLOduration=3.635274013 podStartE2EDuration="3.635274013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:51:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:51:47.609192922 +0000 UTC m=+1734.836368022" watchObservedRunningTime="2026-10-30 16:51:47.635274013 +0000 UTC m=+1734.862449114" Oct 30 16:51:48.492559 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:48.492521 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw" Oct 30 16:51:48.494089 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:48.494067 4493 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Oct 30 16:51:48.494156 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:48.494125 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw" podUID="92c1d749-8598-4a7b-b840-042533ac0bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Oct 30 16:51:49.492798 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:49.492754 4493 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Oct 30 16:51:49.493200 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:49.492820 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw" podUID="92c1d749-8598-4a7b-b840-042533ac0bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Oct 30 16:51:50.493142 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:50.493104 4493 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Oct 30 16:51:50.493663 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:50.493630 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw" podUID="92c1d749-8598-4a7b-b840-042533ac0bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Oct 30 16:51:51.492700 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:51.492668 4493 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Oct 30 16:51:51.492877 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:51.492720 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw" podUID="92c1d749-8598-4a7b-b840-042533ac0bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Oct 30 16:51:52.493041 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:52.493001 4493 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Oct 30 16:51:52.493543 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:52.493054 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw" podUID="92c1d749-8598-4a7b-b840-042533ac0bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Oct 30 16:51:52.639833 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:52.639798 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad1ec56f-l84rm"] Oct 30 16:51:52.644880 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:52.644858 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1ec56f-l84rm" Oct 30 16:51:52.671816 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:52.671773 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:51:52.672112 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:52.672094 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:51:52.672210 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:52.672183 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 16:51:52.677021 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:52.676999 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad1ec56f-l84rm"] Oct 30 16:51:52.677140 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:52.677125 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:51:52.699944 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:52.699913 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/522ee45b-15af-4129-afea-eb8b8f1c8537-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad1ec56f-l84rm\" (UID: \"522ee45b-15af-4129-afea-eb8b8f1c8537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1ec56f-l84rm" Oct 30 16:51:52.700128 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:52.700100 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn5lw\" (UniqueName: \"kubernetes.io/projected/522ee45b-15af-4129-afea-eb8b8f1c8537-kube-api-access-hn5lw\") pod \"maas-api-revoke-sub-ad1ec56f-l84rm\" (UID: \"522ee45b-15af-4129-afea-eb8b8f1c8537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1ec56f-l84rm" Oct 30 16:51:52.700256 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:52.700224 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/522ee45b-15af-4129-afea-eb8b8f1c8537-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad1ec56f-l84rm\" (UID: \"522ee45b-15af-4129-afea-eb8b8f1c8537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1ec56f-l84rm" Oct 30 16:51:52.801230 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:52.801156 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/522ee45b-15af-4129-afea-eb8b8f1c8537-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad1ec56f-l84rm\" (UID: \"522ee45b-15af-4129-afea-eb8b8f1c8537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1ec56f-l84rm" Oct 30 16:51:52.801375 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:52.801236 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/522ee45b-15af-4129-afea-eb8b8f1c8537-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad1ec56f-l84rm\" (UID: \"522ee45b-15af-4129-afea-eb8b8f1c8537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1ec56f-l84rm" Oct 30 16:51:52.801375 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:52.801310 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn5lw\" (UniqueName: \"kubernetes.io/projected/522ee45b-15af-4129-afea-eb8b8f1c8537-kube-api-access-hn5lw\") pod \"maas-api-revoke-sub-ad1ec56f-l84rm\" (UID: \"522ee45b-15af-4129-afea-eb8b8f1c8537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1ec56f-l84rm" Oct 30 16:51:52.801987 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:52.801968 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/522ee45b-15af-4129-afea-eb8b8f1c8537-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad1ec56f-l84rm\" (UID: \"522ee45b-15af-4129-afea-eb8b8f1c8537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1ec56f-l84rm" Oct 30 16:51:52.803729 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:52.803710 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/522ee45b-15af-4129-afea-eb8b8f1c8537-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad1ec56f-l84rm\" (UID: \"522ee45b-15af-4129-afea-eb8b8f1c8537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1ec56f-l84rm" Oct 30 16:51:52.824773 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:52.824745 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn5lw\" (UniqueName: \"kubernetes.io/projected/522ee45b-15af-4129-afea-eb8b8f1c8537-kube-api-access-hn5lw\") pod \"maas-api-revoke-sub-ad1ec56f-l84rm\" (UID: \"522ee45b-15af-4129-afea-eb8b8f1c8537\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1ec56f-l84rm" Oct 30 16:51:52.971494 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:52.971464 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 16:51:52.978863 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:52.978836 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1ec56f-l84rm" Oct 30 16:51:53.305867 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:53.305798 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad1ec56f-l84rm"] Oct 30 16:51:53.306145 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:51:53.306126 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod522ee45b_15af_4129_afea_eb8b8f1c8537.slice/crio-e853cfea796e2289e97f0ee6d91403db90eac1c9bcd1075d9a2609a2b1f6de9b WatchSource:0}: Error finding container e853cfea796e2289e97f0ee6d91403db90eac1c9bcd1075d9a2609a2b1f6de9b: Status 404 returned error can't find the container with id e853cfea796e2289e97f0ee6d91403db90eac1c9bcd1075d9a2609a2b1f6de9b Oct 30 16:51:53.502819 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:53.502740 4493 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Oct 30 16:51:53.503316 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:53.502819 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw" podUID="92c1d749-8598-4a7b-b840-042533ac0bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Oct 30 16:51:53.600519 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:53.600441 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1ec56f-l84rm" event={"ID":"522ee45b-15af-4129-afea-eb8b8f1c8537","Type":"ContainerStarted","Data":"e853cfea796e2289e97f0ee6d91403db90eac1c9bcd1075d9a2609a2b1f6de9b"} Oct 30 16:51:54.497686 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:54.497608 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw" Oct 30 16:51:54.498133 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:54.498106 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw" Oct 30 16:51:54.498779 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:54.498760 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw" Oct 30 16:51:54.584073 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:54.584046 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7"] Oct 30 16:51:54.587122 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:54.586767 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7" podUID="86b5095e-c77a-478b-b5ba-ff477800a066" containerName="istio-proxy" containerID="cri-o://24dfa31fcbb87f3507893c73b0dbeb29b060684ccb851ab123b0b14d7613edb0" gracePeriod=30 Oct 30 16:51:54.621763 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:54.621733 4493 generic.go:356] "Generic (PLEG): container finished" podID="522ee45b-15af-4129-afea-eb8b8f1c8537" containerID="9c10f2462a70774ed7e02c54b83598df03416975d05a4034159014ba2a8c0d6e" exitCode=0 Oct 30 16:51:54.623042 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:54.623017 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1ec56f-l84rm" event={"ID":"522ee45b-15af-4129-afea-eb8b8f1c8537","Type":"ContainerDied","Data":"9c10f2462a70774ed7e02c54b83598df03416975d05a4034159014ba2a8c0d6e"} Oct 30 16:51:55.796668 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:55.796643 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1ec56f-l84rm" Oct 30 16:51:55.805023 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:55.804996 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bec93343-dwl79"] Oct 30 16:51:55.805689 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:55.805669 4493 cpu_manager.go:414] "containerMap: removing container" podUID="522ee45b-15af-4129-afea-eb8b8f1c8537" containerName="revoke-keys" Oct 30 16:51:55.805689 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:55.805689 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="522ee45b-15af-4129-afea-eb8b8f1c8537" containerName="revoke-keys" Oct 30 16:51:55.805850 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:55.805795 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="522ee45b-15af-4129-afea-eb8b8f1c8537" containerName="revoke-keys" Oct 30 16:51:55.814069 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:55.814042 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bec93343-dwl79" Oct 30 16:51:55.832421 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:55.832370 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bec93343-dwl79"] Oct 30 16:51:55.847898 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:55.847868 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/522ee45b-15af-4129-afea-eb8b8f1c8537-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/522ee45b-15af-4129-afea-eb8b8f1c8537-maas-api-xks-ca\") pod \"522ee45b-15af-4129-afea-eb8b8f1c8537\" (UID: \"522ee45b-15af-4129-afea-eb8b8f1c8537\") " Oct 30 16:51:55.848030 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:55.848004 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/522ee45b-15af-4129-afea-eb8b8f1c8537-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/522ee45b-15af-4129-afea-eb8b8f1c8537-maas-api-openshift-ca\") pod \"522ee45b-15af-4129-afea-eb8b8f1c8537\" (UID: \"522ee45b-15af-4129-afea-eb8b8f1c8537\") " Oct 30 16:51:55.848121 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:55.848033 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/522ee45b-15af-4129-afea-eb8b8f1c8537-kube-api-access-hn5lw\" (UniqueName: \"kubernetes.io/projected/522ee45b-15af-4129-afea-eb8b8f1c8537-kube-api-access-hn5lw\") pod \"522ee45b-15af-4129-afea-eb8b8f1c8537\" (UID: \"522ee45b-15af-4129-afea-eb8b8f1c8537\") " Oct 30 16:51:55.849070 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:55.849040 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/522ee45b-15af-4129-afea-eb8b8f1c8537-maas-api-openshift-ca" pod "522ee45b-15af-4129-afea-eb8b8f1c8537" (UID: "522ee45b-15af-4129-afea-eb8b8f1c8537"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:51:55.851607 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:55.851581 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/522ee45b-15af-4129-afea-eb8b8f1c8537-maas-api-xks-ca" pod "522ee45b-15af-4129-afea-eb8b8f1c8537" (UID: "522ee45b-15af-4129-afea-eb8b8f1c8537"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:51:55.857738 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:55.857716 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/522ee45b-15af-4129-afea-eb8b8f1c8537-kube-api-access-hn5lw" pod "522ee45b-15af-4129-afea-eb8b8f1c8537" (UID: "522ee45b-15af-4129-afea-eb8b8f1c8537"). InnerVolumeSpecName "kube-api-access-hn5lw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:51:55.949681 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:55.949647 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59982bb0-0b3c-4bf0-ba41-248f56729a22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bec93343-dwl79\" (UID: \"59982bb0-0b3c-4bf0-ba41-248f56729a22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bec93343-dwl79" Oct 30 16:51:55.949919 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:55.949716 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59982bb0-0b3c-4bf0-ba41-248f56729a22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bec93343-dwl79\" (UID: \"59982bb0-0b3c-4bf0-ba41-248f56729a22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bec93343-dwl79" Oct 30 16:51:55.949919 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:55.949849 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpjmh\" (UniqueName: \"kubernetes.io/projected/59982bb0-0b3c-4bf0-ba41-248f56729a22-kube-api-access-jpjmh\") pod \"maas-api-revoke-sub-bec93343-dwl79\" (UID: \"59982bb0-0b3c-4bf0-ba41-248f56729a22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bec93343-dwl79" Oct 30 16:51:55.950052 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:55.950031 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/522ee45b-15af-4129-afea-eb8b8f1c8537-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:51:55.950086 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:55.950056 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hn5lw\" (UniqueName: \"kubernetes.io/projected/522ee45b-15af-4129-afea-eb8b8f1c8537-kube-api-access-hn5lw\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:51:55.950086 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:55.950067 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/522ee45b-15af-4129-afea-eb8b8f1c8537-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:51:56.051604 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:56.051567 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59982bb0-0b3c-4bf0-ba41-248f56729a22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bec93343-dwl79\" (UID: \"59982bb0-0b3c-4bf0-ba41-248f56729a22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bec93343-dwl79" Oct 30 16:51:56.051823 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:56.051654 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59982bb0-0b3c-4bf0-ba41-248f56729a22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bec93343-dwl79\" (UID: \"59982bb0-0b3c-4bf0-ba41-248f56729a22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bec93343-dwl79" Oct 30 16:51:56.051823 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:56.051691 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpjmh\" (UniqueName: \"kubernetes.io/projected/59982bb0-0b3c-4bf0-ba41-248f56729a22-kube-api-access-jpjmh\") pod \"maas-api-revoke-sub-bec93343-dwl79\" (UID: \"59982bb0-0b3c-4bf0-ba41-248f56729a22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bec93343-dwl79" Oct 30 16:51:56.052210 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:56.052189 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59982bb0-0b3c-4bf0-ba41-248f56729a22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bec93343-dwl79\" (UID: \"59982bb0-0b3c-4bf0-ba41-248f56729a22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bec93343-dwl79" Oct 30 16:51:56.054196 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:56.054166 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59982bb0-0b3c-4bf0-ba41-248f56729a22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bec93343-dwl79\" (UID: \"59982bb0-0b3c-4bf0-ba41-248f56729a22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bec93343-dwl79" Oct 30 16:51:56.074725 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:56.074694 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpjmh\" (UniqueName: \"kubernetes.io/projected/59982bb0-0b3c-4bf0-ba41-248f56729a22-kube-api-access-jpjmh\") pod \"maas-api-revoke-sub-bec93343-dwl79\" (UID: \"59982bb0-0b3c-4bf0-ba41-248f56729a22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bec93343-dwl79" Oct 30 16:51:56.148920 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:56.148893 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bec93343-dwl79" Oct 30 16:51:56.454768 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:51:56.454706 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59982bb0_0b3c_4bf0_ba41_248f56729a22.slice/crio-af9ac211c5ee573fb310982f69ee68f7fff30ffa5ef6c8a1a379a91e6c7e2cda WatchSource:0}: Error finding container af9ac211c5ee573fb310982f69ee68f7fff30ffa5ef6c8a1a379a91e6c7e2cda: Status 404 returned error can't find the container with id af9ac211c5ee573fb310982f69ee68f7fff30ffa5ef6c8a1a379a91e6c7e2cda Oct 30 16:51:56.456100 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:56.456073 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bec93343-dwl79"] Oct 30 16:51:56.640758 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:56.640716 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bec93343-dwl79" event={"ID":"59982bb0-0b3c-4bf0-ba41-248f56729a22","Type":"ContainerStarted","Data":"af9ac211c5ee573fb310982f69ee68f7fff30ffa5ef6c8a1a379a91e6c7e2cda"} Oct 30 16:51:56.642709 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:56.642682 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1ec56f-l84rm" event={"ID":"522ee45b-15af-4129-afea-eb8b8f1c8537","Type":"ContainerDied","Data":"e853cfea796e2289e97f0ee6d91403db90eac1c9bcd1075d9a2609a2b1f6de9b"} Oct 30 16:51:56.642959 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:56.642926 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e853cfea796e2289e97f0ee6d91403db90eac1c9bcd1075d9a2609a2b1f6de9b" Oct 30 16:51:56.642959 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:56.642699 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1ec56f-l84rm" Oct 30 16:51:57.654949 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:57.654912 4493 generic.go:356] "Generic (PLEG): container finished" podID="59982bb0-0b3c-4bf0-ba41-248f56729a22" containerID="2199581244ee161b1e01b4c37ff3cc5670d63a92b07af4c2109783355d3a386e" exitCode=0 Oct 30 16:51:57.655317 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:57.654974 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bec93343-dwl79" event={"ID":"59982bb0-0b3c-4bf0-ba41-248f56729a22","Type":"ContainerDied","Data":"2199581244ee161b1e01b4c37ff3cc5670d63a92b07af4c2109783355d3a386e"} Oct 30 16:51:58.842660 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:58.842629 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bec93343-dwl79" Oct 30 16:51:58.896883 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:58.896384 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59982bb0-0b3c-4bf0-ba41-248f56729a22-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59982bb0-0b3c-4bf0-ba41-248f56729a22-maas-api-xks-ca\") pod \"59982bb0-0b3c-4bf0-ba41-248f56729a22\" (UID: \"59982bb0-0b3c-4bf0-ba41-248f56729a22\") " Oct 30 16:51:58.896883 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:58.896445 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59982bb0-0b3c-4bf0-ba41-248f56729a22-kube-api-access-jpjmh\" (UniqueName: \"kubernetes.io/projected/59982bb0-0b3c-4bf0-ba41-248f56729a22-kube-api-access-jpjmh\") pod \"59982bb0-0b3c-4bf0-ba41-248f56729a22\" (UID: \"59982bb0-0b3c-4bf0-ba41-248f56729a22\") " Oct 30 16:51:58.897116 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:58.897056 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59982bb0-0b3c-4bf0-ba41-248f56729a22-maas-api-openshift-ca" pod "59982bb0-0b3c-4bf0-ba41-248f56729a22" (UID: "59982bb0-0b3c-4bf0-ba41-248f56729a22"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:51:58.897586 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:58.897560 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59982bb0-0b3c-4bf0-ba41-248f56729a22-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59982bb0-0b3c-4bf0-ba41-248f56729a22-maas-api-openshift-ca\") pod \"59982bb0-0b3c-4bf0-ba41-248f56729a22\" (UID: \"59982bb0-0b3c-4bf0-ba41-248f56729a22\") " Oct 30 16:51:58.898113 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:58.898092 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59982bb0-0b3c-4bf0-ba41-248f56729a22-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:51:58.899951 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:58.899922 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59982bb0-0b3c-4bf0-ba41-248f56729a22-kube-api-access-jpjmh" pod "59982bb0-0b3c-4bf0-ba41-248f56729a22" (UID: "59982bb0-0b3c-4bf0-ba41-248f56729a22"). InnerVolumeSpecName "kube-api-access-jpjmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:51:58.900361 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:58.900325 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59982bb0-0b3c-4bf0-ba41-248f56729a22-maas-api-xks-ca" pod "59982bb0-0b3c-4bf0-ba41-248f56729a22" (UID: "59982bb0-0b3c-4bf0-ba41-248f56729a22"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:51:58.999409 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:58.999380 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59982bb0-0b3c-4bf0-ba41-248f56729a22-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:51:58.999409 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:58.999406 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpjmh\" (UniqueName: \"kubernetes.io/projected/59982bb0-0b3c-4bf0-ba41-248f56729a22-kube-api-access-jpjmh\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:51:59.681275 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:59.681249 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bec93343-dwl79" Oct 30 16:51:59.681406 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:59.681282 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bec93343-dwl79" event={"ID":"59982bb0-0b3c-4bf0-ba41-248f56729a22","Type":"ContainerDied","Data":"af9ac211c5ee573fb310982f69ee68f7fff30ffa5ef6c8a1a379a91e6c7e2cda"} Oct 30 16:51:59.681406 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:59.681356 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af9ac211c5ee573fb310982f69ee68f7fff30ffa5ef6c8a1a379a91e6c7e2cda" Oct 30 16:51:59.861037 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:59.860985 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7" Oct 30 16:51:59.909671 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:59.909634 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86b5095e-c77a-478b-b5ba-ff477800a066-kube-api-access-pgvs8\" (UniqueName: \"kubernetes.io/projected/86b5095e-c77a-478b-b5ba-ff477800a066-kube-api-access-pgvs8\") pod \"86b5095e-c77a-478b-b5ba-ff477800a066\" (UID: \"86b5095e-c77a-478b-b5ba-ff477800a066\") " Oct 30 16:51:59.909942 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:59.909739 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/86b5095e-c77a-478b-b5ba-ff477800a066-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86b5095e-c77a-478b-b5ba-ff477800a066-istio-podinfo\") pod \"86b5095e-c77a-478b-b5ba-ff477800a066\" (UID: \"86b5095e-c77a-478b-b5ba-ff477800a066\") " Oct 30 16:51:59.909942 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:59.909771 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86b5095e-c77a-478b-b5ba-ff477800a066-istio-token\" (UniqueName: \"kubernetes.io/projected/86b5095e-c77a-478b-b5ba-ff477800a066-istio-token\") pod \"86b5095e-c77a-478b-b5ba-ff477800a066\" (UID: \"86b5095e-c77a-478b-b5ba-ff477800a066\") " Oct 30 16:51:59.909942 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:59.909905 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86b5095e-c77a-478b-b5ba-ff477800a066-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86b5095e-c77a-478b-b5ba-ff477800a066-workload-certs\") pod \"86b5095e-c77a-478b-b5ba-ff477800a066\" (UID: \"86b5095e-c77a-478b-b5ba-ff477800a066\") " Oct 30 16:51:59.910139 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:59.910026 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86b5095e-c77a-478b-b5ba-ff477800a066-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b5095e-c77a-478b-b5ba-ff477800a066-credential-socket\") pod \"86b5095e-c77a-478b-b5ba-ff477800a066\" (UID: \"86b5095e-c77a-478b-b5ba-ff477800a066\") " Oct 30 16:51:59.910139 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:59.910084 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86b5095e-c77a-478b-b5ba-ff477800a066-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86b5095e-c77a-478b-b5ba-ff477800a066-istio-data\") pod \"86b5095e-c77a-478b-b5ba-ff477800a066\" (UID: \"86b5095e-c77a-478b-b5ba-ff477800a066\") " Oct 30 16:51:59.910774 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:59.910269 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86b5095e-c77a-478b-b5ba-ff477800a066-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86b5095e-c77a-478b-b5ba-ff477800a066-istio-envoy\") pod \"86b5095e-c77a-478b-b5ba-ff477800a066\" (UID: \"86b5095e-c77a-478b-b5ba-ff477800a066\") " Oct 30 16:51:59.910774 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:59.910321 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86b5095e-c77a-478b-b5ba-ff477800a066-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b5095e-c77a-478b-b5ba-ff477800a066-workload-socket\") pod \"86b5095e-c77a-478b-b5ba-ff477800a066\" (UID: \"86b5095e-c77a-478b-b5ba-ff477800a066\") " Oct 30 16:51:59.910774 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:59.910424 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86b5095e-c77a-478b-b5ba-ff477800a066-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86b5095e-c77a-478b-b5ba-ff477800a066-istiod-ca-cert\") pod \"86b5095e-c77a-478b-b5ba-ff477800a066\" (UID: \"86b5095e-c77a-478b-b5ba-ff477800a066\") " Oct 30 16:51:59.910774 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:59.910460 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86b5095e-c77a-478b-b5ba-ff477800a066-workload-certs" pod "86b5095e-c77a-478b-b5ba-ff477800a066" (UID: "86b5095e-c77a-478b-b5ba-ff477800a066"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:59.911057 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:59.910954 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86b5095e-c77a-478b-b5ba-ff477800a066-credential-socket" pod "86b5095e-c77a-478b-b5ba-ff477800a066" (UID: "86b5095e-c77a-478b-b5ba-ff477800a066"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:59.911125 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:59.911086 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86b5095e-c77a-478b-b5ba-ff477800a066-istio-data" pod "86b5095e-c77a-478b-b5ba-ff477800a066" (UID: "86b5095e-c77a-478b-b5ba-ff477800a066"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:59.911393 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:59.911369 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86b5095e-c77a-478b-b5ba-ff477800a066-istiod-ca-cert" pod "86b5095e-c77a-478b-b5ba-ff477800a066" (UID: "86b5095e-c77a-478b-b5ba-ff477800a066"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:51:59.911474 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:59.911442 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86b5095e-c77a-478b-b5ba-ff477800a066-workload-certs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:51:59.911474 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:59.911463 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b5095e-c77a-478b-b5ba-ff477800a066-credential-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:51:59.911588 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:59.911479 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86b5095e-c77a-478b-b5ba-ff477800a066-istio-data\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:51:59.911588 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:59.911548 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86b5095e-c77a-478b-b5ba-ff477800a066-workload-socket" pod "86b5095e-c77a-478b-b5ba-ff477800a066" (UID: "86b5095e-c77a-478b-b5ba-ff477800a066"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:59.914038 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:59.914010 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86b5095e-c77a-478b-b5ba-ff477800a066-istio-token" pod "86b5095e-c77a-478b-b5ba-ff477800a066" (UID: "86b5095e-c77a-478b-b5ba-ff477800a066"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:51:59.915161 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:59.915127 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86b5095e-c77a-478b-b5ba-ff477800a066-istio-envoy" pod "86b5095e-c77a-478b-b5ba-ff477800a066" (UID: "86b5095e-c77a-478b-b5ba-ff477800a066"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:59.915261 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:59.915227 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86b5095e-c77a-478b-b5ba-ff477800a066-kube-api-access-pgvs8" pod "86b5095e-c77a-478b-b5ba-ff477800a066" (UID: "86b5095e-c77a-478b-b5ba-ff477800a066"). InnerVolumeSpecName "kube-api-access-pgvs8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:51:59.915466 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:51:59.915445 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/86b5095e-c77a-478b-b5ba-ff477800a066-istio-podinfo" pod "86b5095e-c77a-478b-b5ba-ff477800a066" (UID: "86b5095e-c77a-478b-b5ba-ff477800a066"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:52:00.013025 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:00.012985 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86b5095e-c77a-478b-b5ba-ff477800a066-istio-envoy\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:52:00.013025 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:00.013018 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86b5095e-c77a-478b-b5ba-ff477800a066-workload-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:52:00.013025 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:00.013031 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86b5095e-c77a-478b-b5ba-ff477800a066-istiod-ca-cert\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:52:00.013241 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:00.013046 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgvs8\" (UniqueName: \"kubernetes.io/projected/86b5095e-c77a-478b-b5ba-ff477800a066-kube-api-access-pgvs8\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:52:00.013241 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:00.013057 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86b5095e-c77a-478b-b5ba-ff477800a066-istio-podinfo\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:52:00.013241 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:00.013069 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86b5095e-c77a-478b-b5ba-ff477800a066-istio-token\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:52:00.344818 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:52:00.344726 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 16:52:00.690128 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:00.690091 4493 generic.go:356] "Generic (PLEG): container finished" podID="86b5095e-c77a-478b-b5ba-ff477800a066" containerID="24dfa31fcbb87f3507893c73b0dbeb29b060684ccb851ab123b0b14d7613edb0" exitCode=0 Oct 30 16:52:00.690312 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:00.690145 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7" Oct 30 16:52:00.690312 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:00.690170 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7" event={"ID":"86b5095e-c77a-478b-b5ba-ff477800a066","Type":"ContainerDied","Data":"24dfa31fcbb87f3507893c73b0dbeb29b060684ccb851ab123b0b14d7613edb0"} Oct 30 16:52:00.690312 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:00.690234 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7" event={"ID":"86b5095e-c77a-478b-b5ba-ff477800a066","Type":"ContainerDied","Data":"00c479a6c8ec2127d1b886cb6211355120edfdaa95fed21974a71774853175a7"} Oct 30 16:52:00.690312 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:00.690260 4493 scope.go:122] "RemoveContainer" containerID="24dfa31fcbb87f3507893c73b0dbeb29b060684ccb851ab123b0b14d7613edb0" Oct 30 16:52:00.702526 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:00.702508 4493 scope.go:122] "RemoveContainer" containerID="7621daac5dad9e6f7afc2c9eb3238930f657b8d9e1ff1d6cdf16adf54de856b0" Oct 30 16:52:00.720587 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:00.719542 4493 scope.go:122] "RemoveContainer" containerID="24dfa31fcbb87f3507893c73b0dbeb29b060684ccb851ab123b0b14d7613edb0" Oct 30 16:52:00.721419 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:52:00.721389 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24dfa31fcbb87f3507893c73b0dbeb29b060684ccb851ab123b0b14d7613edb0\": container with ID starting with 24dfa31fcbb87f3507893c73b0dbeb29b060684ccb851ab123b0b14d7613edb0 not found: ID does not exist" containerID="24dfa31fcbb87f3507893c73b0dbeb29b060684ccb851ab123b0b14d7613edb0" Oct 30 16:52:00.721541 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:00.721430 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24dfa31fcbb87f3507893c73b0dbeb29b060684ccb851ab123b0b14d7613edb0"} err="failed to get container status \"24dfa31fcbb87f3507893c73b0dbeb29b060684ccb851ab123b0b14d7613edb0\": rpc error: code = NotFound desc = could not find container \"24dfa31fcbb87f3507893c73b0dbeb29b060684ccb851ab123b0b14d7613edb0\": container with ID starting with 24dfa31fcbb87f3507893c73b0dbeb29b060684ccb851ab123b0b14d7613edb0 not found: ID does not exist" Oct 30 16:52:00.721541 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:00.721457 4493 scope.go:122] "RemoveContainer" containerID="7621daac5dad9e6f7afc2c9eb3238930f657b8d9e1ff1d6cdf16adf54de856b0" Oct 30 16:52:00.721882 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:52:00.721850 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7621daac5dad9e6f7afc2c9eb3238930f657b8d9e1ff1d6cdf16adf54de856b0\": container with ID starting with 7621daac5dad9e6f7afc2c9eb3238930f657b8d9e1ff1d6cdf16adf54de856b0 not found: ID does not exist" containerID="7621daac5dad9e6f7afc2c9eb3238930f657b8d9e1ff1d6cdf16adf54de856b0" Oct 30 16:52:00.721973 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:00.721923 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7621daac5dad9e6f7afc2c9eb3238930f657b8d9e1ff1d6cdf16adf54de856b0"} err="failed to get container status \"7621daac5dad9e6f7afc2c9eb3238930f657b8d9e1ff1d6cdf16adf54de856b0\": rpc error: code = NotFound desc = could not find container \"7621daac5dad9e6f7afc2c9eb3238930f657b8d9e1ff1d6cdf16adf54de856b0\": container with ID starting with 7621daac5dad9e6f7afc2c9eb3238930f657b8d9e1ff1d6cdf16adf54de856b0 not found: ID does not exist" Oct 30 16:52:00.755521 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:00.755493 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7"] Oct 30 16:52:00.767229 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:00.767205 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-6d5903-openshift-default-b7f74497f-j75t7"] Oct 30 16:52:00.934039 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:00.934005 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86b5095e-c77a-478b-b5ba-ff477800a066" path="/var/lib/kubelet/pods/86b5095e-c77a-478b-b5ba-ff477800a066/volumes" Oct 30 16:52:01.145704 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:01.145673 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt"] Oct 30 16:52:01.146473 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:01.146457 4493 cpu_manager.go:414] "containerMap: removing container" podUID="59982bb0-0b3c-4bf0-ba41-248f56729a22" containerName="revoke-keys" Oct 30 16:52:01.146555 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:01.146476 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59982bb0-0b3c-4bf0-ba41-248f56729a22" containerName="revoke-keys" Oct 30 16:52:01.146555 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:01.146511 4493 cpu_manager.go:414] "containerMap: removing container" podUID="86b5095e-c77a-478b-b5ba-ff477800a066" containerName="istio-proxy" Oct 30 16:52:01.146555 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:01.146519 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86b5095e-c77a-478b-b5ba-ff477800a066" containerName="istio-proxy" Oct 30 16:52:01.146555 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:01.146532 4493 cpu_manager.go:414] "containerMap: removing container" podUID="86b5095e-c77a-478b-b5ba-ff477800a066" containerName="istio-proxy" Oct 30 16:52:01.146555 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:01.146541 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86b5095e-c77a-478b-b5ba-ff477800a066" containerName="istio-proxy" Oct 30 16:52:01.146843 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:01.146663 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="59982bb0-0b3c-4bf0-ba41-248f56729a22" containerName="revoke-keys" Oct 30 16:52:01.146843 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:01.146678 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="86b5095e-c77a-478b-b5ba-ff477800a066" containerName="istio-proxy" Oct 30 16:52:01.146843 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:01.146688 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="86b5095e-c77a-478b-b5ba-ff477800a066" containerName="istio-proxy" Oct 30 16:52:01.152350 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:01.152330 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt" Oct 30 16:52:01.178980 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:01.178950 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt"] Oct 30 16:52:01.226468 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:01.226434 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c800b1d-54c3-4eae-911c-32979c3e23fa-workload-certs\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt\" (UID: \"4c800b1d-54c3-4eae-911c-32979c3e23fa\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt" Oct 30 16:52:01.226607 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:01.226497 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c800b1d-54c3-4eae-911c-32979c3e23fa-istio-podinfo\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt\" (UID: \"4c800b1d-54c3-4eae-911c-32979c3e23fa\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt" Oct 30 16:52:01.226607 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:01.226568 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c800b1d-54c3-4eae-911c-32979c3e23fa-istiod-ca-cert\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt\" (UID: \"4c800b1d-54c3-4eae-911c-32979c3e23fa\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt" Oct 30 16:52:01.226607 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:01.226596 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c800b1d-54c3-4eae-911c-32979c3e23fa-istio-token\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt\" (UID: \"4c800b1d-54c3-4eae-911c-32979c3e23fa\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt" Oct 30 16:52:01.226725 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:01.226618 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c800b1d-54c3-4eae-911c-32979c3e23fa-istio-data\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt\" (UID: \"4c800b1d-54c3-4eae-911c-32979c3e23fa\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt" Oct 30 16:52:01.226757 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:01.226735 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c800b1d-54c3-4eae-911c-32979c3e23fa-credential-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt\" (UID: \"4c800b1d-54c3-4eae-911c-32979c3e23fa\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt" Oct 30 16:52:01.226886 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:01.226809 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c800b1d-54c3-4eae-911c-32979c3e23fa-workload-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt\" (UID: \"4c800b1d-54c3-4eae-911c-32979c3e23fa\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt" Oct 30 16:52:01.226886 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:01.226881 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c800b1d-54c3-4eae-911c-32979c3e23fa-istio-envoy\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt\" (UID: \"4c800b1d-54c3-4eae-911c-32979c3e23fa\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt" Oct 30 16:52:01.226956 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:01.226927 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbj7v\" (UniqueName: \"kubernetes.io/projected/4c800b1d-54c3-4eae-911c-32979c3e23fa-kube-api-access-wbj7v\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt\" (UID: \"4c800b1d-54c3-4eae-911c-32979c3e23fa\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt" Oct 30 16:52:01.328357 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:01.328325 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c800b1d-54c3-4eae-911c-32979c3e23fa-workload-certs\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt\" (UID: \"4c800b1d-54c3-4eae-911c-32979c3e23fa\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt" Oct 30 16:52:01.328520 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:01.328388 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c800b1d-54c3-4eae-911c-32979c3e23fa-istio-podinfo\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt\" (UID: \"4c800b1d-54c3-4eae-911c-32979c3e23fa\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt" Oct 30 16:52:01.328520 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:01.328411 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c800b1d-54c3-4eae-911c-32979c3e23fa-istiod-ca-cert\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt\" (UID: \"4c800b1d-54c3-4eae-911c-32979c3e23fa\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt" Oct 30 16:52:01.328520 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:01.328434 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c800b1d-54c3-4eae-911c-32979c3e23fa-istio-token\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt\" (UID: \"4c800b1d-54c3-4eae-911c-32979c3e23fa\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt" Oct 30 16:52:01.328520 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:01.328452 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c800b1d-54c3-4eae-911c-32979c3e23fa-istio-data\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt\" (UID: \"4c800b1d-54c3-4eae-911c-32979c3e23fa\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt" Oct 30 16:52:01.328520 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:01.328489 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c800b1d-54c3-4eae-911c-32979c3e23fa-credential-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt\" (UID: \"4c800b1d-54c3-4eae-911c-32979c3e23fa\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt" Oct 30 16:52:01.328869 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:01.328529 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c800b1d-54c3-4eae-911c-32979c3e23fa-workload-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt\" (UID: \"4c800b1d-54c3-4eae-911c-32979c3e23fa\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt" Oct 30 16:52:01.328869 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:01.328602 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c800b1d-54c3-4eae-911c-32979c3e23fa-istio-envoy\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt\" (UID: \"4c800b1d-54c3-4eae-911c-32979c3e23fa\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt" Oct 30 16:52:01.328869 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:01.328667 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbj7v\" (UniqueName: \"kubernetes.io/projected/4c800b1d-54c3-4eae-911c-32979c3e23fa-kube-api-access-wbj7v\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt\" (UID: \"4c800b1d-54c3-4eae-911c-32979c3e23fa\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt" Oct 30 16:52:01.328869 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:01.328834 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c800b1d-54c3-4eae-911c-32979c3e23fa-workload-certs\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt\" (UID: \"4c800b1d-54c3-4eae-911c-32979c3e23fa\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt" Oct 30 16:52:01.329061 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:01.329001 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c800b1d-54c3-4eae-911c-32979c3e23fa-istio-data\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt\" (UID: \"4c800b1d-54c3-4eae-911c-32979c3e23fa\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt" Oct 30 16:52:01.329126 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:01.329066 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c800b1d-54c3-4eae-911c-32979c3e23fa-workload-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt\" (UID: \"4c800b1d-54c3-4eae-911c-32979c3e23fa\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt" Oct 30 16:52:01.329314 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:01.329288 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c800b1d-54c3-4eae-911c-32979c3e23fa-credential-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt\" (UID: \"4c800b1d-54c3-4eae-911c-32979c3e23fa\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt" Oct 30 16:52:01.329742 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:01.329608 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c800b1d-54c3-4eae-911c-32979c3e23fa-istiod-ca-cert\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt\" (UID: \"4c800b1d-54c3-4eae-911c-32979c3e23fa\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt" Oct 30 16:52:01.331551 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:01.331527 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c800b1d-54c3-4eae-911c-32979c3e23fa-istio-envoy\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt\" (UID: \"4c800b1d-54c3-4eae-911c-32979c3e23fa\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt" Oct 30 16:52:01.331651 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:01.331581 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c800b1d-54c3-4eae-911c-32979c3e23fa-istio-podinfo\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt\" (UID: \"4c800b1d-54c3-4eae-911c-32979c3e23fa\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt" Oct 30 16:52:01.352471 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:01.352443 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c800b1d-54c3-4eae-911c-32979c3e23fa-istio-token\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt\" (UID: \"4c800b1d-54c3-4eae-911c-32979c3e23fa\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt" Oct 30 16:52:01.352588 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:01.352567 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbj7v\" (UniqueName: \"kubernetes.io/projected/4c800b1d-54c3-4eae-911c-32979c3e23fa-kube-api-access-wbj7v\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt\" (UID: \"4c800b1d-54c3-4eae-911c-32979c3e23fa\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt" Oct 30 16:52:01.466341 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:01.466264 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt" Oct 30 16:52:01.795471 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:52:01.795444 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c800b1d_54c3_4eae_911c_32979c3e23fa.slice/crio-f57c4e75aad0073ff3e341f3246d935b5e9d84cf191a15157bb35f8c6e93b587 WatchSource:0}: Error finding container f57c4e75aad0073ff3e341f3246d935b5e9d84cf191a15157bb35f8c6e93b587: Status 404 returned error can't find the container with id f57c4e75aad0073ff3e341f3246d935b5e9d84cf191a15157bb35f8c6e93b587 Oct 30 16:52:01.795695 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:01.795676 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt"] Oct 30 16:52:01.797895 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:01.797857 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:52:01.797977 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:01.797928 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:52:01.797977 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:01.797957 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:52:02.722810 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:02.718561 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt" event={"ID":"4c800b1d-54c3-4eae-911c-32979c3e23fa","Type":"ContainerStarted","Data":"9b55b8eed833264014d170f82bdfa437f47ee513770f61f670e820830be924e2"} Oct 30 16:52:02.722810 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:02.718662 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt" event={"ID":"4c800b1d-54c3-4eae-911c-32979c3e23fa","Type":"ContainerStarted","Data":"f57c4e75aad0073ff3e341f3246d935b5e9d84cf191a15157bb35f8c6e93b587"} Oct 30 16:52:02.782084 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:02.782027 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt" podStartSLOduration=1.782012647 podStartE2EDuration="1.782012647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:52:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:52:02.780367301 +0000 UTC m=+1750.007542402" watchObservedRunningTime="2026-10-30 16:52:02.782012647 +0000 UTC m=+1750.009187747" Oct 30 16:52:02.803516 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:02.803483 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6035e9fa-rcvnx"] Oct 30 16:52:02.838803 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:02.838342 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6035e9fa-rcvnx" Oct 30 16:52:02.850802 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:02.847575 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6035e9fa-rcvnx"] Oct 30 16:52:02.854804 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:02.854757 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:52:02.855053 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:02.855034 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:52:02.884885 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:02.884855 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:52:02.884885 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:02.884875 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 16:52:02.893487 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:02.893450 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kpqt\" (UniqueName: \"kubernetes.io/projected/fb241152-8f3a-4d18-a038-7b1b663c62c3-kube-api-access-2kpqt\") pod \"maas-api-revoke-sub-6035e9fa-rcvnx\" (UID: \"fb241152-8f3a-4d18-a038-7b1b663c62c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6035e9fa-rcvnx" Oct 30 16:52:02.893647 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:02.893585 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb241152-8f3a-4d18-a038-7b1b663c62c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6035e9fa-rcvnx\" (UID: \"fb241152-8f3a-4d18-a038-7b1b663c62c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6035e9fa-rcvnx" Oct 30 16:52:02.893647 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:02.893617 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb241152-8f3a-4d18-a038-7b1b663c62c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6035e9fa-rcvnx\" (UID: \"fb241152-8f3a-4d18-a038-7b1b663c62c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6035e9fa-rcvnx" Oct 30 16:52:02.995254 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:02.995155 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kpqt\" (UniqueName: \"kubernetes.io/projected/fb241152-8f3a-4d18-a038-7b1b663c62c3-kube-api-access-2kpqt\") pod \"maas-api-revoke-sub-6035e9fa-rcvnx\" (UID: \"fb241152-8f3a-4d18-a038-7b1b663c62c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6035e9fa-rcvnx" Oct 30 16:52:02.995436 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:02.995286 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb241152-8f3a-4d18-a038-7b1b663c62c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6035e9fa-rcvnx\" (UID: \"fb241152-8f3a-4d18-a038-7b1b663c62c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6035e9fa-rcvnx" Oct 30 16:52:02.995436 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:02.995318 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb241152-8f3a-4d18-a038-7b1b663c62c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6035e9fa-rcvnx\" (UID: \"fb241152-8f3a-4d18-a038-7b1b663c62c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6035e9fa-rcvnx" Oct 30 16:52:02.995971 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:02.995944 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb241152-8f3a-4d18-a038-7b1b663c62c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6035e9fa-rcvnx\" (UID: \"fb241152-8f3a-4d18-a038-7b1b663c62c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6035e9fa-rcvnx" Oct 30 16:52:02.997814 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:02.997763 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb241152-8f3a-4d18-a038-7b1b663c62c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6035e9fa-rcvnx\" (UID: \"fb241152-8f3a-4d18-a038-7b1b663c62c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6035e9fa-rcvnx" Oct 30 16:52:03.057837 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:03.056538 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kpqt\" (UniqueName: \"kubernetes.io/projected/fb241152-8f3a-4d18-a038-7b1b663c62c3-kube-api-access-2kpqt\") pod \"maas-api-revoke-sub-6035e9fa-rcvnx\" (UID: \"fb241152-8f3a-4d18-a038-7b1b663c62c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6035e9fa-rcvnx" Oct 30 16:52:03.164633 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:03.164600 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6035e9fa-rcvnx" Oct 30 16:52:03.466539 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:03.466509 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt" Oct 30 16:52:03.468012 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:03.467991 4493 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Oct 30 16:52:03.468086 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:03.468048 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt" podUID="4c800b1d-54c3-4eae-911c-32979c3e23fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Oct 30 16:52:03.541899 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:03.541800 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6035e9fa-rcvnx"] Oct 30 16:52:03.546121 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:52:03.546079 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb241152_8f3a_4d18_a038_7b1b663c62c3.slice/crio-b8de6b5a31e37d55910e09acc21f05aa33351612cd1e9ebc0f03495232fd2416 WatchSource:0}: Error finding container b8de6b5a31e37d55910e09acc21f05aa33351612cd1e9ebc0f03495232fd2416: Status 404 returned error can't find the container with id b8de6b5a31e37d55910e09acc21f05aa33351612cd1e9ebc0f03495232fd2416 Oct 30 16:52:03.744975 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:03.744924 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6035e9fa-rcvnx" event={"ID":"fb241152-8f3a-4d18-a038-7b1b663c62c3","Type":"ContainerStarted","Data":"b8de6b5a31e37d55910e09acc21f05aa33351612cd1e9ebc0f03495232fd2416"} Oct 30 16:52:04.467103 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:04.467075 4493 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Oct 30 16:52:04.467266 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:04.467122 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt" podUID="4c800b1d-54c3-4eae-911c-32979c3e23fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Oct 30 16:52:04.758839 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:04.758776 4493 generic.go:356] "Generic (PLEG): container finished" podID="fb241152-8f3a-4d18-a038-7b1b663c62c3" containerID="8ea8e13c6746cdcfca00954449d7cd0e3a6fb9522cdea3d461c0feb2dbff7215" exitCode=0 Oct 30 16:52:04.759535 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:04.759509 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6035e9fa-rcvnx" event={"ID":"fb241152-8f3a-4d18-a038-7b1b663c62c3","Type":"ContainerDied","Data":"8ea8e13c6746cdcfca00954449d7cd0e3a6fb9522cdea3d461c0feb2dbff7215"} Oct 30 16:52:05.469346 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:05.469310 4493 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Oct 30 16:52:05.469774 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:05.469728 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt" podUID="4c800b1d-54c3-4eae-911c-32979c3e23fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Oct 30 16:52:06.051339 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:06.051315 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6035e9fa-rcvnx" Oct 30 16:52:06.212808 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:06.206720 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb241152-8f3a-4d18-a038-7b1b663c62c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb241152-8f3a-4d18-a038-7b1b663c62c3-maas-api-xks-ca\") pod \"fb241152-8f3a-4d18-a038-7b1b663c62c3\" (UID: \"fb241152-8f3a-4d18-a038-7b1b663c62c3\") " Oct 30 16:52:06.212808 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:06.206834 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb241152-8f3a-4d18-a038-7b1b663c62c3-kube-api-access-2kpqt\" (UniqueName: \"kubernetes.io/projected/fb241152-8f3a-4d18-a038-7b1b663c62c3-kube-api-access-2kpqt\") pod \"fb241152-8f3a-4d18-a038-7b1b663c62c3\" (UID: \"fb241152-8f3a-4d18-a038-7b1b663c62c3\") " Oct 30 16:52:06.212808 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:06.206921 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb241152-8f3a-4d18-a038-7b1b663c62c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb241152-8f3a-4d18-a038-7b1b663c62c3-maas-api-openshift-ca\") pod \"fb241152-8f3a-4d18-a038-7b1b663c62c3\" (UID: \"fb241152-8f3a-4d18-a038-7b1b663c62c3\") " Oct 30 16:52:06.240686 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:06.240650 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb241152-8f3a-4d18-a038-7b1b663c62c3-maas-api-xks-ca" pod "fb241152-8f3a-4d18-a038-7b1b663c62c3" (UID: "fb241152-8f3a-4d18-a038-7b1b663c62c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:52:06.362857 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:06.301969 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb241152-8f3a-4d18-a038-7b1b663c62c3-kube-api-access-2kpqt" pod "fb241152-8f3a-4d18-a038-7b1b663c62c3" (UID: "fb241152-8f3a-4d18-a038-7b1b663c62c3"). InnerVolumeSpecName "kube-api-access-2kpqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:52:06.362857 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:06.315508 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb241152-8f3a-4d18-a038-7b1b663c62c3-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:52:06.362857 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:06.315531 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kpqt\" (UniqueName: \"kubernetes.io/projected/fb241152-8f3a-4d18-a038-7b1b663c62c3-kube-api-access-2kpqt\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:52:06.362857 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:06.316627 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb241152-8f3a-4d18-a038-7b1b663c62c3-maas-api-openshift-ca" pod "fb241152-8f3a-4d18-a038-7b1b663c62c3" (UID: "fb241152-8f3a-4d18-a038-7b1b663c62c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:52:06.420810 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:06.418778 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb241152-8f3a-4d18-a038-7b1b663c62c3-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:52:06.467810 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:06.467007 4493 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Oct 30 16:52:06.467810 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:06.467077 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt" podUID="4c800b1d-54c3-4eae-911c-32979c3e23fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Oct 30 16:52:06.833806 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:06.827280 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6035e9fa-rcvnx" event={"ID":"fb241152-8f3a-4d18-a038-7b1b663c62c3","Type":"ContainerDied","Data":"b8de6b5a31e37d55910e09acc21f05aa33351612cd1e9ebc0f03495232fd2416"} Oct 30 16:52:06.846681 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:06.841766 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6035e9fa-rcvnx" Oct 30 16:52:06.850805 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:06.850724 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8de6b5a31e37d55910e09acc21f05aa33351612cd1e9ebc0f03495232fd2416" Oct 30 16:52:07.475083 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:07.467954 4493 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Oct 30 16:52:07.475083 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:07.468096 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt" podUID="4c800b1d-54c3-4eae-911c-32979c3e23fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Oct 30 16:52:07.916912 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:07.915776 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" podUID="14805045-a727-41ba-8bd0-100d41f9f27d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:07.926940 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:07.920628 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" podUID="e43738f3-d55f-4c0a-a364-bf2ac76605c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:08.472591 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:08.472162 4493 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Oct 30 16:52:08.472591 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:08.472221 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt" podUID="4c800b1d-54c3-4eae-911c-32979c3e23fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Oct 30 16:52:08.827052 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:08.826937 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" podUID="f3e1c8ff-fcc3-4a1f-9730-c617d57b5193" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:08.900825 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:08.897928 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" podUID="e90f43d3-f5eb-45ab-af1e-340d0ce18966" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:08.924534 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:08.923062 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" podUID="e43738f3-d55f-4c0a-a364-bf2ac76605c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:08.924534 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:08.923736 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" Oct 30 16:52:08.961823 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:08.949826 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" podUID="b5abe703-fee0-4185-aa22-88d1db06392b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:08.961823 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:08.949914 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" podUID="14805045-a727-41ba-8bd0-100d41f9f27d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 16:52:08.961823 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:08.950261 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" Oct 30 16:52:08.963010 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:08.962964 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" podUID="80b9bd9a-30d6-45ff-b318-26fcdc608e5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:09.199057 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:09.198921 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" podUID="08baad08-cd72-4305-b92d-c11a84ef276f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:09.225992 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:09.225947 4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" podUID="80b9bd9a-30d6-45ff-b318-26fcdc608e5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:09.467647 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:09.467562 4493 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:52:09.467933 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:09.467898 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv" podUID="8ec886a6-5f5d-4057-a72e-b7886477edaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:09.468114 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:09.468092 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" podUID="0eadc5d4-98c3-4bf2-b3c2-f73fe249182f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:09.493965 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:09.493934 4493 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Oct 30 16:52:09.494220 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:09.494195 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt" podUID="4c800b1d-54c3-4eae-911c-32979c3e23fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Oct 30 16:52:09.563876 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:09.549901 4493 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:52:09.563876 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:09.550005 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br" podUID="b30c2fa0-a3d3-4942-9596-bb6b245a238f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:09.768757 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:09.753331 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" podUID="e76172b1-659d-45bd-91c6-1a0ce4062335" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:09.827827 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:09.827750 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" podUID="f3e1c8ff-fcc3-4a1f-9730-c617d57b5193" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:09.835885 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:09.828924 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" Oct 30 16:52:09.900178 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:09.899375 4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" podUID="f3e1c8ff-fcc3-4a1f-9730-c617d57b5193" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:09.900178 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:09.899466 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" podUID="e90f43d3-f5eb-45ab-af1e-340d0ce18966" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Oct 30 16:52:09.900424 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:09.900291 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" Oct 30 16:52:09.937596 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:09.935118 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" podUID="e43738f3-d55f-4c0a-a364-bf2ac76605c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 16:52:09.977351 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:09.971183 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" podUID="14805045-a727-41ba-8bd0-100d41f9f27d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:09.977351 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:09.971483 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" podUID="80b9bd9a-30d6-45ff-b318-26fcdc608e5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:09.980927 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:09.980894 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" Oct 30 16:52:10.067899 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:10.067614 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" Oct 30 16:52:10.076814 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:10.076289 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" Oct 30 16:52:10.286234 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:10.286180 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" podUID="18b72648-f523-405e-b5cd-1266ff13f8f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:10.318887 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:10.318795 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" podUID="08baad08-cd72-4305-b92d-c11a84ef276f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:10.320006 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:10.319981 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" Oct 30 16:52:10.380726 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:10.380699 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" Oct 30 16:52:10.495232 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:10.495204 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" Oct 30 16:52:10.496470 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:10.496441 4493 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Oct 30 16:52:10.496674 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:10.496646 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt" podUID="4c800b1d-54c3-4eae-911c-32979c3e23fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Oct 30 16:52:10.555889 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:10.555685 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" Oct 30 16:52:10.568874 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:10.568022 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" Oct 30 16:52:11.285561 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:11.285523 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-280c83a4-wrzkk"] Oct 30 16:52:11.286183 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:11.286163 4493 cpu_manager.go:414] "containerMap: removing container" podUID="fb241152-8f3a-4d18-a038-7b1b663c62c3" containerName="revoke-keys" Oct 30 16:52:11.286183 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:11.286182 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb241152-8f3a-4d18-a038-7b1b663c62c3" containerName="revoke-keys" Oct 30 16:52:11.286323 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:11.286292 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb241152-8f3a-4d18-a038-7b1b663c62c3" containerName="revoke-keys" Oct 30 16:52:11.289690 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:11.289668 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-280c83a4-wrzkk" Oct 30 16:52:11.318260 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:11.318088 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:52:11.318260 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:11.318139 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:52:11.318260 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:11.318175 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 16:52:11.318260 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:11.318193 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:52:11.325168 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:11.325141 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-280c83a4-wrzkk"] Oct 30 16:52:11.412407 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:11.412366 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/668dacd9-5a6e-48df-ab0c-f777d9dbe3c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-280c83a4-wrzkk\" (UID: \"668dacd9-5a6e-48df-ab0c-f777d9dbe3c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-280c83a4-wrzkk" Oct 30 16:52:11.412613 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:11.412552 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st7kk\" (UniqueName: \"kubernetes.io/projected/668dacd9-5a6e-48df-ab0c-f777d9dbe3c7-kube-api-access-st7kk\") pod \"maas-api-revoke-sub-280c83a4-wrzkk\" (UID: \"668dacd9-5a6e-48df-ab0c-f777d9dbe3c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-280c83a4-wrzkk" Oct 30 16:52:11.412821 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:11.412764 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/668dacd9-5a6e-48df-ab0c-f777d9dbe3c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-280c83a4-wrzkk\" (UID: \"668dacd9-5a6e-48df-ab0c-f777d9dbe3c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-280c83a4-wrzkk" Oct 30 16:52:11.514310 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:11.514267 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st7kk\" (UniqueName: \"kubernetes.io/projected/668dacd9-5a6e-48df-ab0c-f777d9dbe3c7-kube-api-access-st7kk\") pod \"maas-api-revoke-sub-280c83a4-wrzkk\" (UID: \"668dacd9-5a6e-48df-ab0c-f777d9dbe3c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-280c83a4-wrzkk" Oct 30 16:52:11.514516 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:11.514486 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/668dacd9-5a6e-48df-ab0c-f777d9dbe3c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-280c83a4-wrzkk\" (UID: \"668dacd9-5a6e-48df-ab0c-f777d9dbe3c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-280c83a4-wrzkk" Oct 30 16:52:11.514596 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:11.514558 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/668dacd9-5a6e-48df-ab0c-f777d9dbe3c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-280c83a4-wrzkk\" (UID: \"668dacd9-5a6e-48df-ab0c-f777d9dbe3c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-280c83a4-wrzkk" Oct 30 16:52:11.515366 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:11.515332 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/668dacd9-5a6e-48df-ab0c-f777d9dbe3c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-280c83a4-wrzkk\" (UID: \"668dacd9-5a6e-48df-ab0c-f777d9dbe3c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-280c83a4-wrzkk" Oct 30 16:52:11.517866 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:11.517840 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/668dacd9-5a6e-48df-ab0c-f777d9dbe3c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-280c83a4-wrzkk\" (UID: \"668dacd9-5a6e-48df-ab0c-f777d9dbe3c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-280c83a4-wrzkk" Oct 30 16:52:11.543585 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:11.543521 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-st7kk\" (UniqueName: \"kubernetes.io/projected/668dacd9-5a6e-48df-ab0c-f777d9dbe3c7-kube-api-access-st7kk\") pod \"maas-api-revoke-sub-280c83a4-wrzkk\" (UID: \"668dacd9-5a6e-48df-ab0c-f777d9dbe3c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-280c83a4-wrzkk" Oct 30 16:52:11.604047 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:11.603957 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-280c83a4-wrzkk" Oct 30 16:52:11.937644 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:11.936633 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-280c83a4-wrzkk"] Oct 30 16:52:11.937929 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:52:11.937818 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod668dacd9_5a6e_48df_ab0c_f777d9dbe3c7.slice/crio-7d0904e792aa82c21047ff65fea148482cda94b91f2c87336ff7715a4fe6ffbb WatchSource:0}: Error finding container 7d0904e792aa82c21047ff65fea148482cda94b91f2c87336ff7715a4fe6ffbb: Status 404 returned error can't find the container with id 7d0904e792aa82c21047ff65fea148482cda94b91f2c87336ff7715a4fe6ffbb Oct 30 16:52:11.949739 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:11.949559 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt" Oct 30 16:52:11.950855 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:11.950767 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt" Oct 30 16:52:11.951608 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:11.951541 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt" Oct 30 16:52:12.470342 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:12.470292 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-280c83a4-wrzkk" event={"ID":"668dacd9-5a6e-48df-ab0c-f777d9dbe3c7","Type":"ContainerStarted","Data":"1bbc78bc0fbd07544e6f4249197814e29b45d18d629b2ce3e99c3c8b7bbfff9d"} Oct 30 16:52:12.470818 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:12.470362 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-280c83a4-wrzkk" event={"ID":"668dacd9-5a6e-48df-ab0c-f777d9dbe3c7","Type":"ContainerStarted","Data":"7d0904e792aa82c21047ff65fea148482cda94b91f2c87336ff7715a4fe6ffbb"} Oct 30 16:52:12.542777 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:12.542711 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-280c83a4-wrzkk" podStartSLOduration=1.542691354 podStartE2EDuration="1.542691354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:52:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:52:12.538534011 +0000 UTC m=+1759.765709114" watchObservedRunningTime="2026-10-30 16:52:12.542691354 +0000 UTC m=+1759.769866455" Oct 30 16:52:13.479886 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:13.479851 4493 generic.go:356] "Generic (PLEG): container finished" podID="668dacd9-5a6e-48df-ab0c-f777d9dbe3c7" containerID="1bbc78bc0fbd07544e6f4249197814e29b45d18d629b2ce3e99c3c8b7bbfff9d" exitCode=0 Oct 30 16:52:13.480304 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:13.479935 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-280c83a4-wrzkk" event={"ID":"668dacd9-5a6e-48df-ab0c-f777d9dbe3c7","Type":"ContainerDied","Data":"1bbc78bc0fbd07544e6f4249197814e29b45d18d629b2ce3e99c3c8b7bbfff9d"} Oct 30 16:52:14.647580 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:14.647557 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-280c83a4-wrzkk" Oct 30 16:52:14.676466 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:14.676432 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/668dacd9-5a6e-48df-ab0c-f777d9dbe3c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/668dacd9-5a6e-48df-ab0c-f777d9dbe3c7-maas-api-openshift-ca\") pod \"668dacd9-5a6e-48df-ab0c-f777d9dbe3c7\" (UID: \"668dacd9-5a6e-48df-ab0c-f777d9dbe3c7\") " Oct 30 16:52:14.676599 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:14.676482 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/668dacd9-5a6e-48df-ab0c-f777d9dbe3c7-kube-api-access-st7kk\" (UniqueName: \"kubernetes.io/projected/668dacd9-5a6e-48df-ab0c-f777d9dbe3c7-kube-api-access-st7kk\") pod \"668dacd9-5a6e-48df-ab0c-f777d9dbe3c7\" (UID: \"668dacd9-5a6e-48df-ab0c-f777d9dbe3c7\") " Oct 30 16:52:14.676599 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:14.676548 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/668dacd9-5a6e-48df-ab0c-f777d9dbe3c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/668dacd9-5a6e-48df-ab0c-f777d9dbe3c7-maas-api-xks-ca\") pod \"668dacd9-5a6e-48df-ab0c-f777d9dbe3c7\" (UID: \"668dacd9-5a6e-48df-ab0c-f777d9dbe3c7\") " Oct 30 16:52:14.676846 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:14.676818 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/668dacd9-5a6e-48df-ab0c-f777d9dbe3c7-maas-api-openshift-ca" pod "668dacd9-5a6e-48df-ab0c-f777d9dbe3c7" (UID: "668dacd9-5a6e-48df-ab0c-f777d9dbe3c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:52:14.677583 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:14.677550 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/668dacd9-5a6e-48df-ab0c-f777d9dbe3c7-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:52:14.679103 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:14.679077 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/668dacd9-5a6e-48df-ab0c-f777d9dbe3c7-kube-api-access-st7kk" pod "668dacd9-5a6e-48df-ab0c-f777d9dbe3c7" (UID: "668dacd9-5a6e-48df-ab0c-f777d9dbe3c7"). InnerVolumeSpecName "kube-api-access-st7kk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:52:14.679312 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:14.679289 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/668dacd9-5a6e-48df-ab0c-f777d9dbe3c7-maas-api-xks-ca" pod "668dacd9-5a6e-48df-ab0c-f777d9dbe3c7" (UID: "668dacd9-5a6e-48df-ab0c-f777d9dbe3c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:52:14.778669 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:14.778593 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-st7kk\" (UniqueName: \"kubernetes.io/projected/668dacd9-5a6e-48df-ab0c-f777d9dbe3c7-kube-api-access-st7kk\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:52:14.778669 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:14.778624 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/668dacd9-5a6e-48df-ab0c-f777d9dbe3c7-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:52:15.499582 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:15.499553 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-280c83a4-wrzkk" Oct 30 16:52:15.499816 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:15.499587 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-280c83a4-wrzkk" event={"ID":"668dacd9-5a6e-48df-ab0c-f777d9dbe3c7","Type":"ContainerDied","Data":"7d0904e792aa82c21047ff65fea148482cda94b91f2c87336ff7715a4fe6ffbb"} Oct 30 16:52:15.499816 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:15.499635 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d0904e792aa82c21047ff65fea148482cda94b91f2c87336ff7715a4fe6ffbb" Oct 30 16:52:23.021306 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:23.021270 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01e76a8b-slkgf"] Oct 30 16:52:23.022113 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:23.022093 4493 cpu_manager.go:414] "containerMap: removing container" podUID="668dacd9-5a6e-48df-ab0c-f777d9dbe3c7" containerName="revoke-keys" Oct 30 16:52:23.022204 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:23.022115 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="668dacd9-5a6e-48df-ab0c-f777d9dbe3c7" containerName="revoke-keys" Oct 30 16:52:23.022289 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:23.022276 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="668dacd9-5a6e-48df-ab0c-f777d9dbe3c7" containerName="revoke-keys" Oct 30 16:52:23.026215 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:23.026192 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01e76a8b-slkgf" Oct 30 16:52:23.062649 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:23.062618 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01e76a8b-slkgf"] Oct 30 16:52:23.068165 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:23.068088 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:52:23.068309 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:23.068169 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:52:23.068309 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:23.068180 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:52:23.068309 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:23.068265 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 16:52:23.084110 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:23.084083 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cb7aa16-13ef-4b08-a9ec-62be7dfb5c3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01e76a8b-slkgf\" (UID: \"7cb7aa16-13ef-4b08-a9ec-62be7dfb5c3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01e76a8b-slkgf" Oct 30 16:52:23.084252 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:23.084137 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cb7aa16-13ef-4b08-a9ec-62be7dfb5c3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01e76a8b-slkgf\" (UID: \"7cb7aa16-13ef-4b08-a9ec-62be7dfb5c3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01e76a8b-slkgf" Oct 30 16:52:23.084252 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:23.084164 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b69t9\" (UniqueName: \"kubernetes.io/projected/7cb7aa16-13ef-4b08-a9ec-62be7dfb5c3f-kube-api-access-b69t9\") pod \"maas-api-revoke-sub-01e76a8b-slkgf\" (UID: \"7cb7aa16-13ef-4b08-a9ec-62be7dfb5c3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01e76a8b-slkgf" Oct 30 16:52:23.184975 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:23.184949 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cb7aa16-13ef-4b08-a9ec-62be7dfb5c3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01e76a8b-slkgf\" (UID: \"7cb7aa16-13ef-4b08-a9ec-62be7dfb5c3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01e76a8b-slkgf" Oct 30 16:52:23.185116 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:23.184984 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cb7aa16-13ef-4b08-a9ec-62be7dfb5c3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01e76a8b-slkgf\" (UID: \"7cb7aa16-13ef-4b08-a9ec-62be7dfb5c3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01e76a8b-slkgf" Oct 30 16:52:23.185116 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:23.185004 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b69t9\" (UniqueName: \"kubernetes.io/projected/7cb7aa16-13ef-4b08-a9ec-62be7dfb5c3f-kube-api-access-b69t9\") pod \"maas-api-revoke-sub-01e76a8b-slkgf\" (UID: \"7cb7aa16-13ef-4b08-a9ec-62be7dfb5c3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01e76a8b-slkgf" Oct 30 16:52:23.185556 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:23.185535 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cb7aa16-13ef-4b08-a9ec-62be7dfb5c3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01e76a8b-slkgf\" (UID: \"7cb7aa16-13ef-4b08-a9ec-62be7dfb5c3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01e76a8b-slkgf" Oct 30 16:52:23.187475 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:23.187453 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cb7aa16-13ef-4b08-a9ec-62be7dfb5c3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01e76a8b-slkgf\" (UID: \"7cb7aa16-13ef-4b08-a9ec-62be7dfb5c3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01e76a8b-slkgf" Oct 30 16:52:23.208027 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:23.207997 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b69t9\" (UniqueName: \"kubernetes.io/projected/7cb7aa16-13ef-4b08-a9ec-62be7dfb5c3f-kube-api-access-b69t9\") pod \"maas-api-revoke-sub-01e76a8b-slkgf\" (UID: \"7cb7aa16-13ef-4b08-a9ec-62be7dfb5c3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01e76a8b-slkgf" Oct 30 16:52:23.346476 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:23.346388 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01e76a8b-slkgf" Oct 30 16:52:23.664249 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:52:23.664220 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cb7aa16_13ef_4b08_a9ec_62be7dfb5c3f.slice/crio-9a6b744f4877ff06b06c5350d6e9f3bbccfe52b962c80282320140b7dd204fbc WatchSource:0}: Error finding container 9a6b744f4877ff06b06c5350d6e9f3bbccfe52b962c80282320140b7dd204fbc: Status 404 returned error can't find the container with id 9a6b744f4877ff06b06c5350d6e9f3bbccfe52b962c80282320140b7dd204fbc Oct 30 16:52:23.664386 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:23.664233 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01e76a8b-slkgf"] Oct 30 16:52:24.580374 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:24.580340 4493 generic.go:356] "Generic (PLEG): container finished" podID="7cb7aa16-13ef-4b08-a9ec-62be7dfb5c3f" containerID="377f66ea46fca1cbf97d20ea487e29a6565e3ec4cf8129a6c04dc66fb9c90fe1" exitCode=0 Oct 30 16:52:24.580762 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:24.580426 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01e76a8b-slkgf" event={"ID":"7cb7aa16-13ef-4b08-a9ec-62be7dfb5c3f","Type":"ContainerDied","Data":"377f66ea46fca1cbf97d20ea487e29a6565e3ec4cf8129a6c04dc66fb9c90fe1"} Oct 30 16:52:24.580762 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:24.580476 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01e76a8b-slkgf" event={"ID":"7cb7aa16-13ef-4b08-a9ec-62be7dfb5c3f","Type":"ContainerStarted","Data":"9a6b744f4877ff06b06c5350d6e9f3bbccfe52b962c80282320140b7dd204fbc"} Oct 30 16:52:25.753889 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:25.753857 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01e76a8b-slkgf" Oct 30 16:52:25.827854 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:25.827825 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cb7aa16-13ef-4b08-a9ec-62be7dfb5c3f-kube-api-access-b69t9\" (UniqueName: \"kubernetes.io/projected/7cb7aa16-13ef-4b08-a9ec-62be7dfb5c3f-kube-api-access-b69t9\") pod \"7cb7aa16-13ef-4b08-a9ec-62be7dfb5c3f\" (UID: \"7cb7aa16-13ef-4b08-a9ec-62be7dfb5c3f\") " Oct 30 16:52:25.828050 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:25.827870 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cb7aa16-13ef-4b08-a9ec-62be7dfb5c3f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cb7aa16-13ef-4b08-a9ec-62be7dfb5c3f-maas-api-openshift-ca\") pod \"7cb7aa16-13ef-4b08-a9ec-62be7dfb5c3f\" (UID: \"7cb7aa16-13ef-4b08-a9ec-62be7dfb5c3f\") " Oct 30 16:52:25.828050 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:25.827968 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7cb7aa16-13ef-4b08-a9ec-62be7dfb5c3f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cb7aa16-13ef-4b08-a9ec-62be7dfb5c3f-maas-api-xks-ca\") pod \"7cb7aa16-13ef-4b08-a9ec-62be7dfb5c3f\" (UID: \"7cb7aa16-13ef-4b08-a9ec-62be7dfb5c3f\") " Oct 30 16:52:25.828356 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:25.828314 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cb7aa16-13ef-4b08-a9ec-62be7dfb5c3f-maas-api-openshift-ca" pod "7cb7aa16-13ef-4b08-a9ec-62be7dfb5c3f" (UID: "7cb7aa16-13ef-4b08-a9ec-62be7dfb5c3f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:52:25.828665 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:25.828633 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cb7aa16-13ef-4b08-a9ec-62be7dfb5c3f-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:52:25.830942 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:25.830919 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cb7aa16-13ef-4b08-a9ec-62be7dfb5c3f-maas-api-xks-ca" pod "7cb7aa16-13ef-4b08-a9ec-62be7dfb5c3f" (UID: "7cb7aa16-13ef-4b08-a9ec-62be7dfb5c3f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:52:25.831027 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:25.830941 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cb7aa16-13ef-4b08-a9ec-62be7dfb5c3f-kube-api-access-b69t9" pod "7cb7aa16-13ef-4b08-a9ec-62be7dfb5c3f" (UID: "7cb7aa16-13ef-4b08-a9ec-62be7dfb5c3f"). InnerVolumeSpecName "kube-api-access-b69t9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:52:25.930168 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:25.930133 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b69t9\" (UniqueName: \"kubernetes.io/projected/7cb7aa16-13ef-4b08-a9ec-62be7dfb5c3f-kube-api-access-b69t9\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:52:25.930168 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:25.930164 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cb7aa16-13ef-4b08-a9ec-62be7dfb5c3f-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:52:26.601409 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:26.601233 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01e76a8b-slkgf" event={"ID":"7cb7aa16-13ef-4b08-a9ec-62be7dfb5c3f","Type":"ContainerDied","Data":"9a6b744f4877ff06b06c5350d6e9f3bbccfe52b962c80282320140b7dd204fbc"} Oct 30 16:52:26.601625 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:26.601439 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a6b744f4877ff06b06c5350d6e9f3bbccfe52b962c80282320140b7dd204fbc" Oct 30 16:52:26.601625 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:26.601505 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01e76a8b-slkgf" Oct 30 16:52:26.785675 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:26.785638 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 16:52:28.050648 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:28.050607 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46c3c12a-4cd8v"] Oct 30 16:52:28.051983 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:28.051957 4493 cpu_manager.go:414] "containerMap: removing container" podUID="7cb7aa16-13ef-4b08-a9ec-62be7dfb5c3f" containerName="revoke-keys" Oct 30 16:52:28.052120 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:28.052107 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cb7aa16-13ef-4b08-a9ec-62be7dfb5c3f" containerName="revoke-keys" Oct 30 16:52:28.052417 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:28.052403 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cb7aa16-13ef-4b08-a9ec-62be7dfb5c3f" containerName="revoke-keys" Oct 30 16:52:28.056907 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:28.056885 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46c3c12a-4cd8v" Oct 30 16:52:28.073603 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:28.073576 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:52:28.093863 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:28.090725 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:52:28.093863 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:28.090876 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 16:52:28.093863 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:28.090927 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:52:28.112427 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:28.112399 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46c3c12a-4cd8v"] Oct 30 16:52:28.191055 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:28.191014 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bca5b866-90c9-484e-b043-e048c3ca2c23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46c3c12a-4cd8v\" (UID: \"bca5b866-90c9-484e-b043-e048c3ca2c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46c3c12a-4cd8v" Oct 30 16:52:28.191770 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:28.191311 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzq5v\" (UniqueName: \"kubernetes.io/projected/bca5b866-90c9-484e-b043-e048c3ca2c23-kube-api-access-hzq5v\") pod \"maas-api-revoke-sub-46c3c12a-4cd8v\" (UID: \"bca5b866-90c9-484e-b043-e048c3ca2c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46c3c12a-4cd8v" Oct 30 16:52:28.191770 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:28.191475 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bca5b866-90c9-484e-b043-e048c3ca2c23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46c3c12a-4cd8v\" (UID: \"bca5b866-90c9-484e-b043-e048c3ca2c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46c3c12a-4cd8v" Oct 30 16:52:28.292808 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:28.292517 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bca5b866-90c9-484e-b043-e048c3ca2c23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46c3c12a-4cd8v\" (UID: \"bca5b866-90c9-484e-b043-e048c3ca2c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46c3c12a-4cd8v" Oct 30 16:52:28.292808 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:28.292591 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzq5v\" (UniqueName: \"kubernetes.io/projected/bca5b866-90c9-484e-b043-e048c3ca2c23-kube-api-access-hzq5v\") pod \"maas-api-revoke-sub-46c3c12a-4cd8v\" (UID: \"bca5b866-90c9-484e-b043-e048c3ca2c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46c3c12a-4cd8v" Oct 30 16:52:28.292808 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:28.292722 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bca5b866-90c9-484e-b043-e048c3ca2c23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46c3c12a-4cd8v\" (UID: \"bca5b866-90c9-484e-b043-e048c3ca2c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46c3c12a-4cd8v" Oct 30 16:52:28.296804 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:28.293696 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bca5b866-90c9-484e-b043-e048c3ca2c23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46c3c12a-4cd8v\" (UID: \"bca5b866-90c9-484e-b043-e048c3ca2c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46c3c12a-4cd8v" Oct 30 16:52:28.308848 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:28.301263 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bca5b866-90c9-484e-b043-e048c3ca2c23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46c3c12a-4cd8v\" (UID: \"bca5b866-90c9-484e-b043-e048c3ca2c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46c3c12a-4cd8v" Oct 30 16:52:28.328654 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:28.328605 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzq5v\" (UniqueName: \"kubernetes.io/projected/bca5b866-90c9-484e-b043-e048c3ca2c23-kube-api-access-hzq5v\") pod \"maas-api-revoke-sub-46c3c12a-4cd8v\" (UID: \"bca5b866-90c9-484e-b043-e048c3ca2c23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46c3c12a-4cd8v" Oct 30 16:52:28.378328 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:28.378046 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46c3c12a-4cd8v" Oct 30 16:52:29.238148 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:29.228250 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46c3c12a-4cd8v"] Oct 30 16:52:29.616969 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:29.616926 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" podUID="80b9bd9a-30d6-45ff-b318-26fcdc608e5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:29.717810 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:29.717170 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" podUID="e76172b1-659d-45bd-91c6-1a0ce4062335" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:29.855810 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:29.851338 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b595920-sbs2r"] Oct 30 16:52:29.879808 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:29.873775 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b595920-sbs2r" Oct 30 16:52:29.944335 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:29.928822 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz855\" (UniqueName: \"kubernetes.io/projected/fff2bfc7-3c82-4a8f-a726-100912be8523-kube-api-access-jz855\") pod \"maas-api-revoke-sub-6b595920-sbs2r\" (UID: \"fff2bfc7-3c82-4a8f-a726-100912be8523\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b595920-sbs2r" Oct 30 16:52:29.944699 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:29.944669 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fff2bfc7-3c82-4a8f-a726-100912be8523-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b595920-sbs2r\" (UID: \"fff2bfc7-3c82-4a8f-a726-100912be8523\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b595920-sbs2r" Oct 30 16:52:30.059277 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:30.054301 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fff2bfc7-3c82-4a8f-a726-100912be8523-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b595920-sbs2r\" (UID: \"fff2bfc7-3c82-4a8f-a726-100912be8523\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b595920-sbs2r" Oct 30 16:52:30.059277 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:29.969472 4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" podUID="f3e1c8ff-fcc3-4a1f-9730-c617d57b5193" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/health\": context deadline exceeded" Oct 30 16:52:30.059277 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:29.969437 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" podUID="e43738f3-d55f-4c0a-a364-bf2ac76605c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:30.196809 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:30.165112 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fff2bfc7-3c82-4a8f-a726-100912be8523-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b595920-sbs2r\" (UID: \"fff2bfc7-3c82-4a8f-a726-100912be8523\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b595920-sbs2r" Oct 30 16:52:30.196809 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:30.165176 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fff2bfc7-3c82-4a8f-a726-100912be8523-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b595920-sbs2r\" (UID: \"fff2bfc7-3c82-4a8f-a726-100912be8523\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b595920-sbs2r" Oct 30 16:52:30.196809 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:30.165382 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz855\" (UniqueName: \"kubernetes.io/projected/fff2bfc7-3c82-4a8f-a726-100912be8523-kube-api-access-jz855\") pod \"maas-api-revoke-sub-6b595920-sbs2r\" (UID: \"fff2bfc7-3c82-4a8f-a726-100912be8523\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b595920-sbs2r" Oct 30 16:52:30.221818 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:30.219213 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fff2bfc7-3c82-4a8f-a726-100912be8523-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b595920-sbs2r\" (UID: \"fff2bfc7-3c82-4a8f-a726-100912be8523\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b595920-sbs2r" Oct 30 16:52:30.246819 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:30.236083 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz855\" (UniqueName: \"kubernetes.io/projected/fff2bfc7-3c82-4a8f-a726-100912be8523-kube-api-access-jz855\") pod \"maas-api-revoke-sub-6b595920-sbs2r\" (UID: \"fff2bfc7-3c82-4a8f-a726-100912be8523\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b595920-sbs2r" Oct 30 16:52:30.280814 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:30.264875 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" podUID="18b72648-f523-405e-b5cd-1266ff13f8f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:30.343837 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:30.343576 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fff2bfc7-3c82-4a8f-a726-100912be8523-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b595920-sbs2r\" (UID: \"fff2bfc7-3c82-4a8f-a726-100912be8523\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b595920-sbs2r" Oct 30 16:52:30.362865 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:30.352941 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b595920-sbs2r" Oct 30 16:52:30.392700 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:30.392533 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" podUID="08baad08-cd72-4305-b92d-c11a84ef276f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:30.438349 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:30.438311 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46c3c12a-4cd8v" event={"ID":"bca5b866-90c9-484e-b043-e048c3ca2c23","Type":"ContainerStarted","Data":"f144b4b78e6d9b509eb2c59152e5dff55c008cf44c4a4137fd1ca3a1d0f2e1e2"} Oct 30 16:52:30.438594 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:30.438577 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46c3c12a-4cd8v" event={"ID":"bca5b866-90c9-484e-b043-e048c3ca2c23","Type":"ContainerStarted","Data":"2f7d697bfe2765603af9cfdc3d9097882c4cb6e848cefaa7c13a855d1607be49"} Oct 30 16:52:30.543764 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:30.542977 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b595920-sbs2r"] Oct 30 16:52:30.617419 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:30.617312 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" podUID="80b9bd9a-30d6-45ff-b318-26fcdc608e5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 16:52:30.619393 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:30.618317 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" Oct 30 16:52:30.749383 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:52:30.747971 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 16:52:30.749383 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:30.748156 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" podUID="e76172b1-659d-45bd-91c6-1a0ce4062335" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:30.751712 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:30.749743 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" Oct 30 16:52:30.840599 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:30.840503 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46c3c12a-4cd8v" podStartSLOduration=2.840476661 podStartE2EDuration="2.840476661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:52:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:52:30.609964611 +0000 UTC m=+1777.837139711" watchObservedRunningTime="2026-10-30 16:52:30.840476661 +0000 UTC m=+1778.067651763" Oct 30 16:52:30.861015 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:30.860984 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp"] Oct 30 16:52:31.073419 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:31.073372 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" podUID="e43738f3-d55f-4c0a-a364-bf2ac76605c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:31.082056 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:31.080157 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" podUID="b5abe703-fee0-4185-aa22-88d1db06392b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Oct 30 16:52:31.093418 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:31.093349 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" podUID="e90f43d3-f5eb-45ab-af1e-340d0ce18966" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:31.158389 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:31.158351 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp"] Oct 30 16:52:31.158577 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:31.158515 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" Oct 30 16:52:31.159085 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:31.159061 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp" Oct 30 16:52:31.183501 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:31.183479 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" Oct 30 16:52:31.183688 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:31.183677 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" Oct 30 16:52:31.183811 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:31.183800 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b595920-sbs2r"] Oct 30 16:52:31.183931 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:31.183922 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" Oct 30 16:52:31.287992 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:31.287958 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c88b7e3-1f93-456a-87ed-4894a1de4b6c-istio-token\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp\" (UID: \"8c88b7e3-1f93-456a-87ed-4894a1de4b6c\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp" Oct 30 16:52:31.288392 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:31.288035 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c88b7e3-1f93-456a-87ed-4894a1de4b6c-istio-envoy\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp\" (UID: \"8c88b7e3-1f93-456a-87ed-4894a1de4b6c\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp" Oct 30 16:52:31.288392 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:31.288146 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c88b7e3-1f93-456a-87ed-4894a1de4b6c-workload-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp\" (UID: \"8c88b7e3-1f93-456a-87ed-4894a1de4b6c\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp" Oct 30 16:52:31.288392 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:31.288181 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c88b7e3-1f93-456a-87ed-4894a1de4b6c-workload-certs\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp\" (UID: \"8c88b7e3-1f93-456a-87ed-4894a1de4b6c\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp" Oct 30 16:52:31.288392 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:31.288287 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c88b7e3-1f93-456a-87ed-4894a1de4b6c-credential-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp\" (UID: \"8c88b7e3-1f93-456a-87ed-4894a1de4b6c\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp" Oct 30 16:52:31.288392 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:31.288354 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c88b7e3-1f93-456a-87ed-4894a1de4b6c-istiod-ca-cert\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp\" (UID: \"8c88b7e3-1f93-456a-87ed-4894a1de4b6c\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp" Oct 30 16:52:31.288642 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:31.288441 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctj54\" (UniqueName: \"kubernetes.io/projected/8c88b7e3-1f93-456a-87ed-4894a1de4b6c-kube-api-access-ctj54\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp\" (UID: \"8c88b7e3-1f93-456a-87ed-4894a1de4b6c\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp" Oct 30 16:52:31.288700 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:31.288657 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c88b7e3-1f93-456a-87ed-4894a1de4b6c-istio-podinfo\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp\" (UID: \"8c88b7e3-1f93-456a-87ed-4894a1de4b6c\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp" Oct 30 16:52:31.288846 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:31.288826 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c88b7e3-1f93-456a-87ed-4894a1de4b6c-istio-data\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp\" (UID: \"8c88b7e3-1f93-456a-87ed-4894a1de4b6c\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp" Oct 30 16:52:31.390245 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:31.390160 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c88b7e3-1f93-456a-87ed-4894a1de4b6c-istio-data\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp\" (UID: \"8c88b7e3-1f93-456a-87ed-4894a1de4b6c\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp" Oct 30 16:52:31.390397 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:31.390250 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c88b7e3-1f93-456a-87ed-4894a1de4b6c-istio-token\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp\" (UID: \"8c88b7e3-1f93-456a-87ed-4894a1de4b6c\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp" Oct 30 16:52:31.390397 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:31.390290 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c88b7e3-1f93-456a-87ed-4894a1de4b6c-istio-envoy\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp\" (UID: \"8c88b7e3-1f93-456a-87ed-4894a1de4b6c\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp" Oct 30 16:52:31.390397 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:31.390345 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c88b7e3-1f93-456a-87ed-4894a1de4b6c-workload-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp\" (UID: \"8c88b7e3-1f93-456a-87ed-4894a1de4b6c\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp" Oct 30 16:52:31.390397 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:31.390372 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c88b7e3-1f93-456a-87ed-4894a1de4b6c-workload-certs\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp\" (UID: \"8c88b7e3-1f93-456a-87ed-4894a1de4b6c\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp" Oct 30 16:52:31.390657 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:31.390405 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c88b7e3-1f93-456a-87ed-4894a1de4b6c-credential-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp\" (UID: \"8c88b7e3-1f93-456a-87ed-4894a1de4b6c\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp" Oct 30 16:52:31.390657 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:31.390454 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c88b7e3-1f93-456a-87ed-4894a1de4b6c-istiod-ca-cert\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp\" (UID: \"8c88b7e3-1f93-456a-87ed-4894a1de4b6c\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp" Oct 30 16:52:31.390657 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:31.390490 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctj54\" (UniqueName: \"kubernetes.io/projected/8c88b7e3-1f93-456a-87ed-4894a1de4b6c-kube-api-access-ctj54\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp\" (UID: \"8c88b7e3-1f93-456a-87ed-4894a1de4b6c\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp" Oct 30 16:52:31.390657 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:31.390567 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c88b7e3-1f93-456a-87ed-4894a1de4b6c-istio-podinfo\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp\" (UID: \"8c88b7e3-1f93-456a-87ed-4894a1de4b6c\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp" Oct 30 16:52:31.390657 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:31.390612 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c88b7e3-1f93-456a-87ed-4894a1de4b6c-istio-data\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp\" (UID: \"8c88b7e3-1f93-456a-87ed-4894a1de4b6c\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp" Oct 30 16:52:31.390975 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:31.390898 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c88b7e3-1f93-456a-87ed-4894a1de4b6c-credential-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp\" (UID: \"8c88b7e3-1f93-456a-87ed-4894a1de4b6c\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp" Oct 30 16:52:31.391156 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:31.391134 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c88b7e3-1f93-456a-87ed-4894a1de4b6c-workload-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp\" (UID: \"8c88b7e3-1f93-456a-87ed-4894a1de4b6c\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp" Oct 30 16:52:31.391470 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:31.391449 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c88b7e3-1f93-456a-87ed-4894a1de4b6c-istiod-ca-cert\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp\" (UID: \"8c88b7e3-1f93-456a-87ed-4894a1de4b6c\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp" Oct 30 16:52:31.391668 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:31.391651 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c88b7e3-1f93-456a-87ed-4894a1de4b6c-workload-certs\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp\" (UID: \"8c88b7e3-1f93-456a-87ed-4894a1de4b6c\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp" Oct 30 16:52:31.404385 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:31.404360 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c88b7e3-1f93-456a-87ed-4894a1de4b6c-istio-envoy\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp\" (UID: \"8c88b7e3-1f93-456a-87ed-4894a1de4b6c\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp" Oct 30 16:52:31.405130 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:31.405105 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c88b7e3-1f93-456a-87ed-4894a1de4b6c-istio-podinfo\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp\" (UID: \"8c88b7e3-1f93-456a-87ed-4894a1de4b6c\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp" Oct 30 16:52:31.413415 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:31.413383 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c88b7e3-1f93-456a-87ed-4894a1de4b6c-istio-token\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp\" (UID: \"8c88b7e3-1f93-456a-87ed-4894a1de4b6c\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp" Oct 30 16:52:31.413750 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:31.413728 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctj54\" (UniqueName: \"kubernetes.io/projected/8c88b7e3-1f93-456a-87ed-4894a1de4b6c-kube-api-access-ctj54\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp\" (UID: \"8c88b7e3-1f93-456a-87ed-4894a1de4b6c\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp" Oct 30 16:52:31.449588 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:31.449555 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b595920-sbs2r" event={"ID":"fff2bfc7-3c82-4a8f-a726-100912be8523","Type":"ContainerStarted","Data":"e2c61b4be9e7e57ae811dcea1b6dc08c5742e4529299433e2895e822254897b0"} Oct 30 16:52:31.451830 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:31.451766 4493 generic.go:356] "Generic (PLEG): container finished" podID="bca5b866-90c9-484e-b043-e048c3ca2c23" containerID="f144b4b78e6d9b509eb2c59152e5dff55c008cf44c4a4137fd1ca3a1d0f2e1e2" exitCode=0 Oct 30 16:52:31.452176 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:31.452143 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46c3c12a-4cd8v" event={"ID":"bca5b866-90c9-484e-b043-e048c3ca2c23","Type":"ContainerDied","Data":"f144b4b78e6d9b509eb2c59152e5dff55c008cf44c4a4137fd1ca3a1d0f2e1e2"} Oct 30 16:52:31.490087 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:31.490062 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp" Oct 30 16:52:31.831310 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:31.831280 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp"] Oct 30 16:52:31.832584 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:52:31.832552 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c88b7e3_1f93_456a_87ed_4894a1de4b6c.slice/crio-6ba30ce32cfcdf975d31f13c1a73a941dc7208179031cee1146651eb0c09c9cf WatchSource:0}: Error finding container 6ba30ce32cfcdf975d31f13c1a73a941dc7208179031cee1146651eb0c09c9cf: Status 404 returned error can't find the container with id 6ba30ce32cfcdf975d31f13c1a73a941dc7208179031cee1146651eb0c09c9cf Oct 30 16:52:31.835806 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:31.835757 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:52:31.835904 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:31.835850 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:52:31.835976 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:31.835908 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:52:32.463569 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:32.463535 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp" event={"ID":"8c88b7e3-1f93-456a-87ed-4894a1de4b6c","Type":"ContainerStarted","Data":"d85e60dc92c35d3c7d597c161adb0d1f2ebfd6787730288e39a4bc07579f4dd4"} Oct 30 16:52:32.464025 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:32.463599 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp" event={"ID":"8c88b7e3-1f93-456a-87ed-4894a1de4b6c","Type":"ContainerStarted","Data":"6ba30ce32cfcdf975d31f13c1a73a941dc7208179031cee1146651eb0c09c9cf"} Oct 30 16:52:32.465187 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:32.465156 4493 generic.go:356] "Generic (PLEG): container finished" podID="fff2bfc7-3c82-4a8f-a726-100912be8523" containerID="30ac8d1067882e11748c33eed3e082385242dda3da430b44c34f171e93b871e5" exitCode=0 Oct 30 16:52:32.465311 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:32.465227 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b595920-sbs2r" event={"ID":"fff2bfc7-3c82-4a8f-a726-100912be8523","Type":"ContainerDied","Data":"30ac8d1067882e11748c33eed3e082385242dda3da430b44c34f171e93b871e5"} Oct 30 16:52:32.490958 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:32.490927 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp" Oct 30 16:52:32.492559 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:32.492535 4493 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body= Oct 30 16:52:32.492714 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:32.492580 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp" podUID="8c88b7e3-1f93-456a-87ed-4894a1de4b6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" Oct 30 16:52:32.545952 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:32.521979 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp" podStartSLOduration=2.521962218 podStartE2EDuration="2.521962218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:52:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:52:32.507677234 +0000 UTC m=+1779.734852335" watchObservedRunningTime="2026-10-30 16:52:32.521962218 +0000 UTC m=+1779.749137318" Oct 30 16:52:32.656875 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:32.656849 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46c3c12a-4cd8v" Oct 30 16:52:32.706656 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:32.706626 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bca5b866-90c9-484e-b043-e048c3ca2c23-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bca5b866-90c9-484e-b043-e048c3ca2c23-maas-api-openshift-ca\") pod \"bca5b866-90c9-484e-b043-e048c3ca2c23\" (UID: \"bca5b866-90c9-484e-b043-e048c3ca2c23\") " Oct 30 16:52:32.706903 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:32.706874 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bca5b866-90c9-484e-b043-e048c3ca2c23-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bca5b866-90c9-484e-b043-e048c3ca2c23-maas-api-xks-ca\") pod \"bca5b866-90c9-484e-b043-e048c3ca2c23\" (UID: \"bca5b866-90c9-484e-b043-e048c3ca2c23\") " Oct 30 16:52:32.706996 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:32.706950 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bca5b866-90c9-484e-b043-e048c3ca2c23-kube-api-access-hzq5v\" (UniqueName: \"kubernetes.io/projected/bca5b866-90c9-484e-b043-e048c3ca2c23-kube-api-access-hzq5v\") pod \"bca5b866-90c9-484e-b043-e048c3ca2c23\" (UID: \"bca5b866-90c9-484e-b043-e048c3ca2c23\") " Oct 30 16:52:32.707244 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:32.707213 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bca5b866-90c9-484e-b043-e048c3ca2c23-maas-api-openshift-ca" pod "bca5b866-90c9-484e-b043-e048c3ca2c23" (UID: "bca5b866-90c9-484e-b043-e048c3ca2c23"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:52:32.707818 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:32.707758 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bca5b866-90c9-484e-b043-e048c3ca2c23-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:52:32.709218 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:32.709188 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bca5b866-90c9-484e-b043-e048c3ca2c23-maas-api-xks-ca" pod "bca5b866-90c9-484e-b043-e048c3ca2c23" (UID: "bca5b866-90c9-484e-b043-e048c3ca2c23"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:52:32.709848 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:32.709762 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bca5b866-90c9-484e-b043-e048c3ca2c23-kube-api-access-hzq5v" pod "bca5b866-90c9-484e-b043-e048c3ca2c23" (UID: "bca5b866-90c9-484e-b043-e048c3ca2c23"). InnerVolumeSpecName "kube-api-access-hzq5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:52:32.809181 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:32.809151 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bca5b866-90c9-484e-b043-e048c3ca2c23-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:52:32.809319 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:32.809185 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzq5v\" (UniqueName: \"kubernetes.io/projected/bca5b866-90c9-484e-b043-e048c3ca2c23-kube-api-access-hzq5v\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:52:33.474801 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:33.474767 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46c3c12a-4cd8v" Oct 30 16:52:33.475265 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:33.474763 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46c3c12a-4cd8v" event={"ID":"bca5b866-90c9-484e-b043-e048c3ca2c23","Type":"ContainerDied","Data":"2f7d697bfe2765603af9cfdc3d9097882c4cb6e848cefaa7c13a855d1607be49"} Oct 30 16:52:33.475265 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:33.474922 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f7d697bfe2765603af9cfdc3d9097882c4cb6e848cefaa7c13a855d1607be49" Oct 30 16:52:33.490426 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:33.490391 4493 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body= Oct 30 16:52:33.490580 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:33.490437 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp" podUID="8c88b7e3-1f93-456a-87ed-4894a1de4b6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" Oct 30 16:52:33.687689 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:33.687668 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b595920-sbs2r" Oct 30 16:52:33.747892 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:33.747806 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fff2bfc7-3c82-4a8f-a726-100912be8523-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fff2bfc7-3c82-4a8f-a726-100912be8523-maas-api-openshift-ca\") pod \"fff2bfc7-3c82-4a8f-a726-100912be8523\" (UID: \"fff2bfc7-3c82-4a8f-a726-100912be8523\") " Oct 30 16:52:33.747892 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:33.747871 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fff2bfc7-3c82-4a8f-a726-100912be8523-kube-api-access-jz855\" (UniqueName: \"kubernetes.io/projected/fff2bfc7-3c82-4a8f-a726-100912be8523-kube-api-access-jz855\") pod \"fff2bfc7-3c82-4a8f-a726-100912be8523\" (UID: \"fff2bfc7-3c82-4a8f-a726-100912be8523\") " Oct 30 16:52:33.748132 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:33.748000 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fff2bfc7-3c82-4a8f-a726-100912be8523-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fff2bfc7-3c82-4a8f-a726-100912be8523-maas-api-xks-ca\") pod \"fff2bfc7-3c82-4a8f-a726-100912be8523\" (UID: \"fff2bfc7-3c82-4a8f-a726-100912be8523\") " Oct 30 16:52:33.748632 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:33.748382 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fff2bfc7-3c82-4a8f-a726-100912be8523-maas-api-openshift-ca" pod "fff2bfc7-3c82-4a8f-a726-100912be8523" (UID: "fff2bfc7-3c82-4a8f-a726-100912be8523"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:52:33.748990 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:33.748967 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fff2bfc7-3c82-4a8f-a726-100912be8523-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:52:33.750766 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:33.750739 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fff2bfc7-3c82-4a8f-a726-100912be8523-maas-api-xks-ca" pod "fff2bfc7-3c82-4a8f-a726-100912be8523" (UID: "fff2bfc7-3c82-4a8f-a726-100912be8523"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:52:33.750766 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:33.750752 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fff2bfc7-3c82-4a8f-a726-100912be8523-kube-api-access-jz855" pod "fff2bfc7-3c82-4a8f-a726-100912be8523" (UID: "fff2bfc7-3c82-4a8f-a726-100912be8523"). InnerVolumeSpecName "kube-api-access-jz855". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:52:33.849757 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:33.849721 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fff2bfc7-3c82-4a8f-a726-100912be8523-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:52:33.849757 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:33.849752 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jz855\" (UniqueName: \"kubernetes.io/projected/fff2bfc7-3c82-4a8f-a726-100912be8523-kube-api-access-jz855\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:52:34.485475 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:34.485443 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b595920-sbs2r" Oct 30 16:52:34.485909 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:34.485472 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b595920-sbs2r" event={"ID":"fff2bfc7-3c82-4a8f-a726-100912be8523","Type":"ContainerDied","Data":"e2c61b4be9e7e57ae811dcea1b6dc08c5742e4529299433e2895e822254897b0"} Oct 30 16:52:34.485909 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:34.485546 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2c61b4be9e7e57ae811dcea1b6dc08c5742e4529299433e2895e822254897b0" Oct 30 16:52:34.490530 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:34.490500 4493 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body= Oct 30 16:52:34.490652 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:34.490565 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp" podUID="8c88b7e3-1f93-456a-87ed-4894a1de4b6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" Oct 30 16:52:34.516810 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:52:34.514320 4493 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfff2bfc7_3c82_4a8f_a726_100912be8523.slice/crio-e2c61b4be9e7e57ae811dcea1b6dc08c5742e4529299433e2895e822254897b0\": RecentStats: unable to find data in memory cache]" Oct 30 16:52:35.490402 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:35.490372 4493 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body= Oct 30 16:52:35.490851 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:35.490428 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp" podUID="8c88b7e3-1f93-456a-87ed-4894a1de4b6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" Oct 30 16:52:36.251531 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:36.250691 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39cda64b-6zghk"] Oct 30 16:52:36.251531 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:36.251254 4493 cpu_manager.go:414] "containerMap: removing container" podUID="fff2bfc7-3c82-4a8f-a726-100912be8523" containerName="revoke-keys" Oct 30 16:52:36.251531 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:36.251267 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fff2bfc7-3c82-4a8f-a726-100912be8523" containerName="revoke-keys" Oct 30 16:52:36.251531 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:36.251288 4493 cpu_manager.go:414] "containerMap: removing container" podUID="bca5b866-90c9-484e-b043-e048c3ca2c23" containerName="revoke-keys" Oct 30 16:52:36.251531 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:36.251293 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bca5b866-90c9-484e-b043-e048c3ca2c23" containerName="revoke-keys" Oct 30 16:52:36.251531 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:36.251400 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="fff2bfc7-3c82-4a8f-a726-100912be8523" containerName="revoke-keys" Oct 30 16:52:36.251531 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:36.251413 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="bca5b866-90c9-484e-b043-e048c3ca2c23" containerName="revoke-keys" Oct 30 16:52:36.255852 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:36.254719 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39cda64b-6zghk" Oct 30 16:52:36.270060 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:36.270034 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:52:36.287045 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:36.281915 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:52:36.287045 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:36.281951 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 16:52:36.287045 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:36.281979 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:52:36.287045 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:36.283136 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fdbeb20-d0c6-4f03-a849-f05582689593-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39cda64b-6zghk\" (UID: \"3fdbeb20-d0c6-4f03-a849-f05582689593\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39cda64b-6zghk" Oct 30 16:52:36.287045 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:36.283259 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fdbeb20-d0c6-4f03-a849-f05582689593-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39cda64b-6zghk\" (UID: \"3fdbeb20-d0c6-4f03-a849-f05582689593\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39cda64b-6zghk" Oct 30 16:52:36.287045 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:36.283361 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vqdc\" (UniqueName: \"kubernetes.io/projected/3fdbeb20-d0c6-4f03-a849-f05582689593-kube-api-access-7vqdc\") pod \"maas-api-revoke-sub-39cda64b-6zghk\" (UID: \"3fdbeb20-d0c6-4f03-a849-f05582689593\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39cda64b-6zghk" Oct 30 16:52:36.289153 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:36.289126 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39cda64b-6zghk"] Oct 30 16:52:36.384380 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:36.384347 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fdbeb20-d0c6-4f03-a849-f05582689593-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39cda64b-6zghk\" (UID: \"3fdbeb20-d0c6-4f03-a849-f05582689593\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39cda64b-6zghk" Oct 30 16:52:36.384583 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:36.384454 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vqdc\" (UniqueName: \"kubernetes.io/projected/3fdbeb20-d0c6-4f03-a849-f05582689593-kube-api-access-7vqdc\") pod \"maas-api-revoke-sub-39cda64b-6zghk\" (UID: \"3fdbeb20-d0c6-4f03-a849-f05582689593\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39cda64b-6zghk" Oct 30 16:52:36.384583 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:36.384565 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fdbeb20-d0c6-4f03-a849-f05582689593-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39cda64b-6zghk\" (UID: \"3fdbeb20-d0c6-4f03-a849-f05582689593\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39cda64b-6zghk" Oct 30 16:52:36.385255 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:36.385186 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fdbeb20-d0c6-4f03-a849-f05582689593-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39cda64b-6zghk\" (UID: \"3fdbeb20-d0c6-4f03-a849-f05582689593\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39cda64b-6zghk" Oct 30 16:52:36.386875 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:36.386857 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fdbeb20-d0c6-4f03-a849-f05582689593-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39cda64b-6zghk\" (UID: \"3fdbeb20-d0c6-4f03-a849-f05582689593\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39cda64b-6zghk" Oct 30 16:52:36.433807 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:36.432561 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vqdc\" (UniqueName: \"kubernetes.io/projected/3fdbeb20-d0c6-4f03-a849-f05582689593-kube-api-access-7vqdc\") pod \"maas-api-revoke-sub-39cda64b-6zghk\" (UID: \"3fdbeb20-d0c6-4f03-a849-f05582689593\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39cda64b-6zghk" Oct 30 16:52:36.491439 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:36.491399 4493 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body= Oct 30 16:52:36.491872 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:36.491462 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp" podUID="8c88b7e3-1f93-456a-87ed-4894a1de4b6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" Oct 30 16:52:36.570481 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:36.570395 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39cda64b-6zghk" Oct 30 16:52:36.956079 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:36.955838 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39cda64b-6zghk"] Oct 30 16:52:37.490439 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:37.490408 4493 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body= Oct 30 16:52:37.490621 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:37.490460 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp" podUID="8c88b7e3-1f93-456a-87ed-4894a1de4b6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" Oct 30 16:52:37.536383 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:37.536347 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39cda64b-6zghk" event={"ID":"3fdbeb20-d0c6-4f03-a849-f05582689593","Type":"ContainerStarted","Data":"83a0f46347960874008431b4961415c78ceb5d3ad125210f1ffddb1bd60fe499"} Oct 30 16:52:37.536803 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:37.536411 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39cda64b-6zghk" event={"ID":"3fdbeb20-d0c6-4f03-a849-f05582689593","Type":"ContainerStarted","Data":"81009629087f93ccd36c1bbd94b610c498a91d708b4b3c65a01caef2dba07fdc"} Oct 30 16:52:37.579444 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:37.579392 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39cda64b-6zghk" podStartSLOduration=1.5793784199999998 podStartE2EDuration="1.57937842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:52:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:52:37.575748233 +0000 UTC m=+1784.802923333" watchObservedRunningTime="2026-10-30 16:52:37.57937842 +0000 UTC m=+1784.806553519" Oct 30 16:52:38.491176 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:38.491143 4493 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body= Oct 30 16:52:38.491354 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:38.491200 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp" podUID="8c88b7e3-1f93-456a-87ed-4894a1de4b6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" Oct 30 16:52:38.546739 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:38.546703 4493 generic.go:356] "Generic (PLEG): container finished" podID="3fdbeb20-d0c6-4f03-a849-f05582689593" containerID="83a0f46347960874008431b4961415c78ceb5d3ad125210f1ffddb1bd60fe499" exitCode=0 Oct 30 16:52:38.547167 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:38.546740 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39cda64b-6zghk" event={"ID":"3fdbeb20-d0c6-4f03-a849-f05582689593","Type":"ContainerDied","Data":"83a0f46347960874008431b4961415c78ceb5d3ad125210f1ffddb1bd60fe499"} Oct 30 16:52:39.727645 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:39.727623 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39cda64b-6zghk" Oct 30 16:52:39.811232 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:39.809630 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp" Oct 30 16:52:39.811232 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:39.810478 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp" Oct 30 16:52:39.814310 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:39.814261 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp" Oct 30 16:52:39.862070 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:39.862043 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fdbeb20-d0c6-4f03-a849-f05582689593-kube-api-access-7vqdc\" (UniqueName: \"kubernetes.io/projected/3fdbeb20-d0c6-4f03-a849-f05582689593-kube-api-access-7vqdc\") pod \"3fdbeb20-d0c6-4f03-a849-f05582689593\" (UID: \"3fdbeb20-d0c6-4f03-a849-f05582689593\") " Oct 30 16:52:39.862169 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:39.862090 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fdbeb20-d0c6-4f03-a849-f05582689593-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fdbeb20-d0c6-4f03-a849-f05582689593-maas-api-openshift-ca\") pod \"3fdbeb20-d0c6-4f03-a849-f05582689593\" (UID: \"3fdbeb20-d0c6-4f03-a849-f05582689593\") " Oct 30 16:52:39.862169 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:39.862114 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fdbeb20-d0c6-4f03-a849-f05582689593-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fdbeb20-d0c6-4f03-a849-f05582689593-maas-api-xks-ca\") pod \"3fdbeb20-d0c6-4f03-a849-f05582689593\" (UID: \"3fdbeb20-d0c6-4f03-a849-f05582689593\") " Oct 30 16:52:39.862526 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:39.862490 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fdbeb20-d0c6-4f03-a849-f05582689593-maas-api-openshift-ca" pod "3fdbeb20-d0c6-4f03-a849-f05582689593" (UID: "3fdbeb20-d0c6-4f03-a849-f05582689593"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:52:39.864374 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:39.864351 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fdbeb20-d0c6-4f03-a849-f05582689593-maas-api-xks-ca" pod "3fdbeb20-d0c6-4f03-a849-f05582689593" (UID: "3fdbeb20-d0c6-4f03-a849-f05582689593"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:52:39.864465 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:39.864442 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fdbeb20-d0c6-4f03-a849-f05582689593-kube-api-access-7vqdc" pod "3fdbeb20-d0c6-4f03-a849-f05582689593" (UID: "3fdbeb20-d0c6-4f03-a849-f05582689593"). InnerVolumeSpecName "kube-api-access-7vqdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:52:39.963988 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:39.963955 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vqdc\" (UniqueName: \"kubernetes.io/projected/3fdbeb20-d0c6-4f03-a849-f05582689593-kube-api-access-7vqdc\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:52:39.963988 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:39.963989 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fdbeb20-d0c6-4f03-a849-f05582689593-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:52:39.964250 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:39.964006 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fdbeb20-d0c6-4f03-a849-f05582689593-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:52:40.569362 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:40.569326 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39cda64b-6zghk" event={"ID":"3fdbeb20-d0c6-4f03-a849-f05582689593","Type":"ContainerDied","Data":"81009629087f93ccd36c1bbd94b610c498a91d708b4b3c65a01caef2dba07fdc"} Oct 30 16:52:40.569362 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:40.569350 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39cda64b-6zghk" Oct 30 16:52:40.569575 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:40.569376 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81009629087f93ccd36c1bbd94b610c498a91d708b4b3c65a01caef2dba07fdc" Oct 30 16:52:43.453487 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:43.453449 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb8208a8-f5gvb"] Oct 30 16:52:43.454347 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:43.454318 4493 cpu_manager.go:414] "containerMap: removing container" podUID="3fdbeb20-d0c6-4f03-a849-f05582689593" containerName="revoke-keys" Oct 30 16:52:43.454347 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:43.454338 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fdbeb20-d0c6-4f03-a849-f05582689593" containerName="revoke-keys" Oct 30 16:52:43.454553 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:43.454470 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fdbeb20-d0c6-4f03-a849-f05582689593" containerName="revoke-keys" Oct 30 16:52:43.458559 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:43.458541 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8208a8-f5gvb" Oct 30 16:52:43.480488 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:43.480459 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb8208a8-f5gvb"] Oct 30 16:52:43.485048 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:43.485027 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:52:43.485256 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:43.485219 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:52:43.485384 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:43.485370 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:52:43.485468 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:43.485380 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 16:52:43.514977 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:43.514942 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee653e75-3a28-4c94-8e13-a1c1d664f708-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb8208a8-f5gvb\" (UID: \"ee653e75-3a28-4c94-8e13-a1c1d664f708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8208a8-f5gvb" Oct 30 16:52:43.515189 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:43.514993 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmttq\" (UniqueName: \"kubernetes.io/projected/ee653e75-3a28-4c94-8e13-a1c1d664f708-kube-api-access-cmttq\") pod \"maas-api-revoke-sub-eb8208a8-f5gvb\" (UID: \"ee653e75-3a28-4c94-8e13-a1c1d664f708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8208a8-f5gvb" Oct 30 16:52:43.515189 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:43.515028 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee653e75-3a28-4c94-8e13-a1c1d664f708-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb8208a8-f5gvb\" (UID: \"ee653e75-3a28-4c94-8e13-a1c1d664f708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8208a8-f5gvb" Oct 30 16:52:43.616765 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:43.616723 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee653e75-3a28-4c94-8e13-a1c1d664f708-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb8208a8-f5gvb\" (UID: \"ee653e75-3a28-4c94-8e13-a1c1d664f708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8208a8-f5gvb" Oct 30 16:52:43.616986 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:43.616775 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmttq\" (UniqueName: \"kubernetes.io/projected/ee653e75-3a28-4c94-8e13-a1c1d664f708-kube-api-access-cmttq\") pod \"maas-api-revoke-sub-eb8208a8-f5gvb\" (UID: \"ee653e75-3a28-4c94-8e13-a1c1d664f708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8208a8-f5gvb" Oct 30 16:52:43.616986 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:43.616846 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee653e75-3a28-4c94-8e13-a1c1d664f708-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb8208a8-f5gvb\" (UID: \"ee653e75-3a28-4c94-8e13-a1c1d664f708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8208a8-f5gvb" Oct 30 16:52:43.617398 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:43.617376 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee653e75-3a28-4c94-8e13-a1c1d664f708-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb8208a8-f5gvb\" (UID: \"ee653e75-3a28-4c94-8e13-a1c1d664f708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8208a8-f5gvb" Oct 30 16:52:43.619407 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:43.619385 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee653e75-3a28-4c94-8e13-a1c1d664f708-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb8208a8-f5gvb\" (UID: \"ee653e75-3a28-4c94-8e13-a1c1d664f708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8208a8-f5gvb" Oct 30 16:52:43.668866 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:43.668826 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmttq\" (UniqueName: \"kubernetes.io/projected/ee653e75-3a28-4c94-8e13-a1c1d664f708-kube-api-access-cmttq\") pod \"maas-api-revoke-sub-eb8208a8-f5gvb\" (UID: \"ee653e75-3a28-4c94-8e13-a1c1d664f708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8208a8-f5gvb" Oct 30 16:52:43.773891 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:43.773799 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8208a8-f5gvb" Oct 30 16:52:44.105278 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:44.105241 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb8208a8-f5gvb"] Oct 30 16:52:44.105406 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:52:44.105247 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee653e75_3a28_4c94_8e13_a1c1d664f708.slice/crio-f9e69559d3b988e8e62db9cadbd5c796196124d89a76e65aeb2945657f52b974 WatchSource:0}: Error finding container f9e69559d3b988e8e62db9cadbd5c796196124d89a76e65aeb2945657f52b974: Status 404 returned error can't find the container with id f9e69559d3b988e8e62db9cadbd5c796196124d89a76e65aeb2945657f52b974 Oct 30 16:52:44.612039 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:44.612002 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8208a8-f5gvb" event={"ID":"ee653e75-3a28-4c94-8e13-a1c1d664f708","Type":"ContainerStarted","Data":"f9e69559d3b988e8e62db9cadbd5c796196124d89a76e65aeb2945657f52b974"} Oct 30 16:52:45.624312 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:45.624222 4493 generic.go:356] "Generic (PLEG): container finished" podID="ee653e75-3a28-4c94-8e13-a1c1d664f708" containerID="d29fd64292595d492d11f0c631add78c2ede13e2c346fcea1efbbaba254324ba" exitCode=0 Oct 30 16:52:45.624745 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:45.624302 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8208a8-f5gvb" event={"ID":"ee653e75-3a28-4c94-8e13-a1c1d664f708","Type":"ContainerDied","Data":"d29fd64292595d492d11f0c631add78c2ede13e2c346fcea1efbbaba254324ba"} Oct 30 16:52:46.738309 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:46.738267 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5f04f3b-ppcrb"] Oct 30 16:52:46.743444 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:46.743418 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5f04f3b-ppcrb" Oct 30 16:52:46.767297 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:46.767111 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5f04f3b-ppcrb"] Oct 30 16:52:46.822905 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:46.822878 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8208a8-f5gvb" Oct 30 16:52:46.863119 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:46.863072 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrzd6\" (UniqueName: \"kubernetes.io/projected/0cc6b894-299f-474a-ba77-65ccbe882cc6-kube-api-access-zrzd6\") pod \"maas-api-revoke-sub-b5f04f3b-ppcrb\" (UID: \"0cc6b894-299f-474a-ba77-65ccbe882cc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5f04f3b-ppcrb" Oct 30 16:52:46.863324 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:46.863125 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cc6b894-299f-474a-ba77-65ccbe882cc6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5f04f3b-ppcrb\" (UID: \"0cc6b894-299f-474a-ba77-65ccbe882cc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5f04f3b-ppcrb" Oct 30 16:52:46.863324 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:46.863250 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cc6b894-299f-474a-ba77-65ccbe882cc6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5f04f3b-ppcrb\" (UID: \"0cc6b894-299f-474a-ba77-65ccbe882cc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5f04f3b-ppcrb" Oct 30 16:52:46.964297 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:46.964270 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee653e75-3a28-4c94-8e13-a1c1d664f708-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee653e75-3a28-4c94-8e13-a1c1d664f708-maas-api-xks-ca\") pod \"ee653e75-3a28-4c94-8e13-a1c1d664f708\" (UID: \"ee653e75-3a28-4c94-8e13-a1c1d664f708\") " Oct 30 16:52:46.964472 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:46.964322 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee653e75-3a28-4c94-8e13-a1c1d664f708-kube-api-access-cmttq\" (UniqueName: \"kubernetes.io/projected/ee653e75-3a28-4c94-8e13-a1c1d664f708-kube-api-access-cmttq\") pod \"ee653e75-3a28-4c94-8e13-a1c1d664f708\" (UID: \"ee653e75-3a28-4c94-8e13-a1c1d664f708\") " Oct 30 16:52:46.964472 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:46.964377 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee653e75-3a28-4c94-8e13-a1c1d664f708-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee653e75-3a28-4c94-8e13-a1c1d664f708-maas-api-openshift-ca\") pod \"ee653e75-3a28-4c94-8e13-a1c1d664f708\" (UID: \"ee653e75-3a28-4c94-8e13-a1c1d664f708\") " Oct 30 16:52:46.964580 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:46.964565 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cc6b894-299f-474a-ba77-65ccbe882cc6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5f04f3b-ppcrb\" (UID: \"0cc6b894-299f-474a-ba77-65ccbe882cc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5f04f3b-ppcrb" Oct 30 16:52:46.965236 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:46.964739 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrzd6\" (UniqueName: \"kubernetes.io/projected/0cc6b894-299f-474a-ba77-65ccbe882cc6-kube-api-access-zrzd6\") pod \"maas-api-revoke-sub-b5f04f3b-ppcrb\" (UID: \"0cc6b894-299f-474a-ba77-65ccbe882cc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5f04f3b-ppcrb" Oct 30 16:52:46.965236 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:46.964799 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cc6b894-299f-474a-ba77-65ccbe882cc6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5f04f3b-ppcrb\" (UID: \"0cc6b894-299f-474a-ba77-65ccbe882cc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5f04f3b-ppcrb" Oct 30 16:52:46.965236 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:46.964822 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee653e75-3a28-4c94-8e13-a1c1d664f708-maas-api-openshift-ca" pod "ee653e75-3a28-4c94-8e13-a1c1d664f708" (UID: "ee653e75-3a28-4c94-8e13-a1c1d664f708"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:52:46.965514 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:46.965268 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cc6b894-299f-474a-ba77-65ccbe882cc6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5f04f3b-ppcrb\" (UID: \"0cc6b894-299f-474a-ba77-65ccbe882cc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5f04f3b-ppcrb" Oct 30 16:52:46.966725 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:46.966697 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee653e75-3a28-4c94-8e13-a1c1d664f708-maas-api-xks-ca" pod "ee653e75-3a28-4c94-8e13-a1c1d664f708" (UID: "ee653e75-3a28-4c94-8e13-a1c1d664f708"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:52:46.966880 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:46.966829 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee653e75-3a28-4c94-8e13-a1c1d664f708-kube-api-access-cmttq" pod "ee653e75-3a28-4c94-8e13-a1c1d664f708" (UID: "ee653e75-3a28-4c94-8e13-a1c1d664f708"). InnerVolumeSpecName "kube-api-access-cmttq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:52:46.967270 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:46.967253 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cc6b894-299f-474a-ba77-65ccbe882cc6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5f04f3b-ppcrb\" (UID: \"0cc6b894-299f-474a-ba77-65ccbe882cc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5f04f3b-ppcrb" Oct 30 16:52:46.987819 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:46.987767 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrzd6\" (UniqueName: \"kubernetes.io/projected/0cc6b894-299f-474a-ba77-65ccbe882cc6-kube-api-access-zrzd6\") pod \"maas-api-revoke-sub-b5f04f3b-ppcrb\" (UID: \"0cc6b894-299f-474a-ba77-65ccbe882cc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5f04f3b-ppcrb" Oct 30 16:52:47.066047 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:47.065942 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee653e75-3a28-4c94-8e13-a1c1d664f708-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:52:47.066047 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:47.065979 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmttq\" (UniqueName: \"kubernetes.io/projected/ee653e75-3a28-4c94-8e13-a1c1d664f708-kube-api-access-cmttq\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:52:47.066047 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:47.065998 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee653e75-3a28-4c94-8e13-a1c1d664f708-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:52:47.071405 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:47.071381 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5f04f3b-ppcrb" Oct 30 16:52:47.470388 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:52:47.470355 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cc6b894_299f_474a_ba77_65ccbe882cc6.slice/crio-8610d35ae288ee39cc3768e57470940549675c32ce068283324735ad3d28781e WatchSource:0}: Error finding container 8610d35ae288ee39cc3768e57470940549675c32ce068283324735ad3d28781e: Status 404 returned error can't find the container with id 8610d35ae288ee39cc3768e57470940549675c32ce068283324735ad3d28781e Oct 30 16:52:47.474316 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:47.474287 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5f04f3b-ppcrb"] Oct 30 16:52:47.672363 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:47.672322 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5f04f3b-ppcrb" event={"ID":"0cc6b894-299f-474a-ba77-65ccbe882cc6","Type":"ContainerStarted","Data":"8610d35ae288ee39cc3768e57470940549675c32ce068283324735ad3d28781e"} Oct 30 16:52:47.674284 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:47.674260 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8208a8-f5gvb" Oct 30 16:52:47.674427 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:47.674403 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8208a8-f5gvb" event={"ID":"ee653e75-3a28-4c94-8e13-a1c1d664f708","Type":"ContainerDied","Data":"f9e69559d3b988e8e62db9cadbd5c796196124d89a76e65aeb2945657f52b974"} Oct 30 16:52:47.674495 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:47.674452 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9e69559d3b988e8e62db9cadbd5c796196124d89a76e65aeb2945657f52b974" Oct 30 16:52:48.734845 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:48.713159 4493 generic.go:356] "Generic (PLEG): container finished" podID="0cc6b894-299f-474a-ba77-65ccbe882cc6" containerID="ae277907eb181d5e66fc9fc8d4da642b180d1c3de3647b4dde66f4606b5929aa" exitCode=0 Oct 30 16:52:48.734845 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:48.713224 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5f04f3b-ppcrb" event={"ID":"0cc6b894-299f-474a-ba77-65ccbe882cc6","Type":"ContainerDied","Data":"ae277907eb181d5e66fc9fc8d4da642b180d1c3de3647b4dde66f4606b5929aa"} Oct 30 16:52:49.882370 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:49.882345 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5f04f3b-ppcrb" Oct 30 16:52:50.048562 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:50.048458 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cc6b894-299f-474a-ba77-65ccbe882cc6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cc6b894-299f-474a-ba77-65ccbe882cc6-maas-api-xks-ca\") pod \"0cc6b894-299f-474a-ba77-65ccbe882cc6\" (UID: \"0cc6b894-299f-474a-ba77-65ccbe882cc6\") " Oct 30 16:52:50.048744 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:50.048583 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cc6b894-299f-474a-ba77-65ccbe882cc6-kube-api-access-zrzd6\" (UniqueName: \"kubernetes.io/projected/0cc6b894-299f-474a-ba77-65ccbe882cc6-kube-api-access-zrzd6\") pod \"0cc6b894-299f-474a-ba77-65ccbe882cc6\" (UID: \"0cc6b894-299f-474a-ba77-65ccbe882cc6\") " Oct 30 16:52:50.048847 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:50.048825 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cc6b894-299f-474a-ba77-65ccbe882cc6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cc6b894-299f-474a-ba77-65ccbe882cc6-maas-api-openshift-ca\") pod \"0cc6b894-299f-474a-ba77-65ccbe882cc6\" (UID: \"0cc6b894-299f-474a-ba77-65ccbe882cc6\") " Oct 30 16:52:50.049398 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:50.049370 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cc6b894-299f-474a-ba77-65ccbe882cc6-maas-api-openshift-ca" pod "0cc6b894-299f-474a-ba77-65ccbe882cc6" (UID: "0cc6b894-299f-474a-ba77-65ccbe882cc6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:52:50.050891 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:50.050866 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cc6b894-299f-474a-ba77-65ccbe882cc6-maas-api-xks-ca" pod "0cc6b894-299f-474a-ba77-65ccbe882cc6" (UID: "0cc6b894-299f-474a-ba77-65ccbe882cc6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:52:50.050978 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:50.050943 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cc6b894-299f-474a-ba77-65ccbe882cc6-kube-api-access-zrzd6" pod "0cc6b894-299f-474a-ba77-65ccbe882cc6" (UID: "0cc6b894-299f-474a-ba77-65ccbe882cc6"). InnerVolumeSpecName "kube-api-access-zrzd6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:52:50.150802 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:50.150760 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cc6b894-299f-474a-ba77-65ccbe882cc6-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:52:50.150922 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:50.150819 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cc6b894-299f-474a-ba77-65ccbe882cc6-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:52:50.150922 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:50.150834 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrzd6\" (UniqueName: \"kubernetes.io/projected/0cc6b894-299f-474a-ba77-65ccbe882cc6-kube-api-access-zrzd6\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:52:50.736151 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:50.736116 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5f04f3b-ppcrb" Oct 30 16:52:50.736342 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:50.736150 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5f04f3b-ppcrb" event={"ID":"0cc6b894-299f-474a-ba77-65ccbe882cc6","Type":"ContainerDied","Data":"8610d35ae288ee39cc3768e57470940549675c32ce068283324735ad3d28781e"} Oct 30 16:52:50.736342 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:50.736216 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8610d35ae288ee39cc3768e57470940549675c32ce068283324735ad3d28781e" Oct 30 16:52:51.662376 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:51.662336 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 16:52:51.977979 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:51.977900 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-52f3fd-tg7rl"] Oct 30 16:52:51.978574 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:51.978554 4493 cpu_manager.go:414] "containerMap: removing container" podUID="ee653e75-3a28-4c94-8e13-a1c1d664f708" containerName="revoke-keys" Oct 30 16:52:51.978679 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:51.978575 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee653e75-3a28-4c94-8e13-a1c1d664f708" containerName="revoke-keys" Oct 30 16:52:51.978679 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:51.978633 4493 cpu_manager.go:414] "containerMap: removing container" podUID="0cc6b894-299f-474a-ba77-65ccbe882cc6" containerName="revoke-keys" Oct 30 16:52:51.978679 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:51.978641 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cc6b894-299f-474a-ba77-65ccbe882cc6" containerName="revoke-keys" Oct 30 16:52:51.978880 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:51.978777 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cc6b894-299f-474a-ba77-65ccbe882cc6" containerName="revoke-keys" Oct 30 16:52:51.978880 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:51.978820 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee653e75-3a28-4c94-8e13-a1c1d664f708" containerName="revoke-keys" Oct 30 16:52:51.982699 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:51.982671 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-52f3fd-tg7rl" Oct 30 16:52:52.013712 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:52.013435 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:52:52.032837 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:52.025449 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:52:52.032837 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:52.026077 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 16:52:52.032837 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:52.026154 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:52:52.038154 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:52.038119 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-52f3fd-tg7rl"] Oct 30 16:52:52.082647 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:52.081432 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a11e515d-f825-4e01-9b31-31bc06fc2fa5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-52f3fd-tg7rl\" (UID: \"a11e515d-f825-4e01-9b31-31bc06fc2fa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-52f3fd-tg7rl" Oct 30 16:52:52.082647 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:52.081538 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-855th\" (UniqueName: \"kubernetes.io/projected/a11e515d-f825-4e01-9b31-31bc06fc2fa5-kube-api-access-855th\") pod \"maas-api-revoke-keys-e2e-praxis-a-52f3fd-tg7rl\" (UID: \"a11e515d-f825-4e01-9b31-31bc06fc2fa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-52f3fd-tg7rl" Oct 30 16:52:52.082647 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:52.081654 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a11e515d-f825-4e01-9b31-31bc06fc2fa5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-52f3fd-tg7rl\" (UID: \"a11e515d-f825-4e01-9b31-31bc06fc2fa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-52f3fd-tg7rl" Oct 30 16:52:52.183343 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:52.183297 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a11e515d-f825-4e01-9b31-31bc06fc2fa5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-52f3fd-tg7rl\" (UID: \"a11e515d-f825-4e01-9b31-31bc06fc2fa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-52f3fd-tg7rl" Oct 30 16:52:52.183520 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:52.183408 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-855th\" (UniqueName: \"kubernetes.io/projected/a11e515d-f825-4e01-9b31-31bc06fc2fa5-kube-api-access-855th\") pod \"maas-api-revoke-keys-e2e-praxis-a-52f3fd-tg7rl\" (UID: \"a11e515d-f825-4e01-9b31-31bc06fc2fa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-52f3fd-tg7rl" Oct 30 16:52:52.183608 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:52.183524 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a11e515d-f825-4e01-9b31-31bc06fc2fa5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-52f3fd-tg7rl\" (UID: \"a11e515d-f825-4e01-9b31-31bc06fc2fa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-52f3fd-tg7rl" Oct 30 16:52:52.184453 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:52.184424 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a11e515d-f825-4e01-9b31-31bc06fc2fa5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-52f3fd-tg7rl\" (UID: \"a11e515d-f825-4e01-9b31-31bc06fc2fa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-52f3fd-tg7rl" Oct 30 16:52:52.186337 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:52.186310 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a11e515d-f825-4e01-9b31-31bc06fc2fa5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-52f3fd-tg7rl\" (UID: \"a11e515d-f825-4e01-9b31-31bc06fc2fa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-52f3fd-tg7rl" Oct 30 16:52:52.209946 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:52.209916 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-855th\" (UniqueName: \"kubernetes.io/projected/a11e515d-f825-4e01-9b31-31bc06fc2fa5-kube-api-access-855th\") pod \"maas-api-revoke-keys-e2e-praxis-a-52f3fd-tg7rl\" (UID: \"a11e515d-f825-4e01-9b31-31bc06fc2fa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-52f3fd-tg7rl" Oct 30 16:52:52.298765 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:52.298683 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-52f3fd-tg7rl" Oct 30 16:52:53.197551 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:53.197516 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-52f3fd-tg7rl"] Oct 30 16:52:53.745859 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:53.745603 4493 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:52:53.745859 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:53.745659 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8" podUID="e2c02109-c263-42aa-bdad-a995f3577d99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:53.922809 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:53.916262 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" podUID="b5abe703-fee0-4185-aa22-88d1db06392b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:54.048809 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:54.048111 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-52f3fd-tg7rl" event={"ID":"a11e515d-f825-4e01-9b31-31bc06fc2fa5","Type":"ContainerStarted","Data":"d65c5456cce38c77c18d9e6d9c1f182bff25b75668bfd2f133e60ecf5deb2c86"} Oct 30 16:52:54.086712 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:54.086672 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" podUID="e90f43d3-f5eb-45ab-af1e-340d0ce18966" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:54.124812 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:54.088033 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" podUID="f3e1c8ff-fcc3-4a1f-9730-c617d57b5193" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:54.124812 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:54.121660 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" podUID="e76172b1-659d-45bd-91c6-1a0ce4062335" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:54.124812 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:54.121922 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" podUID="14805045-a727-41ba-8bd0-100d41f9f27d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 16:52:54.124812 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:54.122093 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" podUID="80b9bd9a-30d6-45ff-b318-26fcdc608e5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 16:52:54.193819 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:54.182979 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" podUID="e43738f3-d55f-4c0a-a364-bf2ac76605c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:54.358021 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:54.357918 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" podUID="18b72648-f523-405e-b5cd-1266ff13f8f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:54.379275 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:54.379237 4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" podUID="14805045-a727-41ba-8bd0-100d41f9f27d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:54.383583 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:54.383548 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" podUID="08baad08-cd72-4305-b92d-c11a84ef276f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:54.454260 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:54.454219 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" podUID="0eadc5d4-98c3-4bf2-b3c2-f73fe249182f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:54.514186 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:54.504466 4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" podUID="b5abe703-fee0-4185-aa22-88d1db06392b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": context deadline exceeded" Oct 30 16:52:54.544820 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:54.542773 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-56v9s_d3a1c417-2698-4462-b7ff-1d2e66d7a10f/ovn-acl-logging/0.log" Oct 30 16:52:54.548807 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:54.548201 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-56v9s_d3a1c417-2698-4462-b7ff-1d2e66d7a10f/ovn-acl-logging/0.log" Oct 30 16:52:54.596041 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:54.594745 4493 scope.go:122] "RemoveContainer" containerID="94e66a1c53d783955c301187030fb050970346688fb0dc9d6860a7488bb7358d" Oct 30 16:52:54.641698 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:54.641316 4493 scope.go:122] "RemoveContainer" containerID="3cc97ce2593e0967559d03442e739ec10bcd362d7f42ca7f23606aa70c0e375e" Oct 30 16:52:54.719099 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:54.718606 4493 scope.go:122] "RemoveContainer" containerID="60f8d1c80965dc9cd5f4e9f056f5dd2991d09bac5991e5d29fc256597cae4fe4" Oct 30 16:52:54.780109 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:54.780006 4493 scope.go:122] "RemoveContainer" containerID="286ae5724b19f893f6f9880bccf62a763caa00dd1084660f32c384efc918256d" Oct 30 16:52:54.918039 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:54.917990 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" podUID="b5abe703-fee0-4185-aa22-88d1db06392b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Oct 30 16:52:54.922473 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:54.921035 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" Oct 30 16:52:55.070241 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:55.069565 4493 generic.go:356] "Generic (PLEG): container finished" podID="a11e515d-f825-4e01-9b31-31bc06fc2fa5" containerID="6948ebcb153ac6b2fdda4dc668d4e58105654d92d5208842b48ce0d28db5e3a3" exitCode=0 Oct 30 16:52:55.070241 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:55.069823 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-52f3fd-tg7rl" event={"ID":"a11e515d-f825-4e01-9b31-31bc06fc2fa5","Type":"ContainerDied","Data":"6948ebcb153ac6b2fdda4dc668d4e58105654d92d5208842b48ce0d28db5e3a3"} Oct 30 16:52:55.118099 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:55.114295 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" podUID="e90f43d3-f5eb-45ab-af1e-340d0ce18966" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:55.118099 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:55.115837 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" Oct 30 16:52:55.121004 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:55.120249 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" Oct 30 16:52:55.136720 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:55.121723 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" podUID="f3e1c8ff-fcc3-4a1f-9730-c617d57b5193" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:55.136720 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:55.122042 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" podUID="14805045-a727-41ba-8bd0-100d41f9f27d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:55.136720 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:55.122282 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" podUID="80b9bd9a-30d6-45ff-b318-26fcdc608e5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:55.136720 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:55.127225 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" Oct 30 16:52:55.136720 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:55.127278 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" Oct 30 16:52:55.136720 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:55.127314 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" Oct 30 16:52:55.304495 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:55.299152 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" Oct 30 16:52:55.423817 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:55.397164 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" Oct 30 16:52:55.553824 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:55.551894 4493 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:52:55.553824 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:55.551963 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw" podUID="92c1d749-8598-4a7b-b840-042533ac0bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:55.883940 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:55.883900 4493 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:52:55.884274 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:55.884242 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp" podUID="8c88b7e3-1f93-456a-87ed-4894a1de4b6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:56.133805 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:56.123858 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" podUID="14805045-a727-41ba-8bd0-100d41f9f27d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 16:52:56.169346 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:56.157742 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" podUID="e90f43d3-f5eb-45ab-af1e-340d0ce18966" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:56.295842 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:56.295712 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" podUID="18b72648-f523-405e-b5cd-1266ff13f8f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:56.372922 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:56.372868 4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" podUID="e90f43d3-f5eb-45ab-af1e-340d0ce18966" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:56.446485 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:56.446396 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" podUID="0eadc5d4-98c3-4bf2-b3c2-f73fe249182f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:56.452002 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:56.451977 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" Oct 30 16:52:56.604261 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:56.604233 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" Oct 30 16:52:57.060402 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:57.060368 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-52f3fd-tg7rl" Oct 30 16:52:57.166856 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:57.163415 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a11e515d-f825-4e01-9b31-31bc06fc2fa5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a11e515d-f825-4e01-9b31-31bc06fc2fa5-maas-api-xks-ca\") pod \"a11e515d-f825-4e01-9b31-31bc06fc2fa5\" (UID: \"a11e515d-f825-4e01-9b31-31bc06fc2fa5\") " Oct 30 16:52:57.166856 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:57.163468 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a11e515d-f825-4e01-9b31-31bc06fc2fa5-kube-api-access-855th\" (UniqueName: \"kubernetes.io/projected/a11e515d-f825-4e01-9b31-31bc06fc2fa5-kube-api-access-855th\") pod \"a11e515d-f825-4e01-9b31-31bc06fc2fa5\" (UID: \"a11e515d-f825-4e01-9b31-31bc06fc2fa5\") " Oct 30 16:52:57.166856 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:57.163523 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a11e515d-f825-4e01-9b31-31bc06fc2fa5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a11e515d-f825-4e01-9b31-31bc06fc2fa5-maas-api-openshift-ca\") pod \"a11e515d-f825-4e01-9b31-31bc06fc2fa5\" (UID: \"a11e515d-f825-4e01-9b31-31bc06fc2fa5\") " Oct 30 16:52:57.174198 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:57.169297 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a11e515d-f825-4e01-9b31-31bc06fc2fa5-maas-api-xks-ca" pod "a11e515d-f825-4e01-9b31-31bc06fc2fa5" (UID: "a11e515d-f825-4e01-9b31-31bc06fc2fa5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:52:57.174198 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:57.169378 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a11e515d-f825-4e01-9b31-31bc06fc2fa5-maas-api-openshift-ca" pod "a11e515d-f825-4e01-9b31-31bc06fc2fa5" (UID: "a11e515d-f825-4e01-9b31-31bc06fc2fa5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:52:57.174198 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:57.172110 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a11e515d-f825-4e01-9b31-31bc06fc2fa5-kube-api-access-855th" pod "a11e515d-f825-4e01-9b31-31bc06fc2fa5" (UID: "a11e515d-f825-4e01-9b31-31bc06fc2fa5"). InnerVolumeSpecName "kube-api-access-855th". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:52:57.174198 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:57.173613 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-52f3fd-tg7rl" Oct 30 16:52:57.174198 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:57.173648 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-52f3fd-tg7rl" event={"ID":"a11e515d-f825-4e01-9b31-31bc06fc2fa5","Type":"ContainerDied","Data":"d65c5456cce38c77c18d9e6d9c1f182bff25b75668bfd2f133e60ecf5deb2c86"} Oct 30 16:52:57.174198 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:57.173700 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d65c5456cce38c77c18d9e6d9c1f182bff25b75668bfd2f133e60ecf5deb2c86" Oct 30 16:52:57.265384 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:57.265355 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a11e515d-f825-4e01-9b31-31bc06fc2fa5-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:52:57.265384 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:57.265379 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-855th\" (UniqueName: \"kubernetes.io/projected/a11e515d-f825-4e01-9b31-31bc06fc2fa5-kube-api-access-855th\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:52:57.265384 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:52:57.265391 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a11e515d-f825-4e01-9b31-31bc06fc2fa5-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:53:00.341936 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:53:00.341899 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 16:53:03.919933 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:03.919898 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-624df655-smltd"] Oct 30 16:53:03.920715 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:03.920691 4493 cpu_manager.go:414] "containerMap: removing container" podUID="a11e515d-f825-4e01-9b31-31bc06fc2fa5" containerName="revoke-keys" Oct 30 16:53:03.920715 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:03.920715 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a11e515d-f825-4e01-9b31-31bc06fc2fa5" containerName="revoke-keys" Oct 30 16:53:03.920907 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:03.920868 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="a11e515d-f825-4e01-9b31-31bc06fc2fa5" containerName="revoke-keys" Oct 30 16:53:03.924158 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:03.924136 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-624df655-smltd" Oct 30 16:53:03.945830 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:03.945808 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-624df655-smltd"] Oct 30 16:53:03.954829 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:03.952408 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:53:03.954829 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:03.952444 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:53:03.954829 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:03.952476 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:53:03.954829 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:03.952557 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 16:53:03.960265 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:03.960238 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f625b78-72c7-434e-ada0-07b6c1a9ccea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-624df655-smltd\" (UID: \"0f625b78-72c7-434e-ada0-07b6c1a9ccea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-624df655-smltd" Oct 30 16:53:03.960407 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:03.960386 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpkwm\" (UniqueName: \"kubernetes.io/projected/0f625b78-72c7-434e-ada0-07b6c1a9ccea-kube-api-access-fpkwm\") pod \"maas-api-revoke-sub-624df655-smltd\" (UID: \"0f625b78-72c7-434e-ada0-07b6c1a9ccea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-624df655-smltd" Oct 30 16:53:03.960473 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:03.960461 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f625b78-72c7-434e-ada0-07b6c1a9ccea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-624df655-smltd\" (UID: \"0f625b78-72c7-434e-ada0-07b6c1a9ccea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-624df655-smltd" Oct 30 16:53:04.062023 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:04.061988 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f625b78-72c7-434e-ada0-07b6c1a9ccea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-624df655-smltd\" (UID: \"0f625b78-72c7-434e-ada0-07b6c1a9ccea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-624df655-smltd" Oct 30 16:53:04.062228 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:04.062114 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpkwm\" (UniqueName: \"kubernetes.io/projected/0f625b78-72c7-434e-ada0-07b6c1a9ccea-kube-api-access-fpkwm\") pod \"maas-api-revoke-sub-624df655-smltd\" (UID: \"0f625b78-72c7-434e-ada0-07b6c1a9ccea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-624df655-smltd" Oct 30 16:53:04.062228 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:04.062202 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f625b78-72c7-434e-ada0-07b6c1a9ccea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-624df655-smltd\" (UID: \"0f625b78-72c7-434e-ada0-07b6c1a9ccea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-624df655-smltd" Oct 30 16:53:04.062717 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:04.062694 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f625b78-72c7-434e-ada0-07b6c1a9ccea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-624df655-smltd\" (UID: \"0f625b78-72c7-434e-ada0-07b6c1a9ccea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-624df655-smltd" Oct 30 16:53:04.064927 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:04.064897 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f625b78-72c7-434e-ada0-07b6c1a9ccea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-624df655-smltd\" (UID: \"0f625b78-72c7-434e-ada0-07b6c1a9ccea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-624df655-smltd" Oct 30 16:53:04.085856 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:04.085829 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpkwm\" (UniqueName: \"kubernetes.io/projected/0f625b78-72c7-434e-ada0-07b6c1a9ccea-kube-api-access-fpkwm\") pod \"maas-api-revoke-sub-624df655-smltd\" (UID: \"0f625b78-72c7-434e-ada0-07b6c1a9ccea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-624df655-smltd" Oct 30 16:53:04.237490 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:04.237406 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-624df655-smltd" Oct 30 16:53:04.547152 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:04.547123 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-624df655-smltd"] Oct 30 16:53:04.547948 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:53:04.547923 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f625b78_72c7_434e_ada0_07b6c1a9ccea.slice/crio-05118737ebc91fc302bf3f7edf6aab029f9cefee7621fe17daafe5402ea3a905 WatchSource:0}: Error finding container 05118737ebc91fc302bf3f7edf6aab029f9cefee7621fe17daafe5402ea3a905: Status 404 returned error can't find the container with id 05118737ebc91fc302bf3f7edf6aab029f9cefee7621fe17daafe5402ea3a905 Oct 30 16:53:05.274500 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:05.274467 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-624df655-smltd" event={"ID":"0f625b78-72c7-434e-ada0-07b6c1a9ccea","Type":"ContainerStarted","Data":"66303bba1c7588d7534fbf8b6b9566a9ef094e75f33d0c858ee9708fa27714d1"} Oct 30 16:53:05.275025 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:05.274522 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-624df655-smltd" event={"ID":"0f625b78-72c7-434e-ada0-07b6c1a9ccea","Type":"ContainerStarted","Data":"05118737ebc91fc302bf3f7edf6aab029f9cefee7621fe17daafe5402ea3a905"} Oct 30 16:53:05.316634 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:05.316586 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-624df655-smltd" podStartSLOduration=2.316572366 podStartE2EDuration="2.316572366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:53:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:53:05.314661448 +0000 UTC m=+1812.541836548" watchObservedRunningTime="2026-10-30 16:53:05.316572366 +0000 UTC m=+1812.543747510" Oct 30 16:53:06.288845 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:06.288805 4493 generic.go:356] "Generic (PLEG): container finished" podID="0f625b78-72c7-434e-ada0-07b6c1a9ccea" containerID="66303bba1c7588d7534fbf8b6b9566a9ef094e75f33d0c858ee9708fa27714d1" exitCode=0 Oct 30 16:53:06.289693 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:06.289659 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-624df655-smltd" event={"ID":"0f625b78-72c7-434e-ada0-07b6c1a9ccea","Type":"ContainerDied","Data":"66303bba1c7588d7534fbf8b6b9566a9ef094e75f33d0c858ee9708fa27714d1"} Oct 30 16:53:07.463619 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:07.463595 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-624df655-smltd" Oct 30 16:53:07.505174 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:07.505149 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f625b78-72c7-434e-ada0-07b6c1a9ccea-kube-api-access-fpkwm\" (UniqueName: \"kubernetes.io/projected/0f625b78-72c7-434e-ada0-07b6c1a9ccea-kube-api-access-fpkwm\") pod \"0f625b78-72c7-434e-ada0-07b6c1a9ccea\" (UID: \"0f625b78-72c7-434e-ada0-07b6c1a9ccea\") " Oct 30 16:53:07.505316 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:07.505286 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f625b78-72c7-434e-ada0-07b6c1a9ccea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f625b78-72c7-434e-ada0-07b6c1a9ccea-maas-api-openshift-ca\") pod \"0f625b78-72c7-434e-ada0-07b6c1a9ccea\" (UID: \"0f625b78-72c7-434e-ada0-07b6c1a9ccea\") " Oct 30 16:53:07.505463 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:07.505443 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f625b78-72c7-434e-ada0-07b6c1a9ccea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f625b78-72c7-434e-ada0-07b6c1a9ccea-maas-api-xks-ca\") pod \"0f625b78-72c7-434e-ada0-07b6c1a9ccea\" (UID: \"0f625b78-72c7-434e-ada0-07b6c1a9ccea\") " Oct 30 16:53:07.505698 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:07.505675 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f625b78-72c7-434e-ada0-07b6c1a9ccea-maas-api-openshift-ca" pod "0f625b78-72c7-434e-ada0-07b6c1a9ccea" (UID: "0f625b78-72c7-434e-ada0-07b6c1a9ccea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:53:07.507949 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:07.507396 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f625b78-72c7-434e-ada0-07b6c1a9ccea-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:53:07.507949 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:07.507885 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f625b78-72c7-434e-ada0-07b6c1a9ccea-kube-api-access-fpkwm" pod "0f625b78-72c7-434e-ada0-07b6c1a9ccea" (UID: "0f625b78-72c7-434e-ada0-07b6c1a9ccea"). InnerVolumeSpecName "kube-api-access-fpkwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:53:07.509631 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:07.509600 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f625b78-72c7-434e-ada0-07b6c1a9ccea-maas-api-xks-ca" pod "0f625b78-72c7-434e-ada0-07b6c1a9ccea" (UID: "0f625b78-72c7-434e-ada0-07b6c1a9ccea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:53:07.608630 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:07.608544 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f625b78-72c7-434e-ada0-07b6c1a9ccea-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:53:07.608630 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:07.608578 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpkwm\" (UniqueName: \"kubernetes.io/projected/0f625b78-72c7-434e-ada0-07b6c1a9ccea-kube-api-access-fpkwm\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:53:08.316967 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:08.316147 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-624df655-smltd" event={"ID":"0f625b78-72c7-434e-ada0-07b6c1a9ccea","Type":"ContainerDied","Data":"05118737ebc91fc302bf3f7edf6aab029f9cefee7621fe17daafe5402ea3a905"} Oct 30 16:53:08.316967 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:08.316207 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05118737ebc91fc302bf3f7edf6aab029f9cefee7621fe17daafe5402ea3a905" Oct 30 16:53:08.316967 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:08.316271 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-624df655-smltd" Oct 30 16:53:15.866166 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:15.866132 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd"] Oct 30 16:53:15.867001 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:15.866984 4493 cpu_manager.go:414] "containerMap: removing container" podUID="0f625b78-72c7-434e-ada0-07b6c1a9ccea" containerName="revoke-keys" Oct 30 16:53:15.867064 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:15.867004 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f625b78-72c7-434e-ada0-07b6c1a9ccea" containerName="revoke-keys" Oct 30 16:53:15.867179 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:15.867169 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f625b78-72c7-434e-ada0-07b6c1a9ccea" containerName="revoke-keys" Oct 30 16:53:15.871503 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:15.871480 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd" Oct 30 16:53:15.906051 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:15.906022 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd"] Oct 30 16:53:16.035764 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:16.035727 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/948c9256-224e-472e-8ccf-5f6d761b5e0c-istio-podinfo\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd\" (UID: \"948c9256-224e-472e-8ccf-5f6d761b5e0c\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd" Oct 30 16:53:16.035963 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:16.035775 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chff7\" (UniqueName: \"kubernetes.io/projected/948c9256-224e-472e-8ccf-5f6d761b5e0c-kube-api-access-chff7\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd\" (UID: \"948c9256-224e-472e-8ccf-5f6d761b5e0c\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd" Oct 30 16:53:16.035963 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:16.035837 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/948c9256-224e-472e-8ccf-5f6d761b5e0c-istio-envoy\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd\" (UID: \"948c9256-224e-472e-8ccf-5f6d761b5e0c\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd" Oct 30 16:53:16.035963 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:16.035933 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/948c9256-224e-472e-8ccf-5f6d761b5e0c-credential-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd\" (UID: \"948c9256-224e-472e-8ccf-5f6d761b5e0c\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd" Oct 30 16:53:16.036079 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:16.035970 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/948c9256-224e-472e-8ccf-5f6d761b5e0c-istio-data\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd\" (UID: \"948c9256-224e-472e-8ccf-5f6d761b5e0c\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd" Oct 30 16:53:16.036142 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:16.036081 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/948c9256-224e-472e-8ccf-5f6d761b5e0c-workload-certs\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd\" (UID: \"948c9256-224e-472e-8ccf-5f6d761b5e0c\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd" Oct 30 16:53:16.036142 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:16.036101 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/948c9256-224e-472e-8ccf-5f6d761b5e0c-istiod-ca-cert\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd\" (UID: \"948c9256-224e-472e-8ccf-5f6d761b5e0c\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd" Oct 30 16:53:16.036301 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:16.036277 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/948c9256-224e-472e-8ccf-5f6d761b5e0c-istio-token\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd\" (UID: \"948c9256-224e-472e-8ccf-5f6d761b5e0c\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd" Oct 30 16:53:16.036365 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:16.036350 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/948c9256-224e-472e-8ccf-5f6d761b5e0c-workload-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd\" (UID: \"948c9256-224e-472e-8ccf-5f6d761b5e0c\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd" Oct 30 16:53:16.138075 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:16.137996 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/948c9256-224e-472e-8ccf-5f6d761b5e0c-workload-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd\" (UID: \"948c9256-224e-472e-8ccf-5f6d761b5e0c\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd" Oct 30 16:53:16.138075 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:16.138052 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/948c9256-224e-472e-8ccf-5f6d761b5e0c-istio-podinfo\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd\" (UID: \"948c9256-224e-472e-8ccf-5f6d761b5e0c\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd" Oct 30 16:53:16.138075 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:16.138074 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chff7\" (UniqueName: \"kubernetes.io/projected/948c9256-224e-472e-8ccf-5f6d761b5e0c-kube-api-access-chff7\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd\" (UID: \"948c9256-224e-472e-8ccf-5f6d761b5e0c\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd" Oct 30 16:53:16.138374 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:16.138090 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/948c9256-224e-472e-8ccf-5f6d761b5e0c-istio-envoy\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd\" (UID: \"948c9256-224e-472e-8ccf-5f6d761b5e0c\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd" Oct 30 16:53:16.138374 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:16.138135 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/948c9256-224e-472e-8ccf-5f6d761b5e0c-credential-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd\" (UID: \"948c9256-224e-472e-8ccf-5f6d761b5e0c\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd" Oct 30 16:53:16.138374 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:16.138178 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/948c9256-224e-472e-8ccf-5f6d761b5e0c-istio-data\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd\" (UID: \"948c9256-224e-472e-8ccf-5f6d761b5e0c\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd" Oct 30 16:53:16.138374 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:16.138265 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/948c9256-224e-472e-8ccf-5f6d761b5e0c-workload-certs\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd\" (UID: \"948c9256-224e-472e-8ccf-5f6d761b5e0c\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd" Oct 30 16:53:16.138374 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:16.138292 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/948c9256-224e-472e-8ccf-5f6d761b5e0c-istiod-ca-cert\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd\" (UID: \"948c9256-224e-472e-8ccf-5f6d761b5e0c\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd" Oct 30 16:53:16.138645 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:16.138395 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/948c9256-224e-472e-8ccf-5f6d761b5e0c-istio-token\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd\" (UID: \"948c9256-224e-472e-8ccf-5f6d761b5e0c\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd" Oct 30 16:53:16.138645 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:16.138427 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/948c9256-224e-472e-8ccf-5f6d761b5e0c-workload-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd\" (UID: \"948c9256-224e-472e-8ccf-5f6d761b5e0c\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd" Oct 30 16:53:16.138939 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:16.138813 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/948c9256-224e-472e-8ccf-5f6d761b5e0c-credential-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd\" (UID: \"948c9256-224e-472e-8ccf-5f6d761b5e0c\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd" Oct 30 16:53:16.139084 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:16.139063 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/948c9256-224e-472e-8ccf-5f6d761b5e0c-workload-certs\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd\" (UID: \"948c9256-224e-472e-8ccf-5f6d761b5e0c\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd" Oct 30 16:53:16.139149 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:16.139072 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/948c9256-224e-472e-8ccf-5f6d761b5e0c-istiod-ca-cert\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd\" (UID: \"948c9256-224e-472e-8ccf-5f6d761b5e0c\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd" Oct 30 16:53:16.139273 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:16.139241 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/948c9256-224e-472e-8ccf-5f6d761b5e0c-istio-data\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd\" (UID: \"948c9256-224e-472e-8ccf-5f6d761b5e0c\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd" Oct 30 16:53:16.140911 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:16.140883 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/948c9256-224e-472e-8ccf-5f6d761b5e0c-istio-envoy\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd\" (UID: \"948c9256-224e-472e-8ccf-5f6d761b5e0c\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd" Oct 30 16:53:16.141004 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:16.140886 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/948c9256-224e-472e-8ccf-5f6d761b5e0c-istio-podinfo\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd\" (UID: \"948c9256-224e-472e-8ccf-5f6d761b5e0c\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd" Oct 30 16:53:16.161818 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:16.161773 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/948c9256-224e-472e-8ccf-5f6d761b5e0c-istio-token\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd\" (UID: \"948c9256-224e-472e-8ccf-5f6d761b5e0c\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd" Oct 30 16:53:16.162001 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:16.161984 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chff7\" (UniqueName: \"kubernetes.io/projected/948c9256-224e-472e-8ccf-5f6d761b5e0c-kube-api-access-chff7\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd\" (UID: \"948c9256-224e-472e-8ccf-5f6d761b5e0c\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd" Oct 30 16:53:16.186765 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:16.186743 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd" Oct 30 16:53:16.521511 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:16.521482 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd"] Oct 30 16:53:16.522239 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:53:16.522212 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod948c9256_224e_472e_8ccf_5f6d761b5e0c.slice/crio-cb0e9104c15fe47243eb83ae544cb5ded24b45a80ac5f764d39b67248533dcca WatchSource:0}: Error finding container cb0e9104c15fe47243eb83ae544cb5ded24b45a80ac5f764d39b67248533dcca: Status 404 returned error can't find the container with id cb0e9104c15fe47243eb83ae544cb5ded24b45a80ac5f764d39b67248533dcca Oct 30 16:53:16.525525 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:16.525488 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:53:16.525629 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:16.525554 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:53:16.525629 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:16.525585 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:53:17.476807 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:17.475335 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd" event={"ID":"948c9256-224e-472e-8ccf-5f6d761b5e0c","Type":"ContainerStarted","Data":"20026c763422eeba9d8b3b052bf64a433460a11aa284fce30f7d4d83533f6ff3"} Oct 30 16:53:17.476807 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:17.475400 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd" event={"ID":"948c9256-224e-472e-8ccf-5f6d761b5e0c","Type":"ContainerStarted","Data":"cb0e9104c15fe47243eb83ae544cb5ded24b45a80ac5f764d39b67248533dcca"} Oct 30 16:53:17.570209 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:17.570144 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd" podStartSLOduration=2.570122307 podStartE2EDuration="2.570122307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:53:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:53:17.538254677 +0000 UTC m=+1824.765429814" watchObservedRunningTime="2026-10-30 16:53:17.570122307 +0000 UTC m=+1824.797297408" Oct 30 16:53:18.192025 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:18.191989 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd" Oct 30 16:53:18.195490 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:18.195462 4493 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Oct 30 16:53:18.195638 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:18.195515 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd" podUID="948c9256-224e-472e-8ccf-5f6d761b5e0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Oct 30 16:53:19.187707 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:19.187673 4493 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Oct 30 16:53:19.188184 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:19.187725 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd" podUID="948c9256-224e-472e-8ccf-5f6d761b5e0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Oct 30 16:53:20.187262 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:20.187226 4493 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Oct 30 16:53:20.187450 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:20.187298 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd" podUID="948c9256-224e-472e-8ccf-5f6d761b5e0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Oct 30 16:53:21.187839 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:21.187771 4493 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Oct 30 16:53:21.188346 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:21.187852 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd" podUID="948c9256-224e-472e-8ccf-5f6d761b5e0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Oct 30 16:53:22.187652 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:22.187605 4493 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Oct 30 16:53:22.187843 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:22.187676 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd" podUID="948c9256-224e-472e-8ccf-5f6d761b5e0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Oct 30 16:53:22.518859 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:22.515259 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-52f3fd-tg7rl"] Oct 30 16:53:22.557516 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:22.557480 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-52f3fd-tg7rl"] Oct 30 16:53:22.938561 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:22.938521 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a11e515d-f825-4e01-9b31-31bc06fc2fa5" path="/var/lib/kubelet/pods/a11e515d-f825-4e01-9b31-31bc06fc2fa5/volumes" Oct 30 16:53:23.188138 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:23.188097 4493 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Oct 30 16:53:23.188330 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:23.188197 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd" podUID="948c9256-224e-472e-8ccf-5f6d761b5e0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Oct 30 16:53:24.377722 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:24.377688 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd" Oct 30 16:53:24.378244 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:24.378210 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd" Oct 30 16:53:24.379030 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:24.379011 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd" Oct 30 16:53:29.064739 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:29.064698 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 16:53:30.419810 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:53:30.409974 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 16:53:31.086810 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:31.065728 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4a8ab1-vwt8t"] Oct 30 16:53:31.153102 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:31.153069 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4a8ab1-vwt8t" Oct 30 16:53:31.177997 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:31.159559 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4a8ab1-vwt8t"] Oct 30 16:53:31.201273 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:31.201217 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:53:31.262888 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:31.228341 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 16:53:31.262888 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:31.228513 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:53:31.262888 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:31.228613 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:53:31.265060 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:31.265035 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa0be2de-94a8-45dd-a889-4724f87af42a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-4a8ab1-vwt8t\" (UID: \"fa0be2de-94a8-45dd-a889-4724f87af42a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4a8ab1-vwt8t" Oct 30 16:53:31.265295 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:31.265275 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9g2j\" (UniqueName: \"kubernetes.io/projected/fa0be2de-94a8-45dd-a889-4724f87af42a-kube-api-access-n9g2j\") pod \"maas-api-revoke-keys-e2e-praxis-b-4a8ab1-vwt8t\" (UID: \"fa0be2de-94a8-45dd-a889-4724f87af42a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4a8ab1-vwt8t" Oct 30 16:53:31.265420 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:31.265405 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa0be2de-94a8-45dd-a889-4724f87af42a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-4a8ab1-vwt8t\" (UID: \"fa0be2de-94a8-45dd-a889-4724f87af42a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4a8ab1-vwt8t" Oct 30 16:53:31.377140 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:31.377047 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9g2j\" (UniqueName: \"kubernetes.io/projected/fa0be2de-94a8-45dd-a889-4724f87af42a-kube-api-access-n9g2j\") pod \"maas-api-revoke-keys-e2e-praxis-b-4a8ab1-vwt8t\" (UID: \"fa0be2de-94a8-45dd-a889-4724f87af42a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4a8ab1-vwt8t" Oct 30 16:53:31.387931 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:31.387886 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa0be2de-94a8-45dd-a889-4724f87af42a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-4a8ab1-vwt8t\" (UID: \"fa0be2de-94a8-45dd-a889-4724f87af42a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4a8ab1-vwt8t" Oct 30 16:53:31.388378 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:31.388354 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa0be2de-94a8-45dd-a889-4724f87af42a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-4a8ab1-vwt8t\" (UID: \"fa0be2de-94a8-45dd-a889-4724f87af42a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4a8ab1-vwt8t" Oct 30 16:53:31.389660 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:31.389637 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa0be2de-94a8-45dd-a889-4724f87af42a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-4a8ab1-vwt8t\" (UID: \"fa0be2de-94a8-45dd-a889-4724f87af42a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4a8ab1-vwt8t" Oct 30 16:53:31.432810 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:31.431219 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa0be2de-94a8-45dd-a889-4724f87af42a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-4a8ab1-vwt8t\" (UID: \"fa0be2de-94a8-45dd-a889-4724f87af42a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4a8ab1-vwt8t" Oct 30 16:53:31.444599 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:31.444566 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9g2j\" (UniqueName: \"kubernetes.io/projected/fa0be2de-94a8-45dd-a889-4724f87af42a-kube-api-access-n9g2j\") pod \"maas-api-revoke-keys-e2e-praxis-b-4a8ab1-vwt8t\" (UID: \"fa0be2de-94a8-45dd-a889-4724f87af42a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4a8ab1-vwt8t" Oct 30 16:53:31.631229 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:31.631135 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4a8ab1-vwt8t" Oct 30 16:53:31.962395 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:31.962366 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4a8ab1-vwt8t"] Oct 30 16:53:31.963509 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:53:31.963482 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa0be2de_94a8_45dd_a889_4724f87af42a.slice/crio-d23e2a7e847f3bebde32a46f90af813f1e061baeeabbc2bff2a6dba19e5d12bc WatchSource:0}: Error finding container d23e2a7e847f3bebde32a46f90af813f1e061baeeabbc2bff2a6dba19e5d12bc: Status 404 returned error can't find the container with id d23e2a7e847f3bebde32a46f90af813f1e061baeeabbc2bff2a6dba19e5d12bc Oct 30 16:53:32.857436 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:32.857402 4493 generic.go:356] "Generic (PLEG): container finished" podID="fa0be2de-94a8-45dd-a889-4724f87af42a" containerID="666ecdb459cd1a0cb5c487b2898b809c3deb46b0a3906e96f35969fadae0d5e1" exitCode=0 Oct 30 16:53:32.857856 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:32.857485 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4a8ab1-vwt8t" event={"ID":"fa0be2de-94a8-45dd-a889-4724f87af42a","Type":"ContainerDied","Data":"666ecdb459cd1a0cb5c487b2898b809c3deb46b0a3906e96f35969fadae0d5e1"} Oct 30 16:53:32.857856 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:32.857537 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4a8ab1-vwt8t" event={"ID":"fa0be2de-94a8-45dd-a889-4724f87af42a","Type":"ContainerStarted","Data":"d23e2a7e847f3bebde32a46f90af813f1e061baeeabbc2bff2a6dba19e5d12bc"} Oct 30 16:53:34.026379 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:34.026349 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4a8ab1-vwt8t" Oct 30 16:53:34.141105 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:34.140935 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa0be2de-94a8-45dd-a889-4724f87af42a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa0be2de-94a8-45dd-a889-4724f87af42a-maas-api-openshift-ca\") pod \"fa0be2de-94a8-45dd-a889-4724f87af42a\" (UID: \"fa0be2de-94a8-45dd-a889-4724f87af42a\") " Oct 30 16:53:34.141105 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:34.141016 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa0be2de-94a8-45dd-a889-4724f87af42a-kube-api-access-n9g2j\" (UniqueName: \"kubernetes.io/projected/fa0be2de-94a8-45dd-a889-4724f87af42a-kube-api-access-n9g2j\") pod \"fa0be2de-94a8-45dd-a889-4724f87af42a\" (UID: \"fa0be2de-94a8-45dd-a889-4724f87af42a\") " Oct 30 16:53:34.141830 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:34.141754 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa0be2de-94a8-45dd-a889-4724f87af42a-maas-api-openshift-ca" pod "fa0be2de-94a8-45dd-a889-4724f87af42a" (UID: "fa0be2de-94a8-45dd-a889-4724f87af42a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:53:34.141830 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:34.141770 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa0be2de-94a8-45dd-a889-4724f87af42a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa0be2de-94a8-45dd-a889-4724f87af42a-maas-api-xks-ca\") pod \"fa0be2de-94a8-45dd-a889-4724f87af42a\" (UID: \"fa0be2de-94a8-45dd-a889-4724f87af42a\") " Oct 30 16:53:34.142621 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:34.142578 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa0be2de-94a8-45dd-a889-4724f87af42a-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:53:34.146458 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:34.146340 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa0be2de-94a8-45dd-a889-4724f87af42a-maas-api-xks-ca" pod "fa0be2de-94a8-45dd-a889-4724f87af42a" (UID: "fa0be2de-94a8-45dd-a889-4724f87af42a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:53:34.146458 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:34.146382 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa0be2de-94a8-45dd-a889-4724f87af42a-kube-api-access-n9g2j" pod "fa0be2de-94a8-45dd-a889-4724f87af42a" (UID: "fa0be2de-94a8-45dd-a889-4724f87af42a"). InnerVolumeSpecName "kube-api-access-n9g2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:53:34.244219 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:34.244181 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa0be2de-94a8-45dd-a889-4724f87af42a-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:53:34.244219 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:34.244215 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9g2j\" (UniqueName: \"kubernetes.io/projected/fa0be2de-94a8-45dd-a889-4724f87af42a-kube-api-access-n9g2j\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:53:34.879329 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:34.879296 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4a8ab1-vwt8t" Oct 30 16:53:34.879566 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:34.879530 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4a8ab1-vwt8t" event={"ID":"fa0be2de-94a8-45dd-a889-4724f87af42a","Type":"ContainerDied","Data":"d23e2a7e847f3bebde32a46f90af813f1e061baeeabbc2bff2a6dba19e5d12bc"} Oct 30 16:53:34.880758 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:34.880729 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d23e2a7e847f3bebde32a46f90af813f1e061baeeabbc2bff2a6dba19e5d12bc" Oct 30 16:53:35.832826 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:35.832775 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr"] Oct 30 16:53:35.833300 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:35.833009 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr" podUID="3dbdb8f4-a39a-4880-bc1b-525787aa6120" containerName="istio-proxy" containerID="cri-o://b7eb285df4bf73fa42f5e66c470871bf662853edc34e327416c81862cd367993" gracePeriod=30 Oct 30 16:53:41.105023 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:41.104963 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr" Oct 30 16:53:41.265647 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:41.265618 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dbdb8f4-a39a-4880-bc1b-525787aa6120-istio-token\" (UniqueName: \"kubernetes.io/projected/3dbdb8f4-a39a-4880-bc1b-525787aa6120-istio-token\") pod \"3dbdb8f4-a39a-4880-bc1b-525787aa6120\" (UID: \"3dbdb8f4-a39a-4880-bc1b-525787aa6120\") " Oct 30 16:53:41.265871 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:41.265681 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dbdb8f4-a39a-4880-bc1b-525787aa6120-kube-api-access-zks2h\" (UniqueName: \"kubernetes.io/projected/3dbdb8f4-a39a-4880-bc1b-525787aa6120-kube-api-access-zks2h\") pod \"3dbdb8f4-a39a-4880-bc1b-525787aa6120\" (UID: \"3dbdb8f4-a39a-4880-bc1b-525787aa6120\") " Oct 30 16:53:41.265871 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:41.265711 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dbdb8f4-a39a-4880-bc1b-525787aa6120-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dbdb8f4-a39a-4880-bc1b-525787aa6120-istio-data\") pod \"3dbdb8f4-a39a-4880-bc1b-525787aa6120\" (UID: \"3dbdb8f4-a39a-4880-bc1b-525787aa6120\") " Oct 30 16:53:41.265871 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:41.265765 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3dbdb8f4-a39a-4880-bc1b-525787aa6120-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dbdb8f4-a39a-4880-bc1b-525787aa6120-istio-podinfo\") pod \"3dbdb8f4-a39a-4880-bc1b-525787aa6120\" (UID: \"3dbdb8f4-a39a-4880-bc1b-525787aa6120\") " Oct 30 16:53:41.265871 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:41.265867 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dbdb8f4-a39a-4880-bc1b-525787aa6120-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dbdb8f4-a39a-4880-bc1b-525787aa6120-istio-envoy\") pod \"3dbdb8f4-a39a-4880-bc1b-525787aa6120\" (UID: \"3dbdb8f4-a39a-4880-bc1b-525787aa6120\") " Oct 30 16:53:41.266126 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:41.266042 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3dbdb8f4-a39a-4880-bc1b-525787aa6120-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dbdb8f4-a39a-4880-bc1b-525787aa6120-istiod-ca-cert\") pod \"3dbdb8f4-a39a-4880-bc1b-525787aa6120\" (UID: \"3dbdb8f4-a39a-4880-bc1b-525787aa6120\") " Oct 30 16:53:41.266179 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:41.266129 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dbdb8f4-a39a-4880-bc1b-525787aa6120-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbdb8f4-a39a-4880-bc1b-525787aa6120-credential-socket\") pod \"3dbdb8f4-a39a-4880-bc1b-525787aa6120\" (UID: \"3dbdb8f4-a39a-4880-bc1b-525787aa6120\") " Oct 30 16:53:41.266179 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:41.266147 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dbdb8f4-a39a-4880-bc1b-525787aa6120-istio-data" pod "3dbdb8f4-a39a-4880-bc1b-525787aa6120" (UID: "3dbdb8f4-a39a-4880-bc1b-525787aa6120"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:53:41.266294 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:41.266160 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dbdb8f4-a39a-4880-bc1b-525787aa6120-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dbdb8f4-a39a-4880-bc1b-525787aa6120-workload-certs\") pod \"3dbdb8f4-a39a-4880-bc1b-525787aa6120\" (UID: \"3dbdb8f4-a39a-4880-bc1b-525787aa6120\") " Oct 30 16:53:41.266294 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:41.266245 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dbdb8f4-a39a-4880-bc1b-525787aa6120-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbdb8f4-a39a-4880-bc1b-525787aa6120-workload-socket\") pod \"3dbdb8f4-a39a-4880-bc1b-525787aa6120\" (UID: \"3dbdb8f4-a39a-4880-bc1b-525787aa6120\") " Oct 30 16:53:41.266404 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:41.266365 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dbdb8f4-a39a-4880-bc1b-525787aa6120-workload-certs" pod "3dbdb8f4-a39a-4880-bc1b-525787aa6120" (UID: "3dbdb8f4-a39a-4880-bc1b-525787aa6120"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:53:41.267060 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:41.266726 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dbdb8f4-a39a-4880-bc1b-525787aa6120-workload-socket" pod "3dbdb8f4-a39a-4880-bc1b-525787aa6120" (UID: "3dbdb8f4-a39a-4880-bc1b-525787aa6120"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:53:41.267060 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:41.266746 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dbdb8f4-a39a-4880-bc1b-525787aa6120-credential-socket" pod "3dbdb8f4-a39a-4880-bc1b-525787aa6120" (UID: "3dbdb8f4-a39a-4880-bc1b-525787aa6120"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:53:41.267060 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:41.266930 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dbdb8f4-a39a-4880-bc1b-525787aa6120-istiod-ca-cert" pod "3dbdb8f4-a39a-4880-bc1b-525787aa6120" (UID: "3dbdb8f4-a39a-4880-bc1b-525787aa6120"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:53:41.267060 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:41.266996 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbdb8f4-a39a-4880-bc1b-525787aa6120-credential-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:53:41.267060 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:41.267012 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dbdb8f4-a39a-4880-bc1b-525787aa6120-workload-certs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:53:41.267060 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:41.267022 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbdb8f4-a39a-4880-bc1b-525787aa6120-workload-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:53:41.267060 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:41.267036 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dbdb8f4-a39a-4880-bc1b-525787aa6120-istio-data\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:53:41.268193 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:41.268163 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dbdb8f4-a39a-4880-bc1b-525787aa6120-kube-api-access-zks2h" pod "3dbdb8f4-a39a-4880-bc1b-525787aa6120" (UID: "3dbdb8f4-a39a-4880-bc1b-525787aa6120"). InnerVolumeSpecName "kube-api-access-zks2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:53:41.268535 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:41.268514 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dbdb8f4-a39a-4880-bc1b-525787aa6120-istio-token" pod "3dbdb8f4-a39a-4880-bc1b-525787aa6120" (UID: "3dbdb8f4-a39a-4880-bc1b-525787aa6120"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:53:41.268654 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:41.268639 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3dbdb8f4-a39a-4880-bc1b-525787aa6120-istio-podinfo" pod "3dbdb8f4-a39a-4880-bc1b-525787aa6120" (UID: "3dbdb8f4-a39a-4880-bc1b-525787aa6120"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:53:41.268889 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:41.268861 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dbdb8f4-a39a-4880-bc1b-525787aa6120-istio-envoy" pod "3dbdb8f4-a39a-4880-bc1b-525787aa6120" (UID: "3dbdb8f4-a39a-4880-bc1b-525787aa6120"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:53:41.367952 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:41.367875 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dbdb8f4-a39a-4880-bc1b-525787aa6120-istiod-ca-cert\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:53:41.367952 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:41.367903 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dbdb8f4-a39a-4880-bc1b-525787aa6120-istio-token\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:53:41.367952 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:41.367913 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zks2h\" (UniqueName: \"kubernetes.io/projected/3dbdb8f4-a39a-4880-bc1b-525787aa6120-kube-api-access-zks2h\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:53:41.367952 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:41.367924 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dbdb8f4-a39a-4880-bc1b-525787aa6120-istio-podinfo\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:53:41.367952 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:41.367932 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dbdb8f4-a39a-4880-bc1b-525787aa6120-istio-envoy\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:53:41.942236 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:41.942201 4493 generic.go:356] "Generic (PLEG): container finished" podID="3dbdb8f4-a39a-4880-bc1b-525787aa6120" containerID="b7eb285df4bf73fa42f5e66c470871bf662853edc34e327416c81862cd367993" exitCode=0 Oct 30 16:53:41.942431 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:41.942251 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr" Oct 30 16:53:41.942431 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:41.942294 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr" event={"ID":"3dbdb8f4-a39a-4880-bc1b-525787aa6120","Type":"ContainerDied","Data":"b7eb285df4bf73fa42f5e66c470871bf662853edc34e327416c81862cd367993"} Oct 30 16:53:41.942431 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:41.942347 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr" event={"ID":"3dbdb8f4-a39a-4880-bc1b-525787aa6120","Type":"ContainerDied","Data":"bd11401bdfc77bf4189d8a1b36de0e88abde5e672b0fae35031a9c55e3af4375"} Oct 30 16:53:41.942431 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:41.942376 4493 scope.go:122] "RemoveContainer" containerID="b7eb285df4bf73fa42f5e66c470871bf662853edc34e327416c81862cd367993" Oct 30 16:53:41.954609 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:41.954423 4493 scope.go:122] "RemoveContainer" containerID="b7eb285df4bf73fa42f5e66c470871bf662853edc34e327416c81862cd367993" Oct 30 16:53:41.954717 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:53:41.954698 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7eb285df4bf73fa42f5e66c470871bf662853edc34e327416c81862cd367993\": container with ID starting with b7eb285df4bf73fa42f5e66c470871bf662853edc34e327416c81862cd367993 not found: ID does not exist" containerID="b7eb285df4bf73fa42f5e66c470871bf662853edc34e327416c81862cd367993" Oct 30 16:53:41.954761 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:41.954725 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7eb285df4bf73fa42f5e66c470871bf662853edc34e327416c81862cd367993"} err="failed to get container status \"b7eb285df4bf73fa42f5e66c470871bf662853edc34e327416c81862cd367993\": rpc error: code = NotFound desc = could not find container \"b7eb285df4bf73fa42f5e66c470871bf662853edc34e327416c81862cd367993\": container with ID starting with b7eb285df4bf73fa42f5e66c470871bf662853edc34e327416c81862cd367993 not found: ID does not exist" Oct 30 16:53:42.008304 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:42.008274 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr"] Oct 30 16:53:42.021112 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:42.021082 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qq5kr"] Oct 30 16:53:42.935183 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:42.935152 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3dbdb8f4-a39a-4880-bc1b-525787aa6120" path="/var/lib/kubelet/pods/3dbdb8f4-a39a-4880-bc1b-525787aa6120/volumes" Oct 30 16:53:47.243323 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:47.243175 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl"] Oct 30 16:53:47.243829 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:47.243449 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl" podUID="c933c090-d160-449b-a3e3-75aec1c2ce36" containerName="istio-proxy" containerID="cri-o://59ee73106ddcb99e1eabac289f0ffbed1eb433e7c4f5b90bc5dd6d83ead195e8" gracePeriod=30 Oct 30 16:53:52.547611 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:52.547572 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl" Oct 30 16:53:52.676198 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:52.676165 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c933c090-d160-449b-a3e3-75aec1c2ce36-istio-token\" (UniqueName: \"kubernetes.io/projected/c933c090-d160-449b-a3e3-75aec1c2ce36-istio-token\") pod \"c933c090-d160-449b-a3e3-75aec1c2ce36\" (UID: \"c933c090-d160-449b-a3e3-75aec1c2ce36\") " Oct 30 16:53:52.676396 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:52.676208 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c933c090-d160-449b-a3e3-75aec1c2ce36-kube-api-access-6jmfw\" (UniqueName: \"kubernetes.io/projected/c933c090-d160-449b-a3e3-75aec1c2ce36-kube-api-access-6jmfw\") pod \"c933c090-d160-449b-a3e3-75aec1c2ce36\" (UID: \"c933c090-d160-449b-a3e3-75aec1c2ce36\") " Oct 30 16:53:52.676396 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:52.676298 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c933c090-d160-449b-a3e3-75aec1c2ce36-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c933c090-d160-449b-a3e3-75aec1c2ce36-istio-podinfo\") pod \"c933c090-d160-449b-a3e3-75aec1c2ce36\" (UID: \"c933c090-d160-449b-a3e3-75aec1c2ce36\") " Oct 30 16:53:52.676396 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:52.676325 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c933c090-d160-449b-a3e3-75aec1c2ce36-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c933c090-d160-449b-a3e3-75aec1c2ce36-istio-data\") pod \"c933c090-d160-449b-a3e3-75aec1c2ce36\" (UID: \"c933c090-d160-449b-a3e3-75aec1c2ce36\") " Oct 30 16:53:52.676573 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:52.676398 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c933c090-d160-449b-a3e3-75aec1c2ce36-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c933c090-d160-449b-a3e3-75aec1c2ce36-workload-socket\") pod \"c933c090-d160-449b-a3e3-75aec1c2ce36\" (UID: \"c933c090-d160-449b-a3e3-75aec1c2ce36\") " Oct 30 16:53:52.676573 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:52.676506 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c933c090-d160-449b-a3e3-75aec1c2ce36-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c933c090-d160-449b-a3e3-75aec1c2ce36-credential-socket\") pod \"c933c090-d160-449b-a3e3-75aec1c2ce36\" (UID: \"c933c090-d160-449b-a3e3-75aec1c2ce36\") " Oct 30 16:53:52.676573 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:52.676565 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c933c090-d160-449b-a3e3-75aec1c2ce36-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c933c090-d160-449b-a3e3-75aec1c2ce36-istio-envoy\") pod \"c933c090-d160-449b-a3e3-75aec1c2ce36\" (UID: \"c933c090-d160-449b-a3e3-75aec1c2ce36\") " Oct 30 16:53:52.676827 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:52.676804 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c933c090-d160-449b-a3e3-75aec1c2ce36-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c933c090-d160-449b-a3e3-75aec1c2ce36-istiod-ca-cert\") pod \"c933c090-d160-449b-a3e3-75aec1c2ce36\" (UID: \"c933c090-d160-449b-a3e3-75aec1c2ce36\") " Oct 30 16:53:52.676974 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:52.676956 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c933c090-d160-449b-a3e3-75aec1c2ce36-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c933c090-d160-449b-a3e3-75aec1c2ce36-workload-certs\") pod \"c933c090-d160-449b-a3e3-75aec1c2ce36\" (UID: \"c933c090-d160-449b-a3e3-75aec1c2ce36\") " Oct 30 16:53:52.677219 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:52.676828 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c933c090-d160-449b-a3e3-75aec1c2ce36-workload-socket" pod "c933c090-d160-449b-a3e3-75aec1c2ce36" (UID: "c933c090-d160-449b-a3e3-75aec1c2ce36"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:53:52.677340 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:52.677063 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c933c090-d160-449b-a3e3-75aec1c2ce36-credential-socket" pod "c933c090-d160-449b-a3e3-75aec1c2ce36" (UID: "c933c090-d160-449b-a3e3-75aec1c2ce36"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:53:52.677411 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:52.677117 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c933c090-d160-449b-a3e3-75aec1c2ce36-istiod-ca-cert" pod "c933c090-d160-449b-a3e3-75aec1c2ce36" (UID: "c933c090-d160-449b-a3e3-75aec1c2ce36"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:53:52.677411 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:52.677170 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c933c090-d160-449b-a3e3-75aec1c2ce36-istio-data" pod "c933c090-d160-449b-a3e3-75aec1c2ce36" (UID: "c933c090-d160-449b-a3e3-75aec1c2ce36"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:53:52.677411 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:52.677383 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c933c090-d160-449b-a3e3-75aec1c2ce36-workload-certs" pod "c933c090-d160-449b-a3e3-75aec1c2ce36" (UID: "c933c090-d160-449b-a3e3-75aec1c2ce36"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:53:52.678290 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:52.678261 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c933c090-d160-449b-a3e3-75aec1c2ce36-credential-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:53:52.678409 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:52.678292 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c933c090-d160-449b-a3e3-75aec1c2ce36-istiod-ca-cert\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:53:52.678409 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:52.678310 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c933c090-d160-449b-a3e3-75aec1c2ce36-workload-certs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:53:52.678409 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:52.678323 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c933c090-d160-449b-a3e3-75aec1c2ce36-istio-data\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:53:52.678409 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:52.678335 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c933c090-d160-449b-a3e3-75aec1c2ce36-workload-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:53:52.679278 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:52.679235 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c933c090-d160-449b-a3e3-75aec1c2ce36-istio-podinfo" pod "c933c090-d160-449b-a3e3-75aec1c2ce36" (UID: "c933c090-d160-449b-a3e3-75aec1c2ce36"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:53:52.679366 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:52.679305 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c933c090-d160-449b-a3e3-75aec1c2ce36-istio-token" pod "c933c090-d160-449b-a3e3-75aec1c2ce36" (UID: "c933c090-d160-449b-a3e3-75aec1c2ce36"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:53:52.679366 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:52.679333 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c933c090-d160-449b-a3e3-75aec1c2ce36-kube-api-access-6jmfw" pod "c933c090-d160-449b-a3e3-75aec1c2ce36" (UID: "c933c090-d160-449b-a3e3-75aec1c2ce36"). InnerVolumeSpecName "kube-api-access-6jmfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:53:52.679366 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:52.679336 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c933c090-d160-449b-a3e3-75aec1c2ce36-istio-envoy" pod "c933c090-d160-449b-a3e3-75aec1c2ce36" (UID: "c933c090-d160-449b-a3e3-75aec1c2ce36"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:53:52.780010 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:52.779927 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c933c090-d160-449b-a3e3-75aec1c2ce36-istio-envoy\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:53:52.780010 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:52.779962 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c933c090-d160-449b-a3e3-75aec1c2ce36-istio-token\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:53:52.780010 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:52.779978 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jmfw\" (UniqueName: \"kubernetes.io/projected/c933c090-d160-449b-a3e3-75aec1c2ce36-kube-api-access-6jmfw\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:53:52.780010 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:52.779992 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c933c090-d160-449b-a3e3-75aec1c2ce36-istio-podinfo\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:53:53.176500 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:53.176466 4493 generic.go:356] "Generic (PLEG): container finished" podID="c933c090-d160-449b-a3e3-75aec1c2ce36" containerID="59ee73106ddcb99e1eabac289f0ffbed1eb433e7c4f5b90bc5dd6d83ead195e8" exitCode=0 Oct 30 16:53:53.176669 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:53.176527 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl" Oct 30 16:53:53.176669 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:53.176555 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl" event={"ID":"c933c090-d160-449b-a3e3-75aec1c2ce36","Type":"ContainerDied","Data":"59ee73106ddcb99e1eabac289f0ffbed1eb433e7c4f5b90bc5dd6d83ead195e8"} Oct 30 16:53:53.176669 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:53.176613 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl" event={"ID":"c933c090-d160-449b-a3e3-75aec1c2ce36","Type":"ContainerDied","Data":"c064721bf582f63b1bedf86448b2cebbd3740734a984b0f2e4cae3eda00a4525"} Oct 30 16:53:53.176669 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:53.176635 4493 scope.go:122] "RemoveContainer" containerID="59ee73106ddcb99e1eabac289f0ffbed1eb433e7c4f5b90bc5dd6d83ead195e8" Oct 30 16:53:53.214018 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:53.213969 4493 scope.go:122] "RemoveContainer" containerID="59ee73106ddcb99e1eabac289f0ffbed1eb433e7c4f5b90bc5dd6d83ead195e8" Oct 30 16:53:53.226697 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:53:53.215599 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59ee73106ddcb99e1eabac289f0ffbed1eb433e7c4f5b90bc5dd6d83ead195e8\": container with ID starting with 59ee73106ddcb99e1eabac289f0ffbed1eb433e7c4f5b90bc5dd6d83ead195e8 not found: ID does not exist" containerID="59ee73106ddcb99e1eabac289f0ffbed1eb433e7c4f5b90bc5dd6d83ead195e8" Oct 30 16:53:53.226697 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:53.215672 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59ee73106ddcb99e1eabac289f0ffbed1eb433e7c4f5b90bc5dd6d83ead195e8"} err="failed to get container status \"59ee73106ddcb99e1eabac289f0ffbed1eb433e7c4f5b90bc5dd6d83ead195e8\": rpc error: code = NotFound desc = could not find container \"59ee73106ddcb99e1eabac289f0ffbed1eb433e7c4f5b90bc5dd6d83ead195e8\": container with ID starting with 59ee73106ddcb99e1eabac289f0ffbed1eb433e7c4f5b90bc5dd6d83ead195e8 not found: ID does not exist" Oct 30 16:53:53.262228 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:53.261554 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl"] Oct 30 16:53:53.267577 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:53.267547 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-56nwl"] Oct 30 16:53:53.965807 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:53.964928 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 16:53:54.937058 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:54.937018 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c933c090-d160-449b-a3e3-75aec1c2ce36" path="/var/lib/kubelet/pods/c933c090-d160-449b-a3e3-75aec1c2ce36/volumes" Oct 30 16:53:54.976471 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:54.976435 4493 scope.go:122] "RemoveContainer" containerID="b87a436b329c5eaeca15c7101851d4cd8ef395e45f78c1a653f1d3f2e8a2f2fe" Oct 30 16:53:54.989280 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:54.989255 4493 scope.go:122] "RemoveContainer" containerID="4fc9d519c010f36746c0a8cea13fce07edfb89a83c1fac078e468f390ac3409e" Oct 30 16:53:55.003304 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:55.003275 4493 scope.go:122] "RemoveContainer" containerID="eb155eb7828685f12d7685dcfd14ced8ded79e7b218ef31ada07d3ce2a1c7d49" Oct 30 16:53:55.014337 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:53:55.014313 4493 scope.go:122] "RemoveContainer" containerID="d9873cac7ea910c33f169de39ce0609be0169e0a1db1122446be7020a2c61579" Oct 30 16:54:00.346519 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:54:00.346477 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 16:54:02.166203 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:02.166168 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4a8ab1-vwt8t"] Oct 30 16:54:02.176452 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:02.176424 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4a8ab1-vwt8t"] Oct 30 16:54:02.936722 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:02.936687 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa0be2de-94a8-45dd-a889-4724f87af42a" path="/var/lib/kubelet/pods/fa0be2de-94a8-45dd-a889-4724f87af42a/volumes" Oct 30 16:54:11.746084 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:11.746043 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq"] Oct 30 16:54:11.746925 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:11.746892 4493 cpu_manager.go:414] "containerMap: removing container" podUID="3dbdb8f4-a39a-4880-bc1b-525787aa6120" containerName="istio-proxy" Oct 30 16:54:11.746925 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:11.746916 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dbdb8f4-a39a-4880-bc1b-525787aa6120" containerName="istio-proxy" Oct 30 16:54:11.747126 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:11.746959 4493 cpu_manager.go:414] "containerMap: removing container" podUID="c933c090-d160-449b-a3e3-75aec1c2ce36" containerName="istio-proxy" Oct 30 16:54:11.747126 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:11.746967 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c933c090-d160-449b-a3e3-75aec1c2ce36" containerName="istio-proxy" Oct 30 16:54:11.747126 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:11.747003 4493 cpu_manager.go:414] "containerMap: removing container" podUID="fa0be2de-94a8-45dd-a889-4724f87af42a" containerName="revoke-keys" Oct 30 16:54:11.747126 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:11.747011 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa0be2de-94a8-45dd-a889-4724f87af42a" containerName="revoke-keys" Oct 30 16:54:11.747313 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:11.747146 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="c933c090-d160-449b-a3e3-75aec1c2ce36" containerName="istio-proxy" Oct 30 16:54:11.747313 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:11.747162 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa0be2de-94a8-45dd-a889-4724f87af42a" containerName="revoke-keys" Oct 30 16:54:11.747313 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:11.747179 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dbdb8f4-a39a-4880-bc1b-525787aa6120" containerName="istio-proxy" Oct 30 16:54:11.756893 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:11.755480 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq" Oct 30 16:54:11.759904 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:11.759879 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq"] Oct 30 16:54:11.815106 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:11.815079 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-3ca339-openshift-default-dockercfg-nv7n7\"" Oct 30 16:54:11.872616 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:11.872574 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/352047e0-584f-4a19-b56a-2a05bd960f73-workload-certs\") pod \"e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq\" (UID: \"352047e0-584f-4a19-b56a-2a05bd960f73\") " pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq" Oct 30 16:54:11.873265 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:11.873187 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/352047e0-584f-4a19-b56a-2a05bd960f73-workload-socket\") pod \"e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq\" (UID: \"352047e0-584f-4a19-b56a-2a05bd960f73\") " pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq" Oct 30 16:54:11.873426 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:11.873408 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/352047e0-584f-4a19-b56a-2a05bd960f73-istiod-ca-cert\") pod \"e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq\" (UID: \"352047e0-584f-4a19-b56a-2a05bd960f73\") " pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq" Oct 30 16:54:11.873651 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:11.873635 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/352047e0-584f-4a19-b56a-2a05bd960f73-credential-socket\") pod \"e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq\" (UID: \"352047e0-584f-4a19-b56a-2a05bd960f73\") " pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq" Oct 30 16:54:11.873799 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:11.873766 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmvzf\" (UniqueName: \"kubernetes.io/projected/352047e0-584f-4a19-b56a-2a05bd960f73-kube-api-access-rmvzf\") pod \"e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq\" (UID: \"352047e0-584f-4a19-b56a-2a05bd960f73\") " pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq" Oct 30 16:54:11.873951 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:11.873935 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/352047e0-584f-4a19-b56a-2a05bd960f73-istio-data\") pod \"e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq\" (UID: \"352047e0-584f-4a19-b56a-2a05bd960f73\") " pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq" Oct 30 16:54:11.874066 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:11.874052 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/352047e0-584f-4a19-b56a-2a05bd960f73-istio-token\") pod \"e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq\" (UID: \"352047e0-584f-4a19-b56a-2a05bd960f73\") " pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq" Oct 30 16:54:11.874197 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:11.874183 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/352047e0-584f-4a19-b56a-2a05bd960f73-istio-podinfo\") pod \"e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq\" (UID: \"352047e0-584f-4a19-b56a-2a05bd960f73\") " pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq" Oct 30 16:54:11.874319 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:11.874306 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/352047e0-584f-4a19-b56a-2a05bd960f73-istio-envoy\") pod \"e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq\" (UID: \"352047e0-584f-4a19-b56a-2a05bd960f73\") " pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq" Oct 30 16:54:11.984820 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:11.984756 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/352047e0-584f-4a19-b56a-2a05bd960f73-credential-socket\") pod \"e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq\" (UID: \"352047e0-584f-4a19-b56a-2a05bd960f73\") " pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq" Oct 30 16:54:11.985136 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:11.984839 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmvzf\" (UniqueName: \"kubernetes.io/projected/352047e0-584f-4a19-b56a-2a05bd960f73-kube-api-access-rmvzf\") pod \"e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq\" (UID: \"352047e0-584f-4a19-b56a-2a05bd960f73\") " pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq" Oct 30 16:54:11.985136 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:11.984903 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/352047e0-584f-4a19-b56a-2a05bd960f73-istio-data\") pod \"e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq\" (UID: \"352047e0-584f-4a19-b56a-2a05bd960f73\") " pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq" Oct 30 16:54:11.985136 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:11.984933 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/352047e0-584f-4a19-b56a-2a05bd960f73-istio-token\") pod \"e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq\" (UID: \"352047e0-584f-4a19-b56a-2a05bd960f73\") " pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq" Oct 30 16:54:11.985136 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:11.985003 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/352047e0-584f-4a19-b56a-2a05bd960f73-istio-podinfo\") pod \"e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq\" (UID: \"352047e0-584f-4a19-b56a-2a05bd960f73\") " pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq" Oct 30 16:54:11.985136 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:11.985057 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/352047e0-584f-4a19-b56a-2a05bd960f73-istio-envoy\") pod \"e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq\" (UID: \"352047e0-584f-4a19-b56a-2a05bd960f73\") " pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq" Oct 30 16:54:11.985441 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:11.985165 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/352047e0-584f-4a19-b56a-2a05bd960f73-workload-certs\") pod \"e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq\" (UID: \"352047e0-584f-4a19-b56a-2a05bd960f73\") " pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq" Oct 30 16:54:11.985441 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:11.985242 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/352047e0-584f-4a19-b56a-2a05bd960f73-workload-socket\") pod \"e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq\" (UID: \"352047e0-584f-4a19-b56a-2a05bd960f73\") " pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq" Oct 30 16:54:11.985441 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:11.985279 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/352047e0-584f-4a19-b56a-2a05bd960f73-istiod-ca-cert\") pod \"e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq\" (UID: \"352047e0-584f-4a19-b56a-2a05bd960f73\") " pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq" Oct 30 16:54:11.986179 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:11.986148 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/352047e0-584f-4a19-b56a-2a05bd960f73-istiod-ca-cert\") pod \"e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq\" (UID: \"352047e0-584f-4a19-b56a-2a05bd960f73\") " pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq" Oct 30 16:54:11.986762 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:11.986360 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/352047e0-584f-4a19-b56a-2a05bd960f73-credential-socket\") pod \"e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq\" (UID: \"352047e0-584f-4a19-b56a-2a05bd960f73\") " pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq" Oct 30 16:54:11.987093 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:11.987066 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/352047e0-584f-4a19-b56a-2a05bd960f73-istio-data\") pod \"e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq\" (UID: \"352047e0-584f-4a19-b56a-2a05bd960f73\") " pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq" Oct 30 16:54:11.987543 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:11.987523 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/352047e0-584f-4a19-b56a-2a05bd960f73-workload-certs\") pod \"e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq\" (UID: \"352047e0-584f-4a19-b56a-2a05bd960f73\") " pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq" Oct 30 16:54:11.987980 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:11.987959 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/352047e0-584f-4a19-b56a-2a05bd960f73-workload-socket\") pod \"e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq\" (UID: \"352047e0-584f-4a19-b56a-2a05bd960f73\") " pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq" Oct 30 16:54:11.990748 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:11.990726 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/352047e0-584f-4a19-b56a-2a05bd960f73-istio-podinfo\") pod \"e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq\" (UID: \"352047e0-584f-4a19-b56a-2a05bd960f73\") " pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq" Oct 30 16:54:12.012896 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:12.006763 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/352047e0-584f-4a19-b56a-2a05bd960f73-istio-envoy\") pod \"e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq\" (UID: \"352047e0-584f-4a19-b56a-2a05bd960f73\") " pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq" Oct 30 16:54:12.012896 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:12.011731 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmvzf\" (UniqueName: \"kubernetes.io/projected/352047e0-584f-4a19-b56a-2a05bd960f73-kube-api-access-rmvzf\") pod \"e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq\" (UID: \"352047e0-584f-4a19-b56a-2a05bd960f73\") " pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq" Oct 30 16:54:12.016105 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:12.016072 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/352047e0-584f-4a19-b56a-2a05bd960f73-istio-token\") pod \"e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq\" (UID: \"352047e0-584f-4a19-b56a-2a05bd960f73\") " pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq" Oct 30 16:54:12.073644 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:12.073599 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq" Oct 30 16:54:12.441077 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:12.440215 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq"] Oct 30 16:54:12.444664 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:54:12.444080 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod352047e0_584f_4a19_b56a_2a05bd960f73.slice/crio-842810a137f081f4b541a4feb377871e495efd9d943e32fcc964e31a9491fe1a WatchSource:0}: Error finding container 842810a137f081f4b541a4feb377871e495efd9d943e32fcc964e31a9491fe1a: Status 404 returned error can't find the container with id 842810a137f081f4b541a4feb377871e495efd9d943e32fcc964e31a9491fe1a Oct 30 16:54:12.447183 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:12.447141 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:54:12.447274 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:12.447232 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:54:12.447320 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:12.447276 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:54:13.395693 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:13.395660 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq" event={"ID":"352047e0-584f-4a19-b56a-2a05bd960f73","Type":"ContainerStarted","Data":"22845bb6cf127000a5e57dfbdfd9a53b0362cf14638a9303ca3ca8cd8394e5a5"} Oct 30 16:54:13.396213 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:13.395713 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq" event={"ID":"352047e0-584f-4a19-b56a-2a05bd960f73","Type":"ContainerStarted","Data":"842810a137f081f4b541a4feb377871e495efd9d943e32fcc964e31a9491fe1a"} Oct 30 16:54:13.457770 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:13.457719 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq" podStartSLOduration=2.457704838 podStartE2EDuration="2.457704838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:54:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:54:13.452155778 +0000 UTC m=+1880.679330899" watchObservedRunningTime="2026-10-30 16:54:13.457704838 +0000 UTC m=+1880.684879938" Oct 30 16:54:14.074315 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:14.074277 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq" Oct 30 16:54:14.079977 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:14.079952 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq" Oct 30 16:54:14.407279 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:14.407246 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq" Oct 30 16:54:14.408259 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:14.408242 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq" Oct 30 16:54:30.343099 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:54:30.343053 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 16:54:30.985541 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:30.984767 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 16:54:33.044157 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:33.044123 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7686c34-glx8c"] Oct 30 16:54:33.055605 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:33.055573 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7686c34-glx8c"] Oct 30 16:54:34.934899 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:34.934863 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bc54dd3-3017-4f7c-a6d7-3b81a363f3b7" path="/var/lib/kubelet/pods/8bc54dd3-3017-4f7c-a6d7-3b81a363f3b7/volumes" Oct 30 16:54:44.917750 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:44.917715 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-cb6251/auth-test-model-cb6251-kserve-75ff8464df-8qjhr"] Oct 30 16:54:44.923476 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:44.923453 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-cb6251/auth-test-model-cb6251-kserve-75ff8464df-8qjhr" Oct 30 16:54:44.951504 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:44.951473 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-cb6251\"/\"kube-root-ca.crt\"" Oct 30 16:54:44.951677 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:44.951538 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-cb6251\"/\"openshift-service-ca.crt\"" Oct 30 16:54:44.951872 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:44.951851 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-cb6251\"/\"default-dockercfg-z7ngd\"" Oct 30 16:54:44.951967 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:44.951945 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-cb6251\"/\"auth-test-model-cb6251-kserve-self-signed-certs\"" Oct 30 16:54:44.958718 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:44.958667 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-cb6251/auth-test-model-cb6251-kserve-75ff8464df-8qjhr"] Oct 30 16:54:44.999141 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:44.999112 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b506c0ec-f4f5-44aa-a8b7-0a5280586ddd-tls-certs\") pod \"auth-test-model-cb6251-kserve-75ff8464df-8qjhr\" (UID: \"b506c0ec-f4f5-44aa-a8b7-0a5280586ddd\") " pod="ai-tenant-e2e-shared-a-w2-cb6251/auth-test-model-cb6251-kserve-75ff8464df-8qjhr" Oct 30 16:54:44.999447 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:44.999418 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b506c0ec-f4f5-44aa-a8b7-0a5280586ddd-home\") pod \"auth-test-model-cb6251-kserve-75ff8464df-8qjhr\" (UID: \"b506c0ec-f4f5-44aa-a8b7-0a5280586ddd\") " pod="ai-tenant-e2e-shared-a-w2-cb6251/auth-test-model-cb6251-kserve-75ff8464df-8qjhr" Oct 30 16:54:44.999574 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:44.999469 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b506c0ec-f4f5-44aa-a8b7-0a5280586ddd-tmp-dir\") pod \"auth-test-model-cb6251-kserve-75ff8464df-8qjhr\" (UID: \"b506c0ec-f4f5-44aa-a8b7-0a5280586ddd\") " pod="ai-tenant-e2e-shared-a-w2-cb6251/auth-test-model-cb6251-kserve-75ff8464df-8qjhr" Oct 30 16:54:44.999574 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:44.999514 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b506c0ec-f4f5-44aa-a8b7-0a5280586ddd-dshm\") pod \"auth-test-model-cb6251-kserve-75ff8464df-8qjhr\" (UID: \"b506c0ec-f4f5-44aa-a8b7-0a5280586ddd\") " pod="ai-tenant-e2e-shared-a-w2-cb6251/auth-test-model-cb6251-kserve-75ff8464df-8qjhr" Oct 30 16:54:44.999574 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:44.999539 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b506c0ec-f4f5-44aa-a8b7-0a5280586ddd-model-cache\") pod \"auth-test-model-cb6251-kserve-75ff8464df-8qjhr\" (UID: \"b506c0ec-f4f5-44aa-a8b7-0a5280586ddd\") " pod="ai-tenant-e2e-shared-a-w2-cb6251/auth-test-model-cb6251-kserve-75ff8464df-8qjhr" Oct 30 16:54:44.999756 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:44.999579 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcmkt\" (UniqueName: \"kubernetes.io/projected/b506c0ec-f4f5-44aa-a8b7-0a5280586ddd-kube-api-access-kcmkt\") pod \"auth-test-model-cb6251-kserve-75ff8464df-8qjhr\" (UID: \"b506c0ec-f4f5-44aa-a8b7-0a5280586ddd\") " pod="ai-tenant-e2e-shared-a-w2-cb6251/auth-test-model-cb6251-kserve-75ff8464df-8qjhr" Oct 30 16:54:45.100602 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:45.100562 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b506c0ec-f4f5-44aa-a8b7-0a5280586ddd-home\") pod \"auth-test-model-cb6251-kserve-75ff8464df-8qjhr\" (UID: \"b506c0ec-f4f5-44aa-a8b7-0a5280586ddd\") " pod="ai-tenant-e2e-shared-a-w2-cb6251/auth-test-model-cb6251-kserve-75ff8464df-8qjhr" Oct 30 16:54:45.100602 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:45.100606 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b506c0ec-f4f5-44aa-a8b7-0a5280586ddd-tmp-dir\") pod \"auth-test-model-cb6251-kserve-75ff8464df-8qjhr\" (UID: \"b506c0ec-f4f5-44aa-a8b7-0a5280586ddd\") " pod="ai-tenant-e2e-shared-a-w2-cb6251/auth-test-model-cb6251-kserve-75ff8464df-8qjhr" Oct 30 16:54:45.100886 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:45.100624 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b506c0ec-f4f5-44aa-a8b7-0a5280586ddd-dshm\") pod \"auth-test-model-cb6251-kserve-75ff8464df-8qjhr\" (UID: \"b506c0ec-f4f5-44aa-a8b7-0a5280586ddd\") " pod="ai-tenant-e2e-shared-a-w2-cb6251/auth-test-model-cb6251-kserve-75ff8464df-8qjhr" Oct 30 16:54:45.100886 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:45.100642 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b506c0ec-f4f5-44aa-a8b7-0a5280586ddd-model-cache\") pod \"auth-test-model-cb6251-kserve-75ff8464df-8qjhr\" (UID: \"b506c0ec-f4f5-44aa-a8b7-0a5280586ddd\") " pod="ai-tenant-e2e-shared-a-w2-cb6251/auth-test-model-cb6251-kserve-75ff8464df-8qjhr" Oct 30 16:54:45.100886 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:45.100682 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcmkt\" (UniqueName: \"kubernetes.io/projected/b506c0ec-f4f5-44aa-a8b7-0a5280586ddd-kube-api-access-kcmkt\") pod \"auth-test-model-cb6251-kserve-75ff8464df-8qjhr\" (UID: \"b506c0ec-f4f5-44aa-a8b7-0a5280586ddd\") " pod="ai-tenant-e2e-shared-a-w2-cb6251/auth-test-model-cb6251-kserve-75ff8464df-8qjhr" Oct 30 16:54:45.101060 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:45.100933 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b506c0ec-f4f5-44aa-a8b7-0a5280586ddd-tls-certs\") pod \"auth-test-model-cb6251-kserve-75ff8464df-8qjhr\" (UID: \"b506c0ec-f4f5-44aa-a8b7-0a5280586ddd\") " pod="ai-tenant-e2e-shared-a-w2-cb6251/auth-test-model-cb6251-kserve-75ff8464df-8qjhr" Oct 30 16:54:45.101121 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:45.101090 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b506c0ec-f4f5-44aa-a8b7-0a5280586ddd-home\") pod \"auth-test-model-cb6251-kserve-75ff8464df-8qjhr\" (UID: \"b506c0ec-f4f5-44aa-a8b7-0a5280586ddd\") " pod="ai-tenant-e2e-shared-a-w2-cb6251/auth-test-model-cb6251-kserve-75ff8464df-8qjhr" Oct 30 16:54:45.101232 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:45.101211 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b506c0ec-f4f5-44aa-a8b7-0a5280586ddd-model-cache\") pod \"auth-test-model-cb6251-kserve-75ff8464df-8qjhr\" (UID: \"b506c0ec-f4f5-44aa-a8b7-0a5280586ddd\") " pod="ai-tenant-e2e-shared-a-w2-cb6251/auth-test-model-cb6251-kserve-75ff8464df-8qjhr" Oct 30 16:54:45.101818 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:45.101756 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b506c0ec-f4f5-44aa-a8b7-0a5280586ddd-tmp-dir\") pod \"auth-test-model-cb6251-kserve-75ff8464df-8qjhr\" (UID: \"b506c0ec-f4f5-44aa-a8b7-0a5280586ddd\") " pod="ai-tenant-e2e-shared-a-w2-cb6251/auth-test-model-cb6251-kserve-75ff8464df-8qjhr" Oct 30 16:54:45.104169 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:45.104142 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b506c0ec-f4f5-44aa-a8b7-0a5280586ddd-tls-certs\") pod \"auth-test-model-cb6251-kserve-75ff8464df-8qjhr\" (UID: \"b506c0ec-f4f5-44aa-a8b7-0a5280586ddd\") " pod="ai-tenant-e2e-shared-a-w2-cb6251/auth-test-model-cb6251-kserve-75ff8464df-8qjhr" Oct 30 16:54:45.107541 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:45.107513 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b506c0ec-f4f5-44aa-a8b7-0a5280586ddd-dshm\") pod \"auth-test-model-cb6251-kserve-75ff8464df-8qjhr\" (UID: \"b506c0ec-f4f5-44aa-a8b7-0a5280586ddd\") " pod="ai-tenant-e2e-shared-a-w2-cb6251/auth-test-model-cb6251-kserve-75ff8464df-8qjhr" Oct 30 16:54:45.124865 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:45.124836 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcmkt\" (UniqueName: \"kubernetes.io/projected/b506c0ec-f4f5-44aa-a8b7-0a5280586ddd-kube-api-access-kcmkt\") pod \"auth-test-model-cb6251-kserve-75ff8464df-8qjhr\" (UID: \"b506c0ec-f4f5-44aa-a8b7-0a5280586ddd\") " pod="ai-tenant-e2e-shared-a-w2-cb6251/auth-test-model-cb6251-kserve-75ff8464df-8qjhr" Oct 30 16:54:45.248897 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:45.246050 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-cb6251/auth-test-model-cb6251-kserve-75ff8464df-8qjhr" Oct 30 16:54:45.597035 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:45.597004 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-cb6251/auth-test-model-cb6251-kserve-75ff8464df-8qjhr"] Oct 30 16:54:45.597591 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:54:45.597563 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb506c0ec_f4f5_44aa_a8b7_0a5280586ddd.slice/crio-69a3c956d3b61ffa856da72a0a3e71b55f25c7d07e0b9b3e3c9145f3d5d492ca WatchSource:0}: Error finding container 69a3c956d3b61ffa856da72a0a3e71b55f25c7d07e0b9b3e3c9145f3d5d492ca: Status 404 returned error can't find the container with id 69a3c956d3b61ffa856da72a0a3e71b55f25c7d07e0b9b3e3c9145f3d5d492ca Oct 30 16:54:45.760367 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:45.760320 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-cb6251/auth-test-model-cb6251-kserve-75ff8464df-8qjhr" event={"ID":"b506c0ec-f4f5-44aa-a8b7-0a5280586ddd","Type":"ContainerStarted","Data":"290d464e21b1acd504667699a8af4bf91f1cea8f47392fcf33f642712825fe8f"} Oct 30 16:54:45.760367 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:45.760374 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-cb6251/auth-test-model-cb6251-kserve-75ff8464df-8qjhr" event={"ID":"b506c0ec-f4f5-44aa-a8b7-0a5280586ddd","Type":"ContainerStarted","Data":"69a3c956d3b61ffa856da72a0a3e71b55f25c7d07e0b9b3e3c9145f3d5d492ca"} Oct 30 16:54:45.804809 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:45.804729 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-cb6251/auth-test-model-cb6251-kserve-75ff8464df-8qjhr" podStartSLOduration=1.804710689 podStartE2EDuration="1.804710689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:54:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:54:45.801136998 +0000 UTC m=+1913.028312098" watchObservedRunningTime="2026-10-30 16:54:45.804710689 +0000 UTC m=+1913.031885793" Oct 30 16:54:46.247599 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:46.247565 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-cb6251/auth-test-model-cb6251-kserve-75ff8464df-8qjhr" Oct 30 16:54:55.247869 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:55.247828 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-cb6251/auth-test-model-cb6251-kserve-75ff8464df-8qjhr" Oct 30 16:54:55.260845 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:55.260823 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-cb6251/auth-test-model-cb6251-kserve-75ff8464df-8qjhr" Oct 30 16:54:55.917956 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:54:55.917913 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-cb6251/auth-test-model-cb6251-kserve-75ff8464df-8qjhr" Oct 30 16:55:00.364177 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:55:00.364138 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 16:55:07.061343 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:07.061245 4493 generic.go:356] "Generic (PLEG): container finished" podID="352047e0-584f-4a19-b56a-2a05bd960f73" containerID="22845bb6cf127000a5e57dfbdfd9a53b0362cf14638a9303ca3ca8cd8394e5a5" exitCode=0 Oct 30 16:55:07.063221 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:07.061986 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq" event={"ID":"352047e0-584f-4a19-b56a-2a05bd960f73","Type":"ContainerDied","Data":"22845bb6cf127000a5e57dfbdfd9a53b0362cf14638a9303ca3ca8cd8394e5a5"} Oct 30 16:55:07.063221 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:07.062449 4493 scope.go:122] "RemoveContainer" containerID="22845bb6cf127000a5e57dfbdfd9a53b0362cf14638a9303ca3ca8cd8394e5a5" Oct 30 16:55:07.064458 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:07.064417 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:55:07.064557 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:07.064510 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:55:07.064612 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:07.064556 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:55:08.074424 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:08.074391 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq" event={"ID":"352047e0-584f-4a19-b56a-2a05bd960f73","Type":"ContainerStarted","Data":"4ed8965da336e6dad873d92dd2bc6c3737edc05ae4e921f269eda44a0a3100d2"} Oct 30 16:55:09.074999 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:09.074960 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq" Oct 30 16:55:09.075484 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:09.075256 4493 patch_prober.go:29] interesting pod/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Oct 30 16:55:09.075484 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:09.075307 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq" podUID="352047e0-584f-4a19-b56a-2a05bd960f73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Oct 30 16:55:10.074618 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:10.074585 4493 patch_prober.go:29] interesting pod/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Oct 30 16:55:10.074810 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:10.074639 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq" podUID="352047e0-584f-4a19-b56a-2a05bd960f73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Oct 30 16:55:11.074854 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:11.074818 4493 patch_prober.go:29] interesting pod/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Oct 30 16:55:11.075250 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:11.074870 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq" podUID="352047e0-584f-4a19-b56a-2a05bd960f73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Oct 30 16:55:11.164167 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:11.162193 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-dd715e/auth-test-model-dd715e-kserve-7c54c47c97-c4n77"] Oct 30 16:55:11.170616 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:11.170593 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-dd715e/auth-test-model-dd715e-kserve-7c54c47c97-c4n77" Oct 30 16:55:11.173033 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:11.173005 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-dd715e/auth-test-model-dd715e-kserve-7c54c47c97-c4n77"] Oct 30 16:55:11.187893 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:11.187839 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-dd715e\"/\"openshift-service-ca.crt\"" Oct 30 16:55:11.188069 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:11.187974 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-dd715e\"/\"default-dockercfg-mbfzb\"" Oct 30 16:55:11.188069 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:11.188038 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-dd715e\"/\"kube-root-ca.crt\"" Oct 30 16:55:11.188238 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:11.188164 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-dd715e\"/\"auth-test-model-dd715e-kserve-self-signed-certs\"" Oct 30 16:55:11.250134 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:11.250047 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67c90611-0953-4152-8696-695093024e17-tls-certs\") pod \"auth-test-model-dd715e-kserve-7c54c47c97-c4n77\" (UID: \"67c90611-0953-4152-8696-695093024e17\") " pod="ai-tenant-e2e-shared-a-w3-dd715e/auth-test-model-dd715e-kserve-7c54c47c97-c4n77" Oct 30 16:55:11.250571 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:11.250397 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djzbp\" (UniqueName: \"kubernetes.io/projected/67c90611-0953-4152-8696-695093024e17-kube-api-access-djzbp\") pod \"auth-test-model-dd715e-kserve-7c54c47c97-c4n77\" (UID: \"67c90611-0953-4152-8696-695093024e17\") " pod="ai-tenant-e2e-shared-a-w3-dd715e/auth-test-model-dd715e-kserve-7c54c47c97-c4n77" Oct 30 16:55:11.250952 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:11.250932 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67c90611-0953-4152-8696-695093024e17-dshm\") pod \"auth-test-model-dd715e-kserve-7c54c47c97-c4n77\" (UID: \"67c90611-0953-4152-8696-695093024e17\") " pod="ai-tenant-e2e-shared-a-w3-dd715e/auth-test-model-dd715e-kserve-7c54c47c97-c4n77" Oct 30 16:55:11.251245 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:11.251207 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67c90611-0953-4152-8696-695093024e17-tmp-dir\") pod \"auth-test-model-dd715e-kserve-7c54c47c97-c4n77\" (UID: \"67c90611-0953-4152-8696-695093024e17\") " pod="ai-tenant-e2e-shared-a-w3-dd715e/auth-test-model-dd715e-kserve-7c54c47c97-c4n77" Oct 30 16:55:11.252273 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:11.252255 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67c90611-0953-4152-8696-695093024e17-model-cache\") pod \"auth-test-model-dd715e-kserve-7c54c47c97-c4n77\" (UID: \"67c90611-0953-4152-8696-695093024e17\") " pod="ai-tenant-e2e-shared-a-w3-dd715e/auth-test-model-dd715e-kserve-7c54c47c97-c4n77" Oct 30 16:55:11.252593 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:11.252576 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67c90611-0953-4152-8696-695093024e17-home\") pod \"auth-test-model-dd715e-kserve-7c54c47c97-c4n77\" (UID: \"67c90611-0953-4152-8696-695093024e17\") " pod="ai-tenant-e2e-shared-a-w3-dd715e/auth-test-model-dd715e-kserve-7c54c47c97-c4n77" Oct 30 16:55:11.354206 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:11.354100 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67c90611-0953-4152-8696-695093024e17-home\") pod \"auth-test-model-dd715e-kserve-7c54c47c97-c4n77\" (UID: \"67c90611-0953-4152-8696-695093024e17\") " pod="ai-tenant-e2e-shared-a-w3-dd715e/auth-test-model-dd715e-kserve-7c54c47c97-c4n77" Oct 30 16:55:11.354396 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:11.354206 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67c90611-0953-4152-8696-695093024e17-tls-certs\") pod \"auth-test-model-dd715e-kserve-7c54c47c97-c4n77\" (UID: \"67c90611-0953-4152-8696-695093024e17\") " pod="ai-tenant-e2e-shared-a-w3-dd715e/auth-test-model-dd715e-kserve-7c54c47c97-c4n77" Oct 30 16:55:11.354396 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:11.354245 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djzbp\" (UniqueName: \"kubernetes.io/projected/67c90611-0953-4152-8696-695093024e17-kube-api-access-djzbp\") pod \"auth-test-model-dd715e-kserve-7c54c47c97-c4n77\" (UID: \"67c90611-0953-4152-8696-695093024e17\") " pod="ai-tenant-e2e-shared-a-w3-dd715e/auth-test-model-dd715e-kserve-7c54c47c97-c4n77" Oct 30 16:55:11.354396 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:11.354284 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67c90611-0953-4152-8696-695093024e17-dshm\") pod \"auth-test-model-dd715e-kserve-7c54c47c97-c4n77\" (UID: \"67c90611-0953-4152-8696-695093024e17\") " pod="ai-tenant-e2e-shared-a-w3-dd715e/auth-test-model-dd715e-kserve-7c54c47c97-c4n77" Oct 30 16:55:11.354396 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:11.354364 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67c90611-0953-4152-8696-695093024e17-tmp-dir\") pod \"auth-test-model-dd715e-kserve-7c54c47c97-c4n77\" (UID: \"67c90611-0953-4152-8696-695093024e17\") " pod="ai-tenant-e2e-shared-a-w3-dd715e/auth-test-model-dd715e-kserve-7c54c47c97-c4n77" Oct 30 16:55:11.354633 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:11.354453 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67c90611-0953-4152-8696-695093024e17-model-cache\") pod \"auth-test-model-dd715e-kserve-7c54c47c97-c4n77\" (UID: \"67c90611-0953-4152-8696-695093024e17\") " pod="ai-tenant-e2e-shared-a-w3-dd715e/auth-test-model-dd715e-kserve-7c54c47c97-c4n77" Oct 30 16:55:11.354947 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:11.354915 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67c90611-0953-4152-8696-695093024e17-model-cache\") pod \"auth-test-model-dd715e-kserve-7c54c47c97-c4n77\" (UID: \"67c90611-0953-4152-8696-695093024e17\") " pod="ai-tenant-e2e-shared-a-w3-dd715e/auth-test-model-dd715e-kserve-7c54c47c97-c4n77" Oct 30 16:55:11.355213 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:11.355176 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67c90611-0953-4152-8696-695093024e17-home\") pod \"auth-test-model-dd715e-kserve-7c54c47c97-c4n77\" (UID: \"67c90611-0953-4152-8696-695093024e17\") " pod="ai-tenant-e2e-shared-a-w3-dd715e/auth-test-model-dd715e-kserve-7c54c47c97-c4n77" Oct 30 16:55:11.355453 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:11.355428 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67c90611-0953-4152-8696-695093024e17-tmp-dir\") pod \"auth-test-model-dd715e-kserve-7c54c47c97-c4n77\" (UID: \"67c90611-0953-4152-8696-695093024e17\") " pod="ai-tenant-e2e-shared-a-w3-dd715e/auth-test-model-dd715e-kserve-7c54c47c97-c4n77" Oct 30 16:55:11.357856 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:11.357840 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67c90611-0953-4152-8696-695093024e17-dshm\") pod \"auth-test-model-dd715e-kserve-7c54c47c97-c4n77\" (UID: \"67c90611-0953-4152-8696-695093024e17\") " pod="ai-tenant-e2e-shared-a-w3-dd715e/auth-test-model-dd715e-kserve-7c54c47c97-c4n77" Oct 30 16:55:11.358408 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:11.358384 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67c90611-0953-4152-8696-695093024e17-tls-certs\") pod \"auth-test-model-dd715e-kserve-7c54c47c97-c4n77\" (UID: \"67c90611-0953-4152-8696-695093024e17\") " pod="ai-tenant-e2e-shared-a-w3-dd715e/auth-test-model-dd715e-kserve-7c54c47c97-c4n77" Oct 30 16:55:11.378479 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:11.378455 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djzbp\" (UniqueName: \"kubernetes.io/projected/67c90611-0953-4152-8696-695093024e17-kube-api-access-djzbp\") pod \"auth-test-model-dd715e-kserve-7c54c47c97-c4n77\" (UID: \"67c90611-0953-4152-8696-695093024e17\") " pod="ai-tenant-e2e-shared-a-w3-dd715e/auth-test-model-dd715e-kserve-7c54c47c97-c4n77" Oct 30 16:55:11.503823 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:11.501241 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-dd715e/auth-test-model-dd715e-kserve-7c54c47c97-c4n77" Oct 30 16:55:11.942069 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:11.942033 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-dd715e/auth-test-model-dd715e-kserve-7c54c47c97-c4n77"] Oct 30 16:55:11.942867 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:55:11.942840 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod67c90611_0953_4152_8696_695093024e17.slice/crio-b831b171af94b33e2002b098954f41d46029d2c238d4d53f99e0cf6269e5eedc WatchSource:0}: Error finding container b831b171af94b33e2002b098954f41d46029d2c238d4d53f99e0cf6269e5eedc: Status 404 returned error can't find the container with id b831b171af94b33e2002b098954f41d46029d2c238d4d53f99e0cf6269e5eedc Oct 30 16:55:12.074681 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:12.074650 4493 patch_prober.go:29] interesting pod/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Oct 30 16:55:12.074888 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:12.074707 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq" podUID="352047e0-584f-4a19-b56a-2a05bd960f73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Oct 30 16:55:12.114815 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:12.114476 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-dd715e/auth-test-model-dd715e-kserve-7c54c47c97-c4n77" event={"ID":"67c90611-0953-4152-8696-695093024e17","Type":"ContainerStarted","Data":"0d2995d84e2b4bb0f2235dfd3d63669d3037d390ec87d502ec20af9da9c2190c"} Oct 30 16:55:12.114815 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:12.114529 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-dd715e/auth-test-model-dd715e-kserve-7c54c47c97-c4n77" event={"ID":"67c90611-0953-4152-8696-695093024e17","Type":"ContainerStarted","Data":"b831b171af94b33e2002b098954f41d46029d2c238d4d53f99e0cf6269e5eedc"} Oct 30 16:55:12.173036 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:12.172972 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-dd715e/auth-test-model-dd715e-kserve-7c54c47c97-c4n77" podStartSLOduration=1.172954767 podStartE2EDuration="1.172954767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:55:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:55:12.169383194 +0000 UTC m=+1939.396558294" watchObservedRunningTime="2026-10-30 16:55:12.172954767 +0000 UTC m=+1939.400129866" Oct 30 16:55:12.502165 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:12.502123 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-dd715e/auth-test-model-dd715e-kserve-7c54c47c97-c4n77" Oct 30 16:55:13.074918 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:13.074873 4493 patch_prober.go:29] interesting pod/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Oct 30 16:55:13.075475 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:13.074930 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq" podUID="352047e0-584f-4a19-b56a-2a05bd960f73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Oct 30 16:55:14.079817 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:14.079762 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq" Oct 30 16:55:14.080257 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:14.080161 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq" Oct 30 16:55:14.080740 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:14.080721 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq" Oct 30 16:55:21.501941 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:21.501904 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-dd715e/auth-test-model-dd715e-kserve-7c54c47c97-c4n77" Oct 30 16:55:21.519938 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:21.519910 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-dd715e/auth-test-model-dd715e-kserve-7c54c47c97-c4n77" Oct 30 16:55:22.228981 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:22.228951 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-dd715e/auth-test-model-dd715e-kserve-7c54c47c97-c4n77" Oct 30 16:55:23.771809 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:23.766514 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 16:55:25.012202 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:25.012172 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx"] Oct 30 16:55:25.016023 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:25.015998 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx" Oct 30 16:55:25.036566 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:25.036235 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-06e7f6-openshift-default-dockercfg-pf8rn\"" Oct 30 16:55:25.057396 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:25.057363 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx"] Oct 30 16:55:25.089663 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:25.089611 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d917fe1-dd87-48a4-94ff-75ece269728f-istiod-ca-cert\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx\" (UID: \"1d917fe1-dd87-48a4-94ff-75ece269728f\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx" Oct 30 16:55:25.089855 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:25.089686 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d917fe1-dd87-48a4-94ff-75ece269728f-workload-socket\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx\" (UID: \"1d917fe1-dd87-48a4-94ff-75ece269728f\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx" Oct 30 16:55:25.089855 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:25.089829 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d917fe1-dd87-48a4-94ff-75ece269728f-workload-certs\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx\" (UID: \"1d917fe1-dd87-48a4-94ff-75ece269728f\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx" Oct 30 16:55:25.090000 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:25.089877 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d917fe1-dd87-48a4-94ff-75ece269728f-istio-podinfo\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx\" (UID: \"1d917fe1-dd87-48a4-94ff-75ece269728f\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx" Oct 30 16:55:25.090000 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:25.089924 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x8sd\" (UniqueName: \"kubernetes.io/projected/1d917fe1-dd87-48a4-94ff-75ece269728f-kube-api-access-5x8sd\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx\" (UID: \"1d917fe1-dd87-48a4-94ff-75ece269728f\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx" Oct 30 16:55:25.090000 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:25.089957 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d917fe1-dd87-48a4-94ff-75ece269728f-istio-data\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx\" (UID: \"1d917fe1-dd87-48a4-94ff-75ece269728f\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx" Oct 30 16:55:25.090120 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:25.090042 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d917fe1-dd87-48a4-94ff-75ece269728f-credential-socket\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx\" (UID: \"1d917fe1-dd87-48a4-94ff-75ece269728f\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx" Oct 30 16:55:25.090191 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:25.090169 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d917fe1-dd87-48a4-94ff-75ece269728f-istio-envoy\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx\" (UID: \"1d917fe1-dd87-48a4-94ff-75ece269728f\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx" Oct 30 16:55:25.090293 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:25.090273 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d917fe1-dd87-48a4-94ff-75ece269728f-istio-token\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx\" (UID: \"1d917fe1-dd87-48a4-94ff-75ece269728f\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx" Oct 30 16:55:25.195104 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:25.195061 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d917fe1-dd87-48a4-94ff-75ece269728f-istiod-ca-cert\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx\" (UID: \"1d917fe1-dd87-48a4-94ff-75ece269728f\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx" Oct 30 16:55:25.195306 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:25.195125 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d917fe1-dd87-48a4-94ff-75ece269728f-workload-socket\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx\" (UID: \"1d917fe1-dd87-48a4-94ff-75ece269728f\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx" Oct 30 16:55:25.195306 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:25.195147 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d917fe1-dd87-48a4-94ff-75ece269728f-workload-certs\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx\" (UID: \"1d917fe1-dd87-48a4-94ff-75ece269728f\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx" Oct 30 16:55:25.195306 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:25.195177 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d917fe1-dd87-48a4-94ff-75ece269728f-istio-podinfo\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx\" (UID: \"1d917fe1-dd87-48a4-94ff-75ece269728f\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx" Oct 30 16:55:25.195306 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:25.195211 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x8sd\" (UniqueName: \"kubernetes.io/projected/1d917fe1-dd87-48a4-94ff-75ece269728f-kube-api-access-5x8sd\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx\" (UID: \"1d917fe1-dd87-48a4-94ff-75ece269728f\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx" Oct 30 16:55:25.195306 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:25.195241 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d917fe1-dd87-48a4-94ff-75ece269728f-istio-data\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx\" (UID: \"1d917fe1-dd87-48a4-94ff-75ece269728f\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx" Oct 30 16:55:25.195306 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:25.195277 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d917fe1-dd87-48a4-94ff-75ece269728f-credential-socket\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx\" (UID: \"1d917fe1-dd87-48a4-94ff-75ece269728f\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx" Oct 30 16:55:25.195652 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:25.195342 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d917fe1-dd87-48a4-94ff-75ece269728f-istio-envoy\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx\" (UID: \"1d917fe1-dd87-48a4-94ff-75ece269728f\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx" Oct 30 16:55:25.195652 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:25.195387 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d917fe1-dd87-48a4-94ff-75ece269728f-istio-token\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx\" (UID: \"1d917fe1-dd87-48a4-94ff-75ece269728f\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx" Oct 30 16:55:25.195767 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:25.195744 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d917fe1-dd87-48a4-94ff-75ece269728f-workload-certs\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx\" (UID: \"1d917fe1-dd87-48a4-94ff-75ece269728f\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx" Oct 30 16:55:25.195893 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:25.195867 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d917fe1-dd87-48a4-94ff-75ece269728f-credential-socket\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx\" (UID: \"1d917fe1-dd87-48a4-94ff-75ece269728f\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx" Oct 30 16:55:25.196173 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:25.196140 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d917fe1-dd87-48a4-94ff-75ece269728f-istio-data\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx\" (UID: \"1d917fe1-dd87-48a4-94ff-75ece269728f\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx" Oct 30 16:55:25.196356 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:25.196337 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d917fe1-dd87-48a4-94ff-75ece269728f-workload-socket\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx\" (UID: \"1d917fe1-dd87-48a4-94ff-75ece269728f\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx" Oct 30 16:55:25.196749 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:25.196730 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d917fe1-dd87-48a4-94ff-75ece269728f-istiod-ca-cert\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx\" (UID: \"1d917fe1-dd87-48a4-94ff-75ece269728f\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx" Oct 30 16:55:25.201816 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:25.198510 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d917fe1-dd87-48a4-94ff-75ece269728f-istio-envoy\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx\" (UID: \"1d917fe1-dd87-48a4-94ff-75ece269728f\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx" Oct 30 16:55:25.201816 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:25.198985 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d917fe1-dd87-48a4-94ff-75ece269728f-istio-podinfo\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx\" (UID: \"1d917fe1-dd87-48a4-94ff-75ece269728f\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx" Oct 30 16:55:25.221378 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:25.221348 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x8sd\" (UniqueName: \"kubernetes.io/projected/1d917fe1-dd87-48a4-94ff-75ece269728f-kube-api-access-5x8sd\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx\" (UID: \"1d917fe1-dd87-48a4-94ff-75ece269728f\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx" Oct 30 16:55:25.222521 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:25.222497 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d917fe1-dd87-48a4-94ff-75ece269728f-istio-token\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx\" (UID: \"1d917fe1-dd87-48a4-94ff-75ece269728f\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx" Oct 30 16:55:25.345950 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:25.345039 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx" Oct 30 16:55:25.780004 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:25.779969 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx"] Oct 30 16:55:25.780162 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:55:25.780053 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d917fe1_dd87_48a4_94ff_75ece269728f.slice/crio-1cf0c8fe6675a5a6d908c0851fe6bf683df53b657e25a8dbb7aaf1078fee8a83 WatchSource:0}: Error finding container 1cf0c8fe6675a5a6d908c0851fe6bf683df53b657e25a8dbb7aaf1078fee8a83: Status 404 returned error can't find the container with id 1cf0c8fe6675a5a6d908c0851fe6bf683df53b657e25a8dbb7aaf1078fee8a83 Oct 30 16:55:25.783212 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:25.783146 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:55:25.783309 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:25.783263 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:55:25.783361 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:25.783310 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:55:26.261639 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:26.261596 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx" event={"ID":"1d917fe1-dd87-48a4-94ff-75ece269728f","Type":"ContainerStarted","Data":"9c9355ed96a93693e74a7eb26b026e4646934012d696b37e1e19030aafe04913"} Oct 30 16:55:26.262153 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:26.261664 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx" event={"ID":"1d917fe1-dd87-48a4-94ff-75ece269728f","Type":"ContainerStarted","Data":"1cf0c8fe6675a5a6d908c0851fe6bf683df53b657e25a8dbb7aaf1078fee8a83"} Oct 30 16:55:26.330351 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:26.329544 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx" podStartSLOduration=2.329523967 podStartE2EDuration="2.329523967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:55:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:55:26.317678881 +0000 UTC m=+1953.544854002" watchObservedRunningTime="2026-10-30 16:55:26.329523967 +0000 UTC m=+1953.556699068" Oct 30 16:55:26.345954 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:26.345920 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx" Oct 30 16:55:26.347687 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:26.347654 4493 patch_prober.go:29] interesting pod/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Oct 30 16:55:26.347866 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:26.347705 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx" podUID="1d917fe1-dd87-48a4-94ff-75ece269728f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Oct 30 16:55:27.351740 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:27.351710 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx" Oct 30 16:55:28.279413 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:28.279386 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx" Oct 30 16:55:28.280229 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:28.280213 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx" Oct 30 16:55:28.424050 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:28.424018 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-07cb3c-pqxxf"] Oct 30 16:55:28.439999 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:28.439965 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-07cb3c-pqxxf" Oct 30 16:55:28.448614 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:28.448585 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-07cb3c-pqxxf"] Oct 30 16:55:28.460595 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:28.459711 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:55:28.460595 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:28.459877 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 16:55:28.460595 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:28.460513 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:55:28.460872 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:28.460739 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:55:28.550669 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:28.550593 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksbpd\" (UniqueName: \"kubernetes.io/projected/62a5f7c1-d17e-4c2c-a10d-12152f2250a8-kube-api-access-ksbpd\") pod \"maas-api-revoke-keys-e2e-praxis-a-07cb3c-pqxxf\" (UID: \"62a5f7c1-d17e-4c2c-a10d-12152f2250a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-07cb3c-pqxxf" Oct 30 16:55:28.550832 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:28.550671 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62a5f7c1-d17e-4c2c-a10d-12152f2250a8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-07cb3c-pqxxf\" (UID: \"62a5f7c1-d17e-4c2c-a10d-12152f2250a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-07cb3c-pqxxf" Oct 30 16:55:28.550832 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:28.550698 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62a5f7c1-d17e-4c2c-a10d-12152f2250a8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-07cb3c-pqxxf\" (UID: \"62a5f7c1-d17e-4c2c-a10d-12152f2250a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-07cb3c-pqxxf" Oct 30 16:55:28.652302 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:28.652269 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksbpd\" (UniqueName: \"kubernetes.io/projected/62a5f7c1-d17e-4c2c-a10d-12152f2250a8-kube-api-access-ksbpd\") pod \"maas-api-revoke-keys-e2e-praxis-a-07cb3c-pqxxf\" (UID: \"62a5f7c1-d17e-4c2c-a10d-12152f2250a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-07cb3c-pqxxf" Oct 30 16:55:28.652456 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:28.652355 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62a5f7c1-d17e-4c2c-a10d-12152f2250a8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-07cb3c-pqxxf\" (UID: \"62a5f7c1-d17e-4c2c-a10d-12152f2250a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-07cb3c-pqxxf" Oct 30 16:55:28.652456 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:28.652386 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62a5f7c1-d17e-4c2c-a10d-12152f2250a8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-07cb3c-pqxxf\" (UID: \"62a5f7c1-d17e-4c2c-a10d-12152f2250a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-07cb3c-pqxxf" Oct 30 16:55:28.653039 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:28.653014 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62a5f7c1-d17e-4c2c-a10d-12152f2250a8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-07cb3c-pqxxf\" (UID: \"62a5f7c1-d17e-4c2c-a10d-12152f2250a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-07cb3c-pqxxf" Oct 30 16:55:28.654973 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:28.654950 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62a5f7c1-d17e-4c2c-a10d-12152f2250a8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-07cb3c-pqxxf\" (UID: \"62a5f7c1-d17e-4c2c-a10d-12152f2250a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-07cb3c-pqxxf" Oct 30 16:55:28.674988 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:28.674958 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksbpd\" (UniqueName: \"kubernetes.io/projected/62a5f7c1-d17e-4c2c-a10d-12152f2250a8-kube-api-access-ksbpd\") pod \"maas-api-revoke-keys-e2e-praxis-a-07cb3c-pqxxf\" (UID: \"62a5f7c1-d17e-4c2c-a10d-12152f2250a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-07cb3c-pqxxf" Oct 30 16:55:28.768586 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:28.768554 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-07cb3c-pqxxf" Oct 30 16:55:29.088212 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:55:29.088184 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62a5f7c1_d17e_4c2c_a10d_12152f2250a8.slice/crio-0674b17370dd7d86ebf95a339d811675d367e22aaa186d423000139e269fa0fc WatchSource:0}: Error finding container 0674b17370dd7d86ebf95a339d811675d367e22aaa186d423000139e269fa0fc: Status 404 returned error can't find the container with id 0674b17370dd7d86ebf95a339d811675d367e22aaa186d423000139e269fa0fc Oct 30 16:55:29.089129 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:29.089106 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-07cb3c-pqxxf"] Oct 30 16:55:29.288384 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:29.288352 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-07cb3c-pqxxf" event={"ID":"62a5f7c1-d17e-4c2c-a10d-12152f2250a8","Type":"ContainerStarted","Data":"0674b17370dd7d86ebf95a339d811675d367e22aaa186d423000139e269fa0fc"} Oct 30 16:55:30.297838 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:30.297764 4493 generic.go:356] "Generic (PLEG): container finished" podID="62a5f7c1-d17e-4c2c-a10d-12152f2250a8" containerID="2cc76d05b0df466327eb91ffc64e474a793e39c41b328c66a3787a73f41d4559" exitCode=0 Oct 30 16:55:30.298547 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:30.297843 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-07cb3c-pqxxf" event={"ID":"62a5f7c1-d17e-4c2c-a10d-12152f2250a8","Type":"ContainerDied","Data":"2cc76d05b0df466327eb91ffc64e474a793e39c41b328c66a3787a73f41d4559"} Oct 30 16:55:30.343080 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:55:30.343042 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 16:55:31.479569 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:31.479541 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-07cb3c-pqxxf" Oct 30 16:55:31.601713 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:31.601639 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62a5f7c1-d17e-4c2c-a10d-12152f2250a8-kube-api-access-ksbpd\" (UniqueName: \"kubernetes.io/projected/62a5f7c1-d17e-4c2c-a10d-12152f2250a8-kube-api-access-ksbpd\") pod \"62a5f7c1-d17e-4c2c-a10d-12152f2250a8\" (UID: \"62a5f7c1-d17e-4c2c-a10d-12152f2250a8\") " Oct 30 16:55:31.602039 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:31.601738 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62a5f7c1-d17e-4c2c-a10d-12152f2250a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62a5f7c1-d17e-4c2c-a10d-12152f2250a8-maas-api-openshift-ca\") pod \"62a5f7c1-d17e-4c2c-a10d-12152f2250a8\" (UID: \"62a5f7c1-d17e-4c2c-a10d-12152f2250a8\") " Oct 30 16:55:31.602039 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:31.601919 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62a5f7c1-d17e-4c2c-a10d-12152f2250a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62a5f7c1-d17e-4c2c-a10d-12152f2250a8-maas-api-xks-ca\") pod \"62a5f7c1-d17e-4c2c-a10d-12152f2250a8\" (UID: \"62a5f7c1-d17e-4c2c-a10d-12152f2250a8\") " Oct 30 16:55:31.602192 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:31.602172 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62a5f7c1-d17e-4c2c-a10d-12152f2250a8-maas-api-openshift-ca" pod "62a5f7c1-d17e-4c2c-a10d-12152f2250a8" (UID: "62a5f7c1-d17e-4c2c-a10d-12152f2250a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:55:31.602485 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:31.602459 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62a5f7c1-d17e-4c2c-a10d-12152f2250a8-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:55:31.604148 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:31.604125 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62a5f7c1-d17e-4c2c-a10d-12152f2250a8-maas-api-xks-ca" pod "62a5f7c1-d17e-4c2c-a10d-12152f2250a8" (UID: "62a5f7c1-d17e-4c2c-a10d-12152f2250a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:55:31.604243 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:31.604222 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62a5f7c1-d17e-4c2c-a10d-12152f2250a8-kube-api-access-ksbpd" pod "62a5f7c1-d17e-4c2c-a10d-12152f2250a8" (UID: "62a5f7c1-d17e-4c2c-a10d-12152f2250a8"). InnerVolumeSpecName "kube-api-access-ksbpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:55:31.709280 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:31.709085 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksbpd\" (UniqueName: \"kubernetes.io/projected/62a5f7c1-d17e-4c2c-a10d-12152f2250a8-kube-api-access-ksbpd\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:55:31.709280 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:31.709125 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62a5f7c1-d17e-4c2c-a10d-12152f2250a8-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:55:32.320032 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:32.320001 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-07cb3c-pqxxf" Oct 30 16:55:32.320216 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:32.320043 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-07cb3c-pqxxf" event={"ID":"62a5f7c1-d17e-4c2c-a10d-12152f2250a8","Type":"ContainerDied","Data":"0674b17370dd7d86ebf95a339d811675d367e22aaa186d423000139e269fa0fc"} Oct 30 16:55:32.320216 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:32.320091 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0674b17370dd7d86ebf95a339d811675d367e22aaa186d423000139e269fa0fc" Oct 30 16:55:40.037518 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:40.037488 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3a5a675-zdph5"] Oct 30 16:55:40.049150 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:40.049125 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3a5a675-zdph5"] Oct 30 16:55:40.940736 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:40.940700 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad2f82e5-833f-483c-99e9-a460840d8b5a" path="/var/lib/kubelet/pods/ad2f82e5-833f-483c-99e9-a460840d8b5a/volumes" Oct 30 16:55:55.087708 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:55.087671 4493 scope.go:122] "RemoveContainer" containerID="f4957884cf37e49002efd32ccfadb4df27c13e1a37d4996fbda091a438e26da4" Oct 30 16:55:55.099841 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:55.099811 4493 scope.go:122] "RemoveContainer" containerID="0ecc74b6275225a68a2a4d76183555daa06dd9ae80797214ce31f1f506c91418" Oct 30 16:55:55.109269 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:55.109249 4493 scope.go:122] "RemoveContainer" containerID="c560012373a63803dbb746dc6b8ea1989475d264d545a2bafbd4d100d0936aca" Oct 30 16:55:55.118364 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:55.118329 4493 scope.go:122] "RemoveContainer" containerID="94d860faaf973cd622a3d4c5952c3039dfcfd3f0196ed5a2ff08c77e691b5987" Oct 30 16:55:55.941126 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:55.941088 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn"] Oct 30 16:55:55.941817 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:55.941774 4493 cpu_manager.go:414] "containerMap: removing container" podUID="62a5f7c1-d17e-4c2c-a10d-12152f2250a8" containerName="revoke-keys" Oct 30 16:55:55.941817 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:55.941817 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62a5f7c1-d17e-4c2c-a10d-12152f2250a8" containerName="revoke-keys" Oct 30 16:55:55.941991 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:55.941963 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="62a5f7c1-d17e-4c2c-a10d-12152f2250a8" containerName="revoke-keys" Oct 30 16:55:55.946442 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:55.946420 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn" Oct 30 16:55:55.966214 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:55.966189 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-dd715e-openshift-default-dockercfg-fhwwv\"" Oct 30 16:55:55.986662 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:55.986629 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn"] Oct 30 16:55:56.039960 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:56.039929 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60c9faa7-ch855"] Oct 30 16:55:56.053482 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:56.053442 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94c03808-f5c9-4486-9cb9-3251d0de671d-istio-data\") pod \"e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn\" (UID: \"94c03808-f5c9-4486-9cb9-3251d0de671d\") " pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn" Oct 30 16:55:56.053667 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:56.053584 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94c03808-f5c9-4486-9cb9-3251d0de671d-istio-envoy\") pod \"e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn\" (UID: \"94c03808-f5c9-4486-9cb9-3251d0de671d\") " pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn" Oct 30 16:55:56.053667 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:56.053625 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94c03808-f5c9-4486-9cb9-3251d0de671d-istio-token\") pod \"e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn\" (UID: \"94c03808-f5c9-4486-9cb9-3251d0de671d\") " pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn" Oct 30 16:55:56.053869 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:56.053696 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94c03808-f5c9-4486-9cb9-3251d0de671d-workload-certs\") pod \"e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn\" (UID: \"94c03808-f5c9-4486-9cb9-3251d0de671d\") " pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn" Oct 30 16:55:56.053869 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:56.053854 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94c03808-f5c9-4486-9cb9-3251d0de671d-istiod-ca-cert\") pod \"e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn\" (UID: \"94c03808-f5c9-4486-9cb9-3251d0de671d\") " pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn" Oct 30 16:55:56.054011 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:56.053894 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94c03808-f5c9-4486-9cb9-3251d0de671d-workload-socket\") pod \"e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn\" (UID: \"94c03808-f5c9-4486-9cb9-3251d0de671d\") " pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn" Oct 30 16:55:56.054011 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:56.053962 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94c03808-f5c9-4486-9cb9-3251d0de671d-istio-podinfo\") pod \"e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn\" (UID: \"94c03808-f5c9-4486-9cb9-3251d0de671d\") " pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn" Oct 30 16:55:56.054011 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:56.054006 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94c03808-f5c9-4486-9cb9-3251d0de671d-credential-socket\") pod \"e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn\" (UID: \"94c03808-f5c9-4486-9cb9-3251d0de671d\") " pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn" Oct 30 16:55:56.054310 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:56.054060 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gtnk\" (UniqueName: \"kubernetes.io/projected/94c03808-f5c9-4486-9cb9-3251d0de671d-kube-api-access-9gtnk\") pod \"e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn\" (UID: \"94c03808-f5c9-4486-9cb9-3251d0de671d\") " pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn" Oct 30 16:55:56.064835 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:56.064772 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60c9faa7-ch855"] Oct 30 16:55:56.156214 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:56.156178 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94c03808-f5c9-4486-9cb9-3251d0de671d-istio-envoy\") pod \"e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn\" (UID: \"94c03808-f5c9-4486-9cb9-3251d0de671d\") " pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn" Oct 30 16:55:56.156214 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:56.156216 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94c03808-f5c9-4486-9cb9-3251d0de671d-istio-token\") pod \"e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn\" (UID: \"94c03808-f5c9-4486-9cb9-3251d0de671d\") " pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn" Oct 30 16:55:56.156775 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:56.156256 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94c03808-f5c9-4486-9cb9-3251d0de671d-workload-certs\") pod \"e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn\" (UID: \"94c03808-f5c9-4486-9cb9-3251d0de671d\") " pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn" Oct 30 16:55:56.156775 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:56.156317 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94c03808-f5c9-4486-9cb9-3251d0de671d-istiod-ca-cert\") pod \"e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn\" (UID: \"94c03808-f5c9-4486-9cb9-3251d0de671d\") " pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn" Oct 30 16:55:56.156775 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:56.156333 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94c03808-f5c9-4486-9cb9-3251d0de671d-workload-socket\") pod \"e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn\" (UID: \"94c03808-f5c9-4486-9cb9-3251d0de671d\") " pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn" Oct 30 16:55:56.156775 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:56.156373 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94c03808-f5c9-4486-9cb9-3251d0de671d-istio-podinfo\") pod \"e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn\" (UID: \"94c03808-f5c9-4486-9cb9-3251d0de671d\") " pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn" Oct 30 16:55:56.156775 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:56.156396 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94c03808-f5c9-4486-9cb9-3251d0de671d-credential-socket\") pod \"e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn\" (UID: \"94c03808-f5c9-4486-9cb9-3251d0de671d\") " pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn" Oct 30 16:55:56.156775 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:56.156426 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gtnk\" (UniqueName: \"kubernetes.io/projected/94c03808-f5c9-4486-9cb9-3251d0de671d-kube-api-access-9gtnk\") pod \"e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn\" (UID: \"94c03808-f5c9-4486-9cb9-3251d0de671d\") " pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn" Oct 30 16:55:56.156775 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:56.156476 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94c03808-f5c9-4486-9cb9-3251d0de671d-istio-data\") pod \"e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn\" (UID: \"94c03808-f5c9-4486-9cb9-3251d0de671d\") " pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn" Oct 30 16:55:56.156775 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:56.156697 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94c03808-f5c9-4486-9cb9-3251d0de671d-workload-certs\") pod \"e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn\" (UID: \"94c03808-f5c9-4486-9cb9-3251d0de671d\") " pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn" Oct 30 16:55:56.157115 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:56.156779 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94c03808-f5c9-4486-9cb9-3251d0de671d-credential-socket\") pod \"e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn\" (UID: \"94c03808-f5c9-4486-9cb9-3251d0de671d\") " pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn" Oct 30 16:55:56.157115 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:56.156856 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94c03808-f5c9-4486-9cb9-3251d0de671d-istio-data\") pod \"e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn\" (UID: \"94c03808-f5c9-4486-9cb9-3251d0de671d\") " pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn" Oct 30 16:55:56.157185 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:56.157108 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94c03808-f5c9-4486-9cb9-3251d0de671d-istiod-ca-cert\") pod \"e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn\" (UID: \"94c03808-f5c9-4486-9cb9-3251d0de671d\") " pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn" Oct 30 16:55:56.157223 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:56.157192 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94c03808-f5c9-4486-9cb9-3251d0de671d-workload-socket\") pod \"e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn\" (UID: \"94c03808-f5c9-4486-9cb9-3251d0de671d\") " pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn" Oct 30 16:55:56.158736 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:56.158709 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94c03808-f5c9-4486-9cb9-3251d0de671d-istio-envoy\") pod \"e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn\" (UID: \"94c03808-f5c9-4486-9cb9-3251d0de671d\") " pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn" Oct 30 16:55:56.159122 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:56.159103 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94c03808-f5c9-4486-9cb9-3251d0de671d-istio-podinfo\") pod \"e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn\" (UID: \"94c03808-f5c9-4486-9cb9-3251d0de671d\") " pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn" Oct 30 16:55:56.180216 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:56.180184 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gtnk\" (UniqueName: \"kubernetes.io/projected/94c03808-f5c9-4486-9cb9-3251d0de671d-kube-api-access-9gtnk\") pod \"e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn\" (UID: \"94c03808-f5c9-4486-9cb9-3251d0de671d\") " pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn" Oct 30 16:55:56.180354 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:56.180252 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94c03808-f5c9-4486-9cb9-3251d0de671d-istio-token\") pod \"e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn\" (UID: \"94c03808-f5c9-4486-9cb9-3251d0de671d\") " pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn" Oct 30 16:55:56.262975 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:56.262889 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn" Oct 30 16:55:56.593937 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:56.593805 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn"] Oct 30 16:55:56.594050 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:55:56.594006 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94c03808_f5c9_4486_9cb9_3251d0de671d.slice/crio-fa4508f60247fd40ec2e4923d7c935a253d5f04f7b30e1e69a980ba530186953 WatchSource:0}: Error finding container fa4508f60247fd40ec2e4923d7c935a253d5f04f7b30e1e69a980ba530186953: Status 404 returned error can't find the container with id fa4508f60247fd40ec2e4923d7c935a253d5f04f7b30e1e69a980ba530186953 Oct 30 16:55:56.596721 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:56.596685 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:55:56.596821 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:56.596754 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:55:56.596821 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:56.596807 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:55:56.936324 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:56.936287 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e57cb61-fe2f-4a2c-930b-eee27bb86327" path="/var/lib/kubelet/pods/5e57cb61-fe2f-4a2c-930b-eee27bb86327/volumes" Oct 30 16:55:57.568809 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:57.568145 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn" event={"ID":"94c03808-f5c9-4486-9cb9-3251d0de671d","Type":"ContainerStarted","Data":"d173fbcc27f17ac896559d111bf61a822dc3a786e1737baa35bb6a277b8db548"} Oct 30 16:55:57.568809 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:57.568207 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn" event={"ID":"94c03808-f5c9-4486-9cb9-3251d0de671d","Type":"ContainerStarted","Data":"fa4508f60247fd40ec2e4923d7c935a253d5f04f7b30e1e69a980ba530186953"} Oct 30 16:55:57.664172 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:57.661885 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn" podStartSLOduration=2.661849815 podStartE2EDuration="2.661849815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:55:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:55:57.645431072 +0000 UTC m=+1984.872606171" watchObservedRunningTime="2026-10-30 16:55:57.661849815 +0000 UTC m=+1984.889024921" Oct 30 16:55:58.043032 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:58.042994 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8aedbba2-cqbbp"] Oct 30 16:55:58.047688 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:58.047656 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8aedbba2-cqbbp"] Oct 30 16:55:58.263676 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:58.263648 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn" Oct 30 16:55:58.265205 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:58.265181 4493 patch_prober.go:29] interesting pod/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Oct 30 16:55:58.265282 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:58.265247 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn" podUID="94c03808-f5c9-4486-9cb9-3251d0de671d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Oct 30 16:55:58.943103 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:58.943059 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb05f1aa-442f-4999-b8e2-bb01d42e9d7a" path="/var/lib/kubelet/pods/cb05f1aa-442f-4999-b8e2-bb01d42e9d7a/volumes" Oct 30 16:55:59.263317 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:59.263243 4493 patch_prober.go:29] interesting pod/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Oct 30 16:55:59.263317 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:55:59.263293 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn" podUID="94c03808-f5c9-4486-9cb9-3251d0de671d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Oct 30 16:56:00.264024 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:00.263986 4493 patch_prober.go:29] interesting pod/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Oct 30 16:56:00.264519 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:00.264058 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn" podUID="94c03808-f5c9-4486-9cb9-3251d0de671d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Oct 30 16:56:00.368080 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:56:00.368050 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 16:56:01.263162 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:01.263129 4493 patch_prober.go:29] interesting pod/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Oct 30 16:56:01.263374 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:01.263198 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn" podUID="94c03808-f5c9-4486-9cb9-3251d0de671d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Oct 30 16:56:02.263729 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:02.263697 4493 patch_prober.go:29] interesting pod/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Oct 30 16:56:02.264127 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:02.263763 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn" podUID="94c03808-f5c9-4486-9cb9-3251d0de671d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Oct 30 16:56:03.267981 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:03.267950 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn" Oct 30 16:56:03.268496 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:03.268354 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn" Oct 30 16:56:03.268981 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:03.268962 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn" Oct 30 16:56:04.588543 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:04.588501 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2d14cbb-44s9f"] Oct 30 16:56:04.592555 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:04.592530 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2d14cbb-44s9f" Oct 30 16:56:04.616149 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:04.616117 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2d14cbb-44s9f"] Oct 30 16:56:04.621636 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:04.621607 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 16:56:04.625090 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:04.622020 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:56:04.625280 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:04.622048 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:56:04.625391 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:04.624447 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:56:04.703230 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:04.703199 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9b5x\" (UniqueName: \"kubernetes.io/projected/3d340ae1-f1e3-4b62-a1b2-8555489dc185-kube-api-access-x9b5x\") pod \"maas-api-revoke-sub-b2d14cbb-44s9f\" (UID: \"3d340ae1-f1e3-4b62-a1b2-8555489dc185\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2d14cbb-44s9f" Oct 30 16:56:04.703405 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:04.703361 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d340ae1-f1e3-4b62-a1b2-8555489dc185-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2d14cbb-44s9f\" (UID: \"3d340ae1-f1e3-4b62-a1b2-8555489dc185\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2d14cbb-44s9f" Oct 30 16:56:04.703405 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:04.703391 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d340ae1-f1e3-4b62-a1b2-8555489dc185-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2d14cbb-44s9f\" (UID: \"3d340ae1-f1e3-4b62-a1b2-8555489dc185\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2d14cbb-44s9f" Oct 30 16:56:04.804585 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:04.804543 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9b5x\" (UniqueName: \"kubernetes.io/projected/3d340ae1-f1e3-4b62-a1b2-8555489dc185-kube-api-access-x9b5x\") pod \"maas-api-revoke-sub-b2d14cbb-44s9f\" (UID: \"3d340ae1-f1e3-4b62-a1b2-8555489dc185\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2d14cbb-44s9f" Oct 30 16:56:04.804852 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:04.804652 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d340ae1-f1e3-4b62-a1b2-8555489dc185-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2d14cbb-44s9f\" (UID: \"3d340ae1-f1e3-4b62-a1b2-8555489dc185\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2d14cbb-44s9f" Oct 30 16:56:04.804852 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:04.804674 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d340ae1-f1e3-4b62-a1b2-8555489dc185-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2d14cbb-44s9f\" (UID: \"3d340ae1-f1e3-4b62-a1b2-8555489dc185\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2d14cbb-44s9f" Oct 30 16:56:04.805425 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:04.805397 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d340ae1-f1e3-4b62-a1b2-8555489dc185-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2d14cbb-44s9f\" (UID: \"3d340ae1-f1e3-4b62-a1b2-8555489dc185\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2d14cbb-44s9f" Oct 30 16:56:04.807430 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:04.807405 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d340ae1-f1e3-4b62-a1b2-8555489dc185-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2d14cbb-44s9f\" (UID: \"3d340ae1-f1e3-4b62-a1b2-8555489dc185\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2d14cbb-44s9f" Oct 30 16:56:04.832659 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:04.832619 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9b5x\" (UniqueName: \"kubernetes.io/projected/3d340ae1-f1e3-4b62-a1b2-8555489dc185-kube-api-access-x9b5x\") pod \"maas-api-revoke-sub-b2d14cbb-44s9f\" (UID: \"3d340ae1-f1e3-4b62-a1b2-8555489dc185\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2d14cbb-44s9f" Oct 30 16:56:04.906089 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:04.906049 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2d14cbb-44s9f" Oct 30 16:56:05.276718 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:05.276684 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2d14cbb-44s9f"] Oct 30 16:56:05.277288 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:56:05.277259 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d340ae1_f1e3_4b62_a1b2_8555489dc185.slice/crio-7a316612c9b8d3f6a3cb433bea0c9189fbaf196a28b6187d6464d8aad055afb9 WatchSource:0}: Error finding container 7a316612c9b8d3f6a3cb433bea0c9189fbaf196a28b6187d6464d8aad055afb9: Status 404 returned error can't find the container with id 7a316612c9b8d3f6a3cb433bea0c9189fbaf196a28b6187d6464d8aad055afb9 Oct 30 16:56:05.666557 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:05.666522 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2d14cbb-44s9f" event={"ID":"3d340ae1-f1e3-4b62-a1b2-8555489dc185","Type":"ContainerStarted","Data":"c07523f6483e7498f42a9e9a1088e7ab2b12c24eef4b19750fbd332a4bab7145"} Oct 30 16:56:05.666973 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:05.666574 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2d14cbb-44s9f" event={"ID":"3d340ae1-f1e3-4b62-a1b2-8555489dc185","Type":"ContainerStarted","Data":"7a316612c9b8d3f6a3cb433bea0c9189fbaf196a28b6187d6464d8aad055afb9"} Oct 30 16:56:05.710575 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:05.710525 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2d14cbb-44s9f" podStartSLOduration=1.710511553 podStartE2EDuration="1.710511553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:56:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:56:05.70561305 +0000 UTC m=+1992.932788151" watchObservedRunningTime="2026-10-30 16:56:05.710511553 +0000 UTC m=+1992.937686652" Oct 30 16:56:06.676556 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:06.676520 4493 generic.go:356] "Generic (PLEG): container finished" podID="3d340ae1-f1e3-4b62-a1b2-8555489dc185" containerID="c07523f6483e7498f42a9e9a1088e7ab2b12c24eef4b19750fbd332a4bab7145" exitCode=0 Oct 30 16:56:06.677046 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:06.676613 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2d14cbb-44s9f" event={"ID":"3d340ae1-f1e3-4b62-a1b2-8555489dc185","Type":"ContainerDied","Data":"c07523f6483e7498f42a9e9a1088e7ab2b12c24eef4b19750fbd332a4bab7145"} Oct 30 16:56:07.721587 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:07.721556 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 16:56:07.881636 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:07.881611 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2d14cbb-44s9f" Oct 30 16:56:08.060861 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:08.060716 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d340ae1-f1e3-4b62-a1b2-8555489dc185-kube-api-access-x9b5x\" (UniqueName: \"kubernetes.io/projected/3d340ae1-f1e3-4b62-a1b2-8555489dc185-kube-api-access-x9b5x\") pod \"3d340ae1-f1e3-4b62-a1b2-8555489dc185\" (UID: \"3d340ae1-f1e3-4b62-a1b2-8555489dc185\") " Oct 30 16:56:08.060861 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:08.060826 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d340ae1-f1e3-4b62-a1b2-8555489dc185-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d340ae1-f1e3-4b62-a1b2-8555489dc185-maas-api-xks-ca\") pod \"3d340ae1-f1e3-4b62-a1b2-8555489dc185\" (UID: \"3d340ae1-f1e3-4b62-a1b2-8555489dc185\") " Oct 30 16:56:08.061114 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:08.060953 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d340ae1-f1e3-4b62-a1b2-8555489dc185-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d340ae1-f1e3-4b62-a1b2-8555489dc185-maas-api-openshift-ca\") pod \"3d340ae1-f1e3-4b62-a1b2-8555489dc185\" (UID: \"3d340ae1-f1e3-4b62-a1b2-8555489dc185\") " Oct 30 16:56:08.061699 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:08.061664 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d340ae1-f1e3-4b62-a1b2-8555489dc185-maas-api-openshift-ca" pod "3d340ae1-f1e3-4b62-a1b2-8555489dc185" (UID: "3d340ae1-f1e3-4b62-a1b2-8555489dc185"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:56:08.062100 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:08.062077 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d340ae1-f1e3-4b62-a1b2-8555489dc185-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:56:08.063371 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:08.063348 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d340ae1-f1e3-4b62-a1b2-8555489dc185-maas-api-xks-ca" pod "3d340ae1-f1e3-4b62-a1b2-8555489dc185" (UID: "3d340ae1-f1e3-4b62-a1b2-8555489dc185"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:56:08.063571 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:08.063556 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d340ae1-f1e3-4b62-a1b2-8555489dc185-kube-api-access-x9b5x" pod "3d340ae1-f1e3-4b62-a1b2-8555489dc185" (UID: "3d340ae1-f1e3-4b62-a1b2-8555489dc185"). InnerVolumeSpecName "kube-api-access-x9b5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:56:08.163239 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:08.163203 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9b5x\" (UniqueName: \"kubernetes.io/projected/3d340ae1-f1e3-4b62-a1b2-8555489dc185-kube-api-access-x9b5x\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:56:08.163239 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:08.163235 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d340ae1-f1e3-4b62-a1b2-8555489dc185-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:56:08.714769 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:08.714730 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2d14cbb-44s9f" event={"ID":"3d340ae1-f1e3-4b62-a1b2-8555489dc185","Type":"ContainerDied","Data":"7a316612c9b8d3f6a3cb433bea0c9189fbaf196a28b6187d6464d8aad055afb9"} Oct 30 16:56:08.715101 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:08.715079 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a316612c9b8d3f6a3cb433bea0c9189fbaf196a28b6187d6464d8aad055afb9" Oct 30 16:56:08.715286 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:08.715270 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2d14cbb-44s9f" Oct 30 16:56:10.038199 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:10.038158 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d21c86c8-w4d65"] Oct 30 16:56:10.049877 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:10.049845 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d21c86c8-w4d65"] Oct 30 16:56:10.936682 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:10.936649 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fb4b393-7056-4e36-87bf-e3c9f0c7bfd3" path="/var/lib/kubelet/pods/1fb4b393-7056-4e36-87bf-e3c9f0c7bfd3/volumes" Oct 30 16:56:11.036563 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:11.036526 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-793f1b59-dww6v"] Oct 30 16:56:11.046312 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:11.046257 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-793f1b59-dww6v"] Oct 30 16:56:12.949176 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:12.949142 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0959750-e09d-44cb-8a15-86408ff8de6e" path="/var/lib/kubelet/pods/d0959750-e09d-44cb-8a15-86408ff8de6e/volumes" Oct 30 16:56:20.033675 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:20.033638 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-573e69d7-7d9zk"] Oct 30 16:56:20.045050 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:20.045019 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-573e69d7-7d9zk"] Oct 30 16:56:20.934504 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:20.934470 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0bb0921-5020-450a-81bf-3f1e6f385939" path="/var/lib/kubelet/pods/b0bb0921-5020-450a-81bf-3f1e6f385939/volumes" Oct 30 16:56:26.034501 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:26.034467 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b744ec4f-nttt6"] Oct 30 16:56:26.045558 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:26.045532 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b744ec4f-nttt6"] Oct 30 16:56:26.936397 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:26.936359 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de478b61-79a3-4aae-9bc8-cdda3391855d" path="/var/lib/kubelet/pods/de478b61-79a3-4aae-9bc8-cdda3391855d/volumes" Oct 30 16:56:29.043960 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:29.043930 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92963943-c2bqh"] Oct 30 16:56:29.048416 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:29.048390 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92963943-c2bqh"] Oct 30 16:56:30.342658 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:56:30.342618 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 16:56:30.935446 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:30.935409 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a46c947e-b960-472d-a3b4-859cdd639b83" path="/var/lib/kubelet/pods/a46c947e-b960-472d-a3b4-859cdd639b83/volumes" Oct 30 16:56:33.054105 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:33.054061 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52d3111f-gm54m"] Oct 30 16:56:33.065304 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:33.065277 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52d3111f-gm54m"] Oct 30 16:56:34.936105 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:34.936015 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b37484ec-3263-40c3-a268-e39297e58a81" path="/var/lib/kubelet/pods/b37484ec-3263-40c3-a268-e39297e58a81/volumes" Oct 30 16:56:38.036869 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:38.036829 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-293848aa-rzk7x"] Oct 30 16:56:38.051820 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:38.051743 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-293848aa-rzk7x"] Oct 30 16:56:38.585815 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:38.585748 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-07cb3c-pqxxf"] Oct 30 16:56:38.598554 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:38.598510 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-07cb3c-pqxxf"] Oct 30 16:56:38.946098 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:38.946058 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62a5f7c1-d17e-4c2c-a10d-12152f2250a8" path="/var/lib/kubelet/pods/62a5f7c1-d17e-4c2c-a10d-12152f2250a8/volumes" Oct 30 16:56:38.946548 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:38.946532 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c384f2a6-168e-423e-9d81-bf21b618621b" path="/var/lib/kubelet/pods/c384f2a6-168e-423e-9d81-bf21b618621b/volumes" Oct 30 16:56:42.040478 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:42.040444 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0ebf400-2vzj4"] Oct 30 16:56:42.049001 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:42.048974 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0ebf400-2vzj4"] Oct 30 16:56:42.945054 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:42.945020 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91635317-235e-44ef-bd57-c71110a989c8" path="/var/lib/kubelet/pods/91635317-235e-44ef-bd57-c71110a989c8/volumes" Oct 30 16:56:43.051715 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:43.051676 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad31a642-xrxsz"] Oct 30 16:56:43.055061 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:43.055030 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad31a642-xrxsz"] Oct 30 16:56:44.936994 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:44.936949 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7cca371-51e1-4544-a846-dc6fd448f6c0" path="/var/lib/kubelet/pods/d7cca371-51e1-4544-a846-dc6fd448f6c0/volumes" Oct 30 16:56:49.138825 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:49.138763 4493 generic.go:356] "Generic (PLEG): container finished" podID="1d917fe1-dd87-48a4-94ff-75ece269728f" containerID="9c9355ed96a93693e74a7eb26b026e4646934012d696b37e1e19030aafe04913" exitCode=0 Oct 30 16:56:49.138825 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:49.138817 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx" event={"ID":"1d917fe1-dd87-48a4-94ff-75ece269728f","Type":"ContainerDied","Data":"9c9355ed96a93693e74a7eb26b026e4646934012d696b37e1e19030aafe04913"} Oct 30 16:56:49.139447 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:49.139350 4493 scope.go:122] "RemoveContainer" containerID="9c9355ed96a93693e74a7eb26b026e4646934012d696b37e1e19030aafe04913" Oct 30 16:56:49.141343 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:49.141307 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:56:49.141441 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:49.141402 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:56:49.141484 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:49.141448 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 16:56:50.153540 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:50.153509 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx" event={"ID":"1d917fe1-dd87-48a4-94ff-75ece269728f","Type":"ContainerStarted","Data":"1d8d4bce72ac31f56e7d8737dd1fe0aee76c7d4bf7318aea6eca64f90c53f9a8"} Oct 30 16:56:50.345549 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:50.345514 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx" Oct 30 16:56:50.345748 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:50.345694 4493 patch_prober.go:29] interesting pod/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Oct 30 16:56:50.345849 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:50.345746 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx" podUID="1d917fe1-dd87-48a4-94ff-75ece269728f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Oct 30 16:56:51.346099 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:51.346052 4493 patch_prober.go:29] interesting pod/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Oct 30 16:56:51.346588 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:51.346129 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx" podUID="1d917fe1-dd87-48a4-94ff-75ece269728f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Oct 30 16:56:52.345956 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:52.345919 4493 patch_prober.go:29] interesting pod/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Oct 30 16:56:52.346138 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:52.345984 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx" podUID="1d917fe1-dd87-48a4-94ff-75ece269728f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Oct 30 16:56:53.346691 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:53.346660 4493 patch_prober.go:29] interesting pod/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Oct 30 16:56:53.347136 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:53.346712 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx" podUID="1d917fe1-dd87-48a4-94ff-75ece269728f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Oct 30 16:56:54.346234 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:54.346199 4493 patch_prober.go:29] interesting pod/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Oct 30 16:56:54.346428 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:54.346256 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx" podUID="1d917fe1-dd87-48a4-94ff-75ece269728f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Oct 30 16:56:55.171039 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:55.171010 4493 scope.go:122] "RemoveContainer" containerID="b5442b0183aad780769d7e32378bdcb1422081aec4a9ccc853564d6d174ffa77" Oct 30 16:56:55.182353 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:55.182334 4493 scope.go:122] "RemoveContainer" containerID="0beef2c614343479971df943571b5bc6e4098feb43b2dfe9d7458863f85ddb2b" Oct 30 16:56:55.192681 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:55.192664 4493 scope.go:122] "RemoveContainer" containerID="ffd4fea2bc5ca6f425eb0ff20e92deff8a8e2ee9bf96e851e5de334134fbb592" Oct 30 16:56:55.202109 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:55.202086 4493 scope.go:122] "RemoveContainer" containerID="e82892c68529dec976a7682a0721abe2c9c446cbaf46725b596803b04e3704e3" Oct 30 16:56:55.261053 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:55.261026 4493 scope.go:122] "RemoveContainer" containerID="1ca614629d6706ef833fd26e9049bc86994a369bc04fd06e314541917323eeb5" Oct 30 16:56:55.273598 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:55.273575 4493 scope.go:122] "RemoveContainer" containerID="a8b5ec593fe2f5819965453b8d83849adf92e05712cc8ec0494e3dca6a5de649" Oct 30 16:56:55.284128 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:55.284104 4493 scope.go:122] "RemoveContainer" containerID="113eac77e74f169721b5659bf4dc1a0962caeb41bfed1eb9375ae6fa2fa7b8e6" Oct 30 16:56:55.293584 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:55.293565 4493 scope.go:122] "RemoveContainer" containerID="233dbd8fcc7ae5adc593d69826773139334afb62532787f1527fa21b4d81c75c" Oct 30 16:56:55.303293 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:55.303277 4493 scope.go:122] "RemoveContainer" containerID="58bf3ff75e86d78ed28aef17b44a3f28b2dca7f85a25669d2bdddc2b8aa53d33" Oct 30 16:56:55.346481 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:55.346454 4493 patch_prober.go:29] interesting pod/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Oct 30 16:56:55.346589 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:55.346516 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx" podUID="1d917fe1-dd87-48a4-94ff-75ece269728f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Oct 30 16:56:56.036006 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:56.035966 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad1ec56f-l84rm"] Oct 30 16:56:56.039901 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:56.039874 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad1ec56f-l84rm"] Oct 30 16:56:56.351646 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:56.351563 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx" Oct 30 16:56:56.352077 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:56.351982 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx" Oct 30 16:56:56.352568 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:56.352550 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx" Oct 30 16:56:56.935857 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:56.935821 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="522ee45b-15af-4129-afea-eb8b8f1c8537" path="/var/lib/kubelet/pods/522ee45b-15af-4129-afea-eb8b8f1c8537/volumes" Oct 30 16:56:59.046324 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:59.046294 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bec93343-dwl79"] Oct 30 16:56:59.051963 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:56:59.051935 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bec93343-dwl79"] Oct 30 16:57:00.341497 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:57:00.341459 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 16:57:00.957840 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:57:00.957799 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59982bb0-0b3c-4bf0-ba41-248f56729a22" path="/var/lib/kubelet/pods/59982bb0-0b3c-4bf0-ba41-248f56729a22/volumes" Oct 30 16:57:07.040485 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:57:07.040446 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6035e9fa-rcvnx"] Oct 30 16:57:07.051721 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:57:07.051684 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6035e9fa-rcvnx"] Oct 30 16:57:08.935331 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:57:08.935291 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb241152-8f3a-4d18-a038-7b1b663c62c3" path="/var/lib/kubelet/pods/fb241152-8f3a-4d18-a038-7b1b663c62c3/volumes" Oct 30 16:57:09.293200 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:57:09.293115 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 16:57:15.032395 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:57:15.032364 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-280c83a4-wrzkk"] Oct 30 16:57:15.043723 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:57:15.043700 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-280c83a4-wrzkk"] Oct 30 16:57:16.933567 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:57:16.933531 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="668dacd9-5a6e-48df-ab0c-f777d9dbe3c7" path="/var/lib/kubelet/pods/668dacd9-5a6e-48df-ab0c-f777d9dbe3c7/volumes" Oct 30 16:57:26.034980 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:57:26.034947 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01e76a8b-slkgf"] Oct 30 16:57:26.046626 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:57:26.046595 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01e76a8b-slkgf"] Oct 30 16:57:26.938610 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:57:26.938572 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cb7aa16-13ef-4b08-a9ec-62be7dfb5c3f" path="/var/lib/kubelet/pods/7cb7aa16-13ef-4b08-a9ec-62be7dfb5c3f/volumes" Oct 30 16:57:30.345053 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:57:30.345021 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 16:57:33.033253 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:57:33.033220 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46c3c12a-4cd8v"] Oct 30 16:57:33.044977 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:57:33.044953 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46c3c12a-4cd8v"] Oct 30 16:57:34.032022 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:57:34.031989 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b595920-sbs2r"] Oct 30 16:57:34.043451 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:57:34.043426 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b595920-sbs2r"] Oct 30 16:57:34.936723 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:57:34.936684 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bca5b866-90c9-484e-b043-e048c3ca2c23" path="/var/lib/kubelet/pods/bca5b866-90c9-484e-b043-e048c3ca2c23/volumes" Oct 30 16:57:34.937382 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:57:34.937358 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fff2bfc7-3c82-4a8f-a726-100912be8523" path="/var/lib/kubelet/pods/fff2bfc7-3c82-4a8f-a726-100912be8523/volumes" Oct 30 16:57:40.035182 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:57:40.035139 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39cda64b-6zghk"] Oct 30 16:57:40.044919 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:57:40.044892 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39cda64b-6zghk"] Oct 30 16:57:40.934530 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:57:40.934499 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fdbeb20-d0c6-4f03-a849-f05582689593" path="/var/lib/kubelet/pods/3fdbeb20-d0c6-4f03-a849-f05582689593/volumes" Oct 30 16:57:46.467211 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:57:46.467174 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-77f4f4-8hnb8"] Oct 30 16:57:46.467771 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:57:46.467757 4493 cpu_manager.go:414] "containerMap: removing container" podUID="3d340ae1-f1e3-4b62-a1b2-8555489dc185" containerName="revoke-keys" Oct 30 16:57:46.467847 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:57:46.467772 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d340ae1-f1e3-4b62-a1b2-8555489dc185" containerName="revoke-keys" Oct 30 16:57:46.467896 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:57:46.467886 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d340ae1-f1e3-4b62-a1b2-8555489dc185" containerName="revoke-keys" Oct 30 16:57:46.471203 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:57:46.471184 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-77f4f4-8hnb8" Oct 30 16:57:46.485656 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:57:46.485628 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 16:57:46.497411 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:57:46.497378 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:57:46.497545 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:57:46.497469 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:57:46.497545 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:57:46.497495 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:57:46.501875 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:57:46.501851 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-77f4f4-8hnb8"] Oct 30 16:57:46.586979 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:57:46.586942 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c0b2750-993b-4e0d-9acb-906371d6a2a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-77f4f4-8hnb8\" (UID: \"1c0b2750-993b-4e0d-9acb-906371d6a2a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-77f4f4-8hnb8" Oct 30 16:57:46.587181 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:57:46.587038 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c0b2750-993b-4e0d-9acb-906371d6a2a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-77f4f4-8hnb8\" (UID: \"1c0b2750-993b-4e0d-9acb-906371d6a2a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-77f4f4-8hnb8" Oct 30 16:57:46.587181 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:57:46.587069 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qqw5\" (UniqueName: \"kubernetes.io/projected/1c0b2750-993b-4e0d-9acb-906371d6a2a3-kube-api-access-2qqw5\") pod \"maas-api-revoke-keys-e2e-praxis-b-77f4f4-8hnb8\" (UID: \"1c0b2750-993b-4e0d-9acb-906371d6a2a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-77f4f4-8hnb8" Oct 30 16:57:46.688721 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:57:46.688674 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c0b2750-993b-4e0d-9acb-906371d6a2a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-77f4f4-8hnb8\" (UID: \"1c0b2750-993b-4e0d-9acb-906371d6a2a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-77f4f4-8hnb8" Oct 30 16:57:46.688916 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:57:46.688765 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c0b2750-993b-4e0d-9acb-906371d6a2a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-77f4f4-8hnb8\" (UID: \"1c0b2750-993b-4e0d-9acb-906371d6a2a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-77f4f4-8hnb8" Oct 30 16:57:46.688916 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:57:46.688806 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qqw5\" (UniqueName: \"kubernetes.io/projected/1c0b2750-993b-4e0d-9acb-906371d6a2a3-kube-api-access-2qqw5\") pod \"maas-api-revoke-keys-e2e-praxis-b-77f4f4-8hnb8\" (UID: \"1c0b2750-993b-4e0d-9acb-906371d6a2a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-77f4f4-8hnb8" Oct 30 16:57:46.689479 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:57:46.689450 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c0b2750-993b-4e0d-9acb-906371d6a2a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-77f4f4-8hnb8\" (UID: \"1c0b2750-993b-4e0d-9acb-906371d6a2a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-77f4f4-8hnb8" Oct 30 16:57:46.691254 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:57:46.691229 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c0b2750-993b-4e0d-9acb-906371d6a2a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-77f4f4-8hnb8\" (UID: \"1c0b2750-993b-4e0d-9acb-906371d6a2a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-77f4f4-8hnb8" Oct 30 16:57:46.714408 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:57:46.714380 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qqw5\" (UniqueName: \"kubernetes.io/projected/1c0b2750-993b-4e0d-9acb-906371d6a2a3-kube-api-access-2qqw5\") pod \"maas-api-revoke-keys-e2e-praxis-b-77f4f4-8hnb8\" (UID: \"1c0b2750-993b-4e0d-9acb-906371d6a2a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-77f4f4-8hnb8" Oct 30 16:57:46.785132 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:57:46.785052 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-77f4f4-8hnb8" Oct 30 16:57:47.032421 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:57:47.032387 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb8208a8-f5gvb"] Oct 30 16:57:47.035849 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:57:47.035767 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb8208a8-f5gvb"] Oct 30 16:57:47.126840 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:57:47.125347 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-77f4f4-8hnb8"] Oct 30 16:57:47.802901 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:57:47.802771 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-77f4f4-8hnb8" event={"ID":"1c0b2750-993b-4e0d-9acb-906371d6a2a3","Type":"ContainerStarted","Data":"6fd139d3315fd40a540805bd12e40bd5f8d6eb403d08a6b15e8afabb34efa364"} Oct 30 16:57:47.802901 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:57:47.802852 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-77f4f4-8hnb8" event={"ID":"1c0b2750-993b-4e0d-9acb-906371d6a2a3","Type":"ContainerStarted","Data":"ca0becb101d52fe551831773a9531bd31b581bdfafc14c81ec9395d67defa1e1"} Oct 30 16:57:47.850264 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:57:47.850197 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-77f4f4-8hnb8" podStartSLOduration=1.8501782100000002 podStartE2EDuration="1.85017821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:57:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:57:47.848499775 +0000 UTC m=+2095.075674876" watchObservedRunningTime="2026-10-30 16:57:47.85017821 +0000 UTC m=+2095.077353311" Oct 30 16:57:48.398203 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:57:48.398166 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 16:57:48.814013 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:57:48.813920 4493 generic.go:356] "Generic (PLEG): container finished" podID="1c0b2750-993b-4e0d-9acb-906371d6a2a3" containerID="6fd139d3315fd40a540805bd12e40bd5f8d6eb403d08a6b15e8afabb34efa364" exitCode=0 Oct 30 16:57:48.814488 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:57:48.814015 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-77f4f4-8hnb8" event={"ID":"1c0b2750-993b-4e0d-9acb-906371d6a2a3","Type":"ContainerDied","Data":"6fd139d3315fd40a540805bd12e40bd5f8d6eb403d08a6b15e8afabb34efa364"} Oct 30 16:57:48.935325 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:57:48.935283 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee653e75-3a28-4c94-8e13-a1c1d664f708" path="/var/lib/kubelet/pods/ee653e75-3a28-4c94-8e13-a1c1d664f708/volumes" Oct 30 16:57:49.978480 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:57:49.978450 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-77f4f4-8hnb8" Oct 30 16:57:50.039519 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:57:50.039470 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5f04f3b-ppcrb"] Oct 30 16:57:50.043394 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:57:50.043368 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5f04f3b-ppcrb"] Oct 30 16:57:50.068160 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:57:50.068131 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c0b2750-993b-4e0d-9acb-906371d6a2a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c0b2750-993b-4e0d-9acb-906371d6a2a3-maas-api-xks-ca\") pod \"1c0b2750-993b-4e0d-9acb-906371d6a2a3\" (UID: \"1c0b2750-993b-4e0d-9acb-906371d6a2a3\") " Oct 30 16:57:50.068346 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:57:50.068184 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c0b2750-993b-4e0d-9acb-906371d6a2a3-kube-api-access-2qqw5\" (UniqueName: \"kubernetes.io/projected/1c0b2750-993b-4e0d-9acb-906371d6a2a3-kube-api-access-2qqw5\") pod \"1c0b2750-993b-4e0d-9acb-906371d6a2a3\" (UID: \"1c0b2750-993b-4e0d-9acb-906371d6a2a3\") " Oct 30 16:57:50.068477 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:57:50.068383 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c0b2750-993b-4e0d-9acb-906371d6a2a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c0b2750-993b-4e0d-9acb-906371d6a2a3-maas-api-openshift-ca\") pod \"1c0b2750-993b-4e0d-9acb-906371d6a2a3\" (UID: \"1c0b2750-993b-4e0d-9acb-906371d6a2a3\") " Oct 30 16:57:50.068846 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:57:50.068816 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c0b2750-993b-4e0d-9acb-906371d6a2a3-maas-api-openshift-ca" pod "1c0b2750-993b-4e0d-9acb-906371d6a2a3" (UID: "1c0b2750-993b-4e0d-9acb-906371d6a2a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:57:50.069475 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:57:50.069446 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c0b2750-993b-4e0d-9acb-906371d6a2a3-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:57:50.070617 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:57:50.070583 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c0b2750-993b-4e0d-9acb-906371d6a2a3-maas-api-xks-ca" pod "1c0b2750-993b-4e0d-9acb-906371d6a2a3" (UID: "1c0b2750-993b-4e0d-9acb-906371d6a2a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:57:50.071152 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:57:50.071097 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c0b2750-993b-4e0d-9acb-906371d6a2a3-kube-api-access-2qqw5" pod "1c0b2750-993b-4e0d-9acb-906371d6a2a3" (UID: "1c0b2750-993b-4e0d-9acb-906371d6a2a3"). InnerVolumeSpecName "kube-api-access-2qqw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:57:50.170417 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:57:50.170385 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qqw5\" (UniqueName: \"kubernetes.io/projected/1c0b2750-993b-4e0d-9acb-906371d6a2a3-kube-api-access-2qqw5\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:57:50.170417 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:57:50.170415 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c0b2750-993b-4e0d-9acb-906371d6a2a3-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:57:50.833197 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:57:50.833165 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-77f4f4-8hnb8" Oct 30 16:57:50.833417 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:57:50.833191 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-77f4f4-8hnb8" event={"ID":"1c0b2750-993b-4e0d-9acb-906371d6a2a3","Type":"ContainerDied","Data":"ca0becb101d52fe551831773a9531bd31b581bdfafc14c81ec9395d67defa1e1"} Oct 30 16:57:50.833417 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:57:50.833248 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca0becb101d52fe551831773a9531bd31b581bdfafc14c81ec9395d67defa1e1" Oct 30 16:57:50.934643 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:57:50.934597 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cc6b894-299f-474a-ba77-65ccbe882cc6" path="/var/lib/kubelet/pods/0cc6b894-299f-474a-ba77-65ccbe882cc6/volumes" Oct 30 16:57:54.603006 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:57:54.602977 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-56v9s_d3a1c417-2698-4462-b7ff-1d2e66d7a10f/ovn-acl-logging/0.log" Oct 30 16:57:54.610032 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:57:54.610008 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-56v9s_d3a1c417-2698-4462-b7ff-1d2e66d7a10f/ovn-acl-logging/0.log" Oct 30 16:57:55.451872 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:57:55.451838 4493 scope.go:122] "RemoveContainer" containerID="b84f04e469761baad64b62b8e1097d43d06c06ae27efbf37666062c64de42078" Oct 30 16:57:55.462988 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:57:55.462962 4493 scope.go:122] "RemoveContainer" containerID="8de68f126bd1b459ed3c43a6310b564e99dc79ed80c91d27b09076b33a7bbb44" Oct 30 16:57:55.472621 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:57:55.472604 4493 scope.go:122] "RemoveContainer" containerID="9c10f2462a70774ed7e02c54b83598df03416975d05a4034159014ba2a8c0d6e" Oct 30 16:57:55.481776 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:57:55.481758 4493 scope.go:122] "RemoveContainer" containerID="3447ab92ffdcd82a6fb992492283fd884cd46f1fe1cecd4bdbb33dd3e251eb46" Oct 30 16:57:55.491534 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:57:55.491511 4493 scope.go:122] "RemoveContainer" containerID="373273ad35cffe31981a317f7edf4c2633f0bff5c7f7c9e0c408db58b9f39a62" Oct 30 16:57:55.500558 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:57:55.500543 4493 scope.go:122] "RemoveContainer" containerID="fc3dfd491653aa8fd428f419d8d36ff4893675b5d9a0a9e11a87e0d84b808f46" Oct 30 16:57:55.511742 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:57:55.511722 4493 scope.go:122] "RemoveContainer" containerID="debe275a684c902752a54cea4f438ce72b77ea3792559f99fc73716907729cc0" Oct 30 16:57:55.522710 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:57:55.522681 4493 scope.go:122] "RemoveContainer" containerID="757cb325a3947328a8b8bbabcd6680aeec959e3f7c82b92b972cd25b8b687a39" Oct 30 16:57:55.532026 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:57:55.531999 4493 scope.go:122] "RemoveContainer" containerID="e218ffb6a33b42ae4fa3f0251d39d8a776b86fa8ed6078b596cbc38f04d39b45" Oct 30 16:57:55.544445 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:57:55.544420 4493 scope.go:122] "RemoveContainer" containerID="edab1a6f61467248a4b6bae760baf4db511f39601bf2fb4fcc9d4955adfebfe3" Oct 30 16:57:55.556681 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:57:55.556659 4493 scope.go:122] "RemoveContainer" containerID="b53e6e8bd98fb0f91fa6a2ffb1f61ccafd51cecf7efdc816c15aa65ee26139a4" Oct 30 16:58:00.341740 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:58:00.341704 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 16:58:08.035237 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:08.035164 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-624df655-smltd"] Oct 30 16:58:08.046874 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:08.046843 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-624df655-smltd"] Oct 30 16:58:08.934255 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:08.934221 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f625b78-72c7-434e-ada0-07b6c1a9ccea" path="/var/lib/kubelet/pods/0f625b78-72c7-434e-ada0-07b6c1a9ccea/volumes" Oct 30 16:58:11.468303 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:11.468268 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-77f4f4-8hnb8"] Oct 30 16:58:11.494342 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:11.494295 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-77f4f4-8hnb8"] Oct 30 16:58:12.937196 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:12.937162 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c0b2750-993b-4e0d-9acb-906371d6a2a3" path="/var/lib/kubelet/pods/1c0b2750-993b-4e0d-9acb-906371d6a2a3/volumes" Oct 30 16:58:16.189675 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:16.189642 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6n8m6"] Oct 30 16:58:16.190282 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:16.189876 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6n8m6" podUID="5903f4b2-9604-4eab-adf5-e0f7452ffbba" containerName="istio-proxy" containerID="cri-o://c9f797eaa2ae4e5a5a7be88173050e9a96417c80a640f384910aebe19b4ea7f7" gracePeriod=30 Oct 30 16:58:16.765322 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:16.765254 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-6n8m6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 16:58:16.765475 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:16.765319 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6n8m6" podUID="5903f4b2-9604-4eab-adf5-e0f7452ffbba" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 16:58:21.496298 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:21.496274 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6n8m6" Oct 30 16:58:21.621268 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:21.621228 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5903f4b2-9604-4eab-adf5-e0f7452ffbba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5903f4b2-9604-4eab-adf5-e0f7452ffbba-workload-certs\") pod \"5903f4b2-9604-4eab-adf5-e0f7452ffbba\" (UID: \"5903f4b2-9604-4eab-adf5-e0f7452ffbba\") " Oct 30 16:58:21.621458 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:21.621343 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5903f4b2-9604-4eab-adf5-e0f7452ffbba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5903f4b2-9604-4eab-adf5-e0f7452ffbba-credential-socket\") pod \"5903f4b2-9604-4eab-adf5-e0f7452ffbba\" (UID: \"5903f4b2-9604-4eab-adf5-e0f7452ffbba\") " Oct 30 16:58:21.621458 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:21.621399 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5903f4b2-9604-4eab-adf5-e0f7452ffbba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5903f4b2-9604-4eab-adf5-e0f7452ffbba-istio-envoy\") pod \"5903f4b2-9604-4eab-adf5-e0f7452ffbba\" (UID: \"5903f4b2-9604-4eab-adf5-e0f7452ffbba\") " Oct 30 16:58:21.621540 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:21.621500 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5903f4b2-9604-4eab-adf5-e0f7452ffbba-kube-api-access-j8c7h\" (UniqueName: \"kubernetes.io/projected/5903f4b2-9604-4eab-adf5-e0f7452ffbba-kube-api-access-j8c7h\") pod \"5903f4b2-9604-4eab-adf5-e0f7452ffbba\" (UID: \"5903f4b2-9604-4eab-adf5-e0f7452ffbba\") " Oct 30 16:58:21.621540 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:21.621528 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5903f4b2-9604-4eab-adf5-e0f7452ffbba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5903f4b2-9604-4eab-adf5-e0f7452ffbba-workload-socket\") pod \"5903f4b2-9604-4eab-adf5-e0f7452ffbba\" (UID: \"5903f4b2-9604-4eab-adf5-e0f7452ffbba\") " Oct 30 16:58:21.621628 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:21.621553 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5903f4b2-9604-4eab-adf5-e0f7452ffbba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5903f4b2-9604-4eab-adf5-e0f7452ffbba-istio-data\") pod \"5903f4b2-9604-4eab-adf5-e0f7452ffbba\" (UID: \"5903f4b2-9604-4eab-adf5-e0f7452ffbba\") " Oct 30 16:58:21.621628 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:21.621611 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5903f4b2-9604-4eab-adf5-e0f7452ffbba-credential-socket" pod "5903f4b2-9604-4eab-adf5-e0f7452ffbba" (UID: "5903f4b2-9604-4eab-adf5-e0f7452ffbba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:58:21.621741 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:21.621636 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5903f4b2-9604-4eab-adf5-e0f7452ffbba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5903f4b2-9604-4eab-adf5-e0f7452ffbba-istio-podinfo\") pod \"5903f4b2-9604-4eab-adf5-e0f7452ffbba\" (UID: \"5903f4b2-9604-4eab-adf5-e0f7452ffbba\") " Oct 30 16:58:21.621741 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:21.621685 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5903f4b2-9604-4eab-adf5-e0f7452ffbba-istio-token\" (UniqueName: \"kubernetes.io/projected/5903f4b2-9604-4eab-adf5-e0f7452ffbba-istio-token\") pod \"5903f4b2-9604-4eab-adf5-e0f7452ffbba\" (UID: \"5903f4b2-9604-4eab-adf5-e0f7452ffbba\") " Oct 30 16:58:21.621881 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:21.621849 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5903f4b2-9604-4eab-adf5-e0f7452ffbba-istio-data" pod "5903f4b2-9604-4eab-adf5-e0f7452ffbba" (UID: "5903f4b2-9604-4eab-adf5-e0f7452ffbba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:58:21.621881 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:21.621861 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5903f4b2-9604-4eab-adf5-e0f7452ffbba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5903f4b2-9604-4eab-adf5-e0f7452ffbba-istiod-ca-cert\") pod \"5903f4b2-9604-4eab-adf5-e0f7452ffbba\" (UID: \"5903f4b2-9604-4eab-adf5-e0f7452ffbba\") " Oct 30 16:58:21.622013 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:21.621551 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5903f4b2-9604-4eab-adf5-e0f7452ffbba-workload-certs" pod "5903f4b2-9604-4eab-adf5-e0f7452ffbba" (UID: "5903f4b2-9604-4eab-adf5-e0f7452ffbba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:58:21.622255 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:21.622226 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5903f4b2-9604-4eab-adf5-e0f7452ffbba-workload-socket" pod "5903f4b2-9604-4eab-adf5-e0f7452ffbba" (UID: "5903f4b2-9604-4eab-adf5-e0f7452ffbba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:58:21.622435 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:21.622413 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5903f4b2-9604-4eab-adf5-e0f7452ffbba-istio-data\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:58:21.622535 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:21.622441 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5903f4b2-9604-4eab-adf5-e0f7452ffbba-workload-certs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:58:21.622535 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:21.622457 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5903f4b2-9604-4eab-adf5-e0f7452ffbba-credential-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:58:21.622535 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:21.622412 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5903f4b2-9604-4eab-adf5-e0f7452ffbba-istiod-ca-cert" pod "5903f4b2-9604-4eab-adf5-e0f7452ffbba" (UID: "5903f4b2-9604-4eab-adf5-e0f7452ffbba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:58:21.624069 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:21.624043 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5903f4b2-9604-4eab-adf5-e0f7452ffbba-kube-api-access-j8c7h" pod "5903f4b2-9604-4eab-adf5-e0f7452ffbba" (UID: "5903f4b2-9604-4eab-adf5-e0f7452ffbba"). InnerVolumeSpecName "kube-api-access-j8c7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:58:21.624193 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:21.624120 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5903f4b2-9604-4eab-adf5-e0f7452ffbba-istio-envoy" pod "5903f4b2-9604-4eab-adf5-e0f7452ffbba" (UID: "5903f4b2-9604-4eab-adf5-e0f7452ffbba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:58:21.624894 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:21.624869 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5903f4b2-9604-4eab-adf5-e0f7452ffbba-istio-token" pod "5903f4b2-9604-4eab-adf5-e0f7452ffbba" (UID: "5903f4b2-9604-4eab-adf5-e0f7452ffbba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:58:21.624993 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:21.624924 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5903f4b2-9604-4eab-adf5-e0f7452ffbba-istio-podinfo" pod "5903f4b2-9604-4eab-adf5-e0f7452ffbba" (UID: "5903f4b2-9604-4eab-adf5-e0f7452ffbba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:58:21.699077 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:21.699046 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e09cf108-z22kj"] Oct 30 16:58:21.699626 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:21.699613 4493 cpu_manager.go:414] "containerMap: removing container" podUID="5903f4b2-9604-4eab-adf5-e0f7452ffbba" containerName="istio-proxy" Oct 30 16:58:21.699681 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:21.699628 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5903f4b2-9604-4eab-adf5-e0f7452ffbba" containerName="istio-proxy" Oct 30 16:58:21.699681 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:21.699649 4493 cpu_manager.go:414] "containerMap: removing container" podUID="1c0b2750-993b-4e0d-9acb-906371d6a2a3" containerName="revoke-keys" Oct 30 16:58:21.699681 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:21.699653 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c0b2750-993b-4e0d-9acb-906371d6a2a3" containerName="revoke-keys" Oct 30 16:58:21.699681 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:21.699665 4493 cpu_manager.go:414] "containerMap: removing container" podUID="5903f4b2-9604-4eab-adf5-e0f7452ffbba" containerName="istio-proxy" Oct 30 16:58:21.699681 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:21.699670 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5903f4b2-9604-4eab-adf5-e0f7452ffbba" containerName="istio-proxy" Oct 30 16:58:21.699895 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:21.699764 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="5903f4b2-9604-4eab-adf5-e0f7452ffbba" containerName="istio-proxy" Oct 30 16:58:21.699895 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:21.699775 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c0b2750-993b-4e0d-9acb-906371d6a2a3" containerName="revoke-keys" Oct 30 16:58:21.699895 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:21.699796 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="5903f4b2-9604-4eab-adf5-e0f7452ffbba" containerName="istio-proxy" Oct 30 16:58:21.703448 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:21.703431 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09cf108-z22kj" Oct 30 16:58:21.718725 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:21.718677 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:58:21.718911 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:21.718896 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 16:58:21.720216 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:21.720197 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:58:21.722758 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:21.722742 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:58:21.723419 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:21.723402 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5903f4b2-9604-4eab-adf5-e0f7452ffbba-istio-token\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:58:21.723458 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:21.723424 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5903f4b2-9604-4eab-adf5-e0f7452ffbba-istiod-ca-cert\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:58:21.723458 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:21.723433 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5903f4b2-9604-4eab-adf5-e0f7452ffbba-istio-envoy\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:58:21.723458 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:21.723440 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8c7h\" (UniqueName: \"kubernetes.io/projected/5903f4b2-9604-4eab-adf5-e0f7452ffbba-kube-api-access-j8c7h\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:58:21.723458 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:21.723448 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5903f4b2-9604-4eab-adf5-e0f7452ffbba-workload-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:58:21.723458 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:21.723458 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5903f4b2-9604-4eab-adf5-e0f7452ffbba-istio-podinfo\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:58:21.732953 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:21.732928 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e09cf108-z22kj"] Oct 30 16:58:21.824354 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:21.824323 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fbc97f7-5a3b-440a-af9f-628fba76f84c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e09cf108-z22kj\" (UID: \"0fbc97f7-5a3b-440a-af9f-628fba76f84c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09cf108-z22kj" Oct 30 16:58:21.824483 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:21.824357 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnl8c\" (UniqueName: \"kubernetes.io/projected/0fbc97f7-5a3b-440a-af9f-628fba76f84c-kube-api-access-qnl8c\") pod \"maas-api-revoke-sub-e09cf108-z22kj\" (UID: \"0fbc97f7-5a3b-440a-af9f-628fba76f84c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09cf108-z22kj" Oct 30 16:58:21.824598 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:21.824577 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fbc97f7-5a3b-440a-af9f-628fba76f84c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e09cf108-z22kj\" (UID: \"0fbc97f7-5a3b-440a-af9f-628fba76f84c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09cf108-z22kj" Oct 30 16:58:21.926311 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:21.926277 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fbc97f7-5a3b-440a-af9f-628fba76f84c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e09cf108-z22kj\" (UID: \"0fbc97f7-5a3b-440a-af9f-628fba76f84c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09cf108-z22kj" Oct 30 16:58:21.926482 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:21.926364 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fbc97f7-5a3b-440a-af9f-628fba76f84c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e09cf108-z22kj\" (UID: \"0fbc97f7-5a3b-440a-af9f-628fba76f84c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09cf108-z22kj" Oct 30 16:58:21.926482 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:21.926398 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnl8c\" (UniqueName: \"kubernetes.io/projected/0fbc97f7-5a3b-440a-af9f-628fba76f84c-kube-api-access-qnl8c\") pod \"maas-api-revoke-sub-e09cf108-z22kj\" (UID: \"0fbc97f7-5a3b-440a-af9f-628fba76f84c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09cf108-z22kj" Oct 30 16:58:21.927096 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:21.927057 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fbc97f7-5a3b-440a-af9f-628fba76f84c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e09cf108-z22kj\" (UID: \"0fbc97f7-5a3b-440a-af9f-628fba76f84c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09cf108-z22kj" Oct 30 16:58:21.928973 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:21.928954 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fbc97f7-5a3b-440a-af9f-628fba76f84c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e09cf108-z22kj\" (UID: \"0fbc97f7-5a3b-440a-af9f-628fba76f84c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09cf108-z22kj" Oct 30 16:58:21.950252 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:21.950231 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnl8c\" (UniqueName: \"kubernetes.io/projected/0fbc97f7-5a3b-440a-af9f-628fba76f84c-kube-api-access-qnl8c\") pod \"maas-api-revoke-sub-e09cf108-z22kj\" (UID: \"0fbc97f7-5a3b-440a-af9f-628fba76f84c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09cf108-z22kj" Oct 30 16:58:22.018310 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:22.018245 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09cf108-z22kj" Oct 30 16:58:22.121126 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:22.121093 4493 generic.go:356] "Generic (PLEG): container finished" podID="5903f4b2-9604-4eab-adf5-e0f7452ffbba" containerID="c9f797eaa2ae4e5a5a7be88173050e9a96417c80a640f384910aebe19b4ea7f7" exitCode=0 Oct 30 16:58:22.121283 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:22.121155 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6n8m6" event={"ID":"5903f4b2-9604-4eab-adf5-e0f7452ffbba","Type":"ContainerDied","Data":"c9f797eaa2ae4e5a5a7be88173050e9a96417c80a640f384910aebe19b4ea7f7"} Oct 30 16:58:22.121283 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:22.121203 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6n8m6" event={"ID":"5903f4b2-9604-4eab-adf5-e0f7452ffbba","Type":"ContainerDied","Data":"da0f9ee7da7bd7ff4b7662b6203787976a4eb8fe04e772c9426081cbf57161b7"} Oct 30 16:58:22.121283 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:22.121232 4493 scope.go:122] "RemoveContainer" containerID="c9f797eaa2ae4e5a5a7be88173050e9a96417c80a640f384910aebe19b4ea7f7" Oct 30 16:58:22.121421 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:22.121409 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6n8m6" Oct 30 16:58:22.140014 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:22.139988 4493 scope.go:122] "RemoveContainer" containerID="0106d84270cd88a079caf167aba8d8c7db014d9244c9dc23c718710171feaf7f" Oct 30 16:58:22.153766 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:22.153744 4493 scope.go:122] "RemoveContainer" containerID="c9f797eaa2ae4e5a5a7be88173050e9a96417c80a640f384910aebe19b4ea7f7" Oct 30 16:58:22.154100 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:58:22.154079 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9f797eaa2ae4e5a5a7be88173050e9a96417c80a640f384910aebe19b4ea7f7\": container with ID starting with c9f797eaa2ae4e5a5a7be88173050e9a96417c80a640f384910aebe19b4ea7f7 not found: ID does not exist" containerID="c9f797eaa2ae4e5a5a7be88173050e9a96417c80a640f384910aebe19b4ea7f7" Oct 30 16:58:22.154203 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:22.154110 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9f797eaa2ae4e5a5a7be88173050e9a96417c80a640f384910aebe19b4ea7f7"} err="failed to get container status \"c9f797eaa2ae4e5a5a7be88173050e9a96417c80a640f384910aebe19b4ea7f7\": rpc error: code = NotFound desc = could not find container \"c9f797eaa2ae4e5a5a7be88173050e9a96417c80a640f384910aebe19b4ea7f7\": container with ID starting with c9f797eaa2ae4e5a5a7be88173050e9a96417c80a640f384910aebe19b4ea7f7 not found: ID does not exist" Oct 30 16:58:22.154203 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:22.154133 4493 scope.go:122] "RemoveContainer" containerID="0106d84270cd88a079caf167aba8d8c7db014d9244c9dc23c718710171feaf7f" Oct 30 16:58:22.154411 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:58:22.154390 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0106d84270cd88a079caf167aba8d8c7db014d9244c9dc23c718710171feaf7f\": container with ID starting with 0106d84270cd88a079caf167aba8d8c7db014d9244c9dc23c718710171feaf7f not found: ID does not exist" containerID="0106d84270cd88a079caf167aba8d8c7db014d9244c9dc23c718710171feaf7f" Oct 30 16:58:22.154496 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:22.154418 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0106d84270cd88a079caf167aba8d8c7db014d9244c9dc23c718710171feaf7f"} err="failed to get container status \"0106d84270cd88a079caf167aba8d8c7db014d9244c9dc23c718710171feaf7f\": rpc error: code = NotFound desc = could not find container \"0106d84270cd88a079caf167aba8d8c7db014d9244c9dc23c718710171feaf7f\": container with ID starting with 0106d84270cd88a079caf167aba8d8c7db014d9244c9dc23c718710171feaf7f not found: ID does not exist" Oct 30 16:58:22.182024 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:22.181999 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6n8m6"] Oct 30 16:58:22.188531 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:22.188508 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6n8m6"] Oct 30 16:58:22.327019 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:22.326985 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e09cf108-z22kj"] Oct 30 16:58:22.328420 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:58:22.328388 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fbc97f7_5a3b_440a_af9f_628fba76f84c.slice/crio-0664fe7e1309bfb64e33574eddebd0085673e561ba2f4553517e8bf5cc0bf4bf WatchSource:0}: Error finding container 0664fe7e1309bfb64e33574eddebd0085673e561ba2f4553517e8bf5cc0bf4bf: Status 404 returned error can't find the container with id 0664fe7e1309bfb64e33574eddebd0085673e561ba2f4553517e8bf5cc0bf4bf Oct 30 16:58:22.945028 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:22.944994 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5903f4b2-9604-4eab-adf5-e0f7452ffbba" path="/var/lib/kubelet/pods/5903f4b2-9604-4eab-adf5-e0f7452ffbba/volumes" Oct 30 16:58:23.138775 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:23.138734 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09cf108-z22kj" event={"ID":"0fbc97f7-5a3b-440a-af9f-628fba76f84c","Type":"ContainerStarted","Data":"c218947ab011597840bde8598c1ec55ace97574b7cad4e41d565bf459f5e27a0"} Oct 30 16:58:23.139002 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:23.138838 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09cf108-z22kj" event={"ID":"0fbc97f7-5a3b-440a-af9f-628fba76f84c","Type":"ContainerStarted","Data":"0664fe7e1309bfb64e33574eddebd0085673e561ba2f4553517e8bf5cc0bf4bf"} Oct 30 16:58:23.182248 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:23.182175 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09cf108-z22kj" podStartSLOduration=2.18215518 podStartE2EDuration="2.18215518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:58:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:58:23.181151227 +0000 UTC m=+2130.408326328" watchObservedRunningTime="2026-10-30 16:58:23.18215518 +0000 UTC m=+2130.409330281" Oct 30 16:58:24.151650 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:24.151607 4493 generic.go:356] "Generic (PLEG): container finished" podID="0fbc97f7-5a3b-440a-af9f-628fba76f84c" containerID="c218947ab011597840bde8598c1ec55ace97574b7cad4e41d565bf459f5e27a0" exitCode=0 Oct 30 16:58:24.152128 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:24.151681 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09cf108-z22kj" event={"ID":"0fbc97f7-5a3b-440a-af9f-628fba76f84c","Type":"ContainerDied","Data":"c218947ab011597840bde8598c1ec55ace97574b7cad4e41d565bf459f5e27a0"} Oct 30 16:58:25.327073 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:25.327041 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09cf108-z22kj" Oct 30 16:58:25.479954 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:25.479865 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fbc97f7-5a3b-440a-af9f-628fba76f84c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fbc97f7-5a3b-440a-af9f-628fba76f84c-maas-api-xks-ca\") pod \"0fbc97f7-5a3b-440a-af9f-628fba76f84c\" (UID: \"0fbc97f7-5a3b-440a-af9f-628fba76f84c\") " Oct 30 16:58:25.480096 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:25.479967 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fbc97f7-5a3b-440a-af9f-628fba76f84c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fbc97f7-5a3b-440a-af9f-628fba76f84c-maas-api-openshift-ca\") pod \"0fbc97f7-5a3b-440a-af9f-628fba76f84c\" (UID: \"0fbc97f7-5a3b-440a-af9f-628fba76f84c\") " Oct 30 16:58:25.480096 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:25.480013 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fbc97f7-5a3b-440a-af9f-628fba76f84c-kube-api-access-qnl8c\" (UniqueName: \"kubernetes.io/projected/0fbc97f7-5a3b-440a-af9f-628fba76f84c-kube-api-access-qnl8c\") pod \"0fbc97f7-5a3b-440a-af9f-628fba76f84c\" (UID: \"0fbc97f7-5a3b-440a-af9f-628fba76f84c\") " Oct 30 16:58:25.480333 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:25.480302 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fbc97f7-5a3b-440a-af9f-628fba76f84c-maas-api-openshift-ca" pod "0fbc97f7-5a3b-440a-af9f-628fba76f84c" (UID: "0fbc97f7-5a3b-440a-af9f-628fba76f84c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:58:25.480611 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:25.480593 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fbc97f7-5a3b-440a-af9f-628fba76f84c-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:58:25.482357 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:25.482333 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fbc97f7-5a3b-440a-af9f-628fba76f84c-maas-api-xks-ca" pod "0fbc97f7-5a3b-440a-af9f-628fba76f84c" (UID: "0fbc97f7-5a3b-440a-af9f-628fba76f84c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:58:25.482451 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:25.482385 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fbc97f7-5a3b-440a-af9f-628fba76f84c-kube-api-access-qnl8c" pod "0fbc97f7-5a3b-440a-af9f-628fba76f84c" (UID: "0fbc97f7-5a3b-440a-af9f-628fba76f84c"). InnerVolumeSpecName "kube-api-access-qnl8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:58:25.582028 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:25.581998 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fbc97f7-5a3b-440a-af9f-628fba76f84c-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:58:25.582028 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:25.582026 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnl8c\" (UniqueName: \"kubernetes.io/projected/0fbc97f7-5a3b-440a-af9f-628fba76f84c-kube-api-access-qnl8c\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:58:26.167766 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:26.167728 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09cf108-z22kj" event={"ID":"0fbc97f7-5a3b-440a-af9f-628fba76f84c","Type":"ContainerDied","Data":"0664fe7e1309bfb64e33574eddebd0085673e561ba2f4553517e8bf5cc0bf4bf"} Oct 30 16:58:26.167766 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:26.167767 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e09cf108-z22kj" Oct 30 16:58:26.168026 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:26.167805 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0664fe7e1309bfb64e33574eddebd0085673e561ba2f4553517e8bf5cc0bf4bf" Oct 30 16:58:30.342502 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:58:30.342458 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 16:58:32.170994 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:32.170962 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4196ea9-8vvfn"] Oct 30 16:58:32.171723 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:32.171701 4493 cpu_manager.go:414] "containerMap: removing container" podUID="0fbc97f7-5a3b-440a-af9f-628fba76f84c" containerName="revoke-keys" Oct 30 16:58:32.171852 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:32.171725 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fbc97f7-5a3b-440a-af9f-628fba76f84c" containerName="revoke-keys" Oct 30 16:58:32.171933 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:32.171920 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fbc97f7-5a3b-440a-af9f-628fba76f84c" containerName="revoke-keys" Oct 30 16:58:32.175543 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:32.175522 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4196ea9-8vvfn" Oct 30 16:58:32.193571 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:32.193550 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfbf22a8-600e-4cbd-a986-53846ab58d02-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4196ea9-8vvfn\" (UID: \"dfbf22a8-600e-4cbd-a986-53846ab58d02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4196ea9-8vvfn" Oct 30 16:58:32.193664 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:32.193591 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfbf22a8-600e-4cbd-a986-53846ab58d02-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4196ea9-8vvfn\" (UID: \"dfbf22a8-600e-4cbd-a986-53846ab58d02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4196ea9-8vvfn" Oct 30 16:58:32.193725 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:32.193707 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zpkr\" (UniqueName: \"kubernetes.io/projected/dfbf22a8-600e-4cbd-a986-53846ab58d02-kube-api-access-9zpkr\") pod \"maas-api-revoke-sub-a4196ea9-8vvfn\" (UID: \"dfbf22a8-600e-4cbd-a986-53846ab58d02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4196ea9-8vvfn" Oct 30 16:58:32.201550 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:32.201506 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:58:32.201550 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:32.201511 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:58:32.201715 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:32.201564 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 16:58:32.201916 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:32.201887 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:58:32.206681 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:32.206657 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4196ea9-8vvfn"] Oct 30 16:58:32.294862 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:32.294829 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zpkr\" (UniqueName: \"kubernetes.io/projected/dfbf22a8-600e-4cbd-a986-53846ab58d02-kube-api-access-9zpkr\") pod \"maas-api-revoke-sub-a4196ea9-8vvfn\" (UID: \"dfbf22a8-600e-4cbd-a986-53846ab58d02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4196ea9-8vvfn" Oct 30 16:58:32.295063 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:32.294975 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfbf22a8-600e-4cbd-a986-53846ab58d02-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4196ea9-8vvfn\" (UID: \"dfbf22a8-600e-4cbd-a986-53846ab58d02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4196ea9-8vvfn" Oct 30 16:58:32.295063 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:32.295018 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfbf22a8-600e-4cbd-a986-53846ab58d02-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4196ea9-8vvfn\" (UID: \"dfbf22a8-600e-4cbd-a986-53846ab58d02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4196ea9-8vvfn" Oct 30 16:58:32.295637 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:32.295617 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfbf22a8-600e-4cbd-a986-53846ab58d02-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4196ea9-8vvfn\" (UID: \"dfbf22a8-600e-4cbd-a986-53846ab58d02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4196ea9-8vvfn" Oct 30 16:58:32.297523 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:32.297504 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfbf22a8-600e-4cbd-a986-53846ab58d02-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4196ea9-8vvfn\" (UID: \"dfbf22a8-600e-4cbd-a986-53846ab58d02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4196ea9-8vvfn" Oct 30 16:58:32.318754 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:32.318727 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zpkr\" (UniqueName: \"kubernetes.io/projected/dfbf22a8-600e-4cbd-a986-53846ab58d02-kube-api-access-9zpkr\") pod \"maas-api-revoke-sub-a4196ea9-8vvfn\" (UID: \"dfbf22a8-600e-4cbd-a986-53846ab58d02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4196ea9-8vvfn" Oct 30 16:58:32.489530 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:32.489432 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4196ea9-8vvfn" Oct 30 16:58:32.797721 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:32.797692 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4196ea9-8vvfn"] Oct 30 16:58:32.799224 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:58:32.799197 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfbf22a8_600e_4cbd_a986_53846ab58d02.slice/crio-7246d558703738355ec952fe6ae93ca99fab4eec0a44dce50f8e5aa2473019ba WatchSource:0}: Error finding container 7246d558703738355ec952fe6ae93ca99fab4eec0a44dce50f8e5aa2473019ba: Status 404 returned error can't find the container with id 7246d558703738355ec952fe6ae93ca99fab4eec0a44dce50f8e5aa2473019ba Oct 30 16:58:33.231709 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:33.231675 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4196ea9-8vvfn" event={"ID":"dfbf22a8-600e-4cbd-a986-53846ab58d02","Type":"ContainerStarted","Data":"f6aa55b50e871993f97da2007ea80c850e9d63ebc1fd3f200c62a2561b136a34"} Oct 30 16:58:33.232230 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:33.231747 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4196ea9-8vvfn" event={"ID":"dfbf22a8-600e-4cbd-a986-53846ab58d02","Type":"ContainerStarted","Data":"7246d558703738355ec952fe6ae93ca99fab4eec0a44dce50f8e5aa2473019ba"} Oct 30 16:58:33.275241 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:33.275193 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4196ea9-8vvfn" podStartSLOduration=1.2751785469999999 podStartE2EDuration="1.275178547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:58:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:58:33.271686709 +0000 UTC m=+2140.498861811" watchObservedRunningTime="2026-10-30 16:58:33.275178547 +0000 UTC m=+2140.502353646" Oct 30 16:58:34.242534 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:34.242486 4493 generic.go:356] "Generic (PLEG): container finished" podID="dfbf22a8-600e-4cbd-a986-53846ab58d02" containerID="f6aa55b50e871993f97da2007ea80c850e9d63ebc1fd3f200c62a2561b136a34" exitCode=0 Oct 30 16:58:34.243023 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:34.242815 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4196ea9-8vvfn" event={"ID":"dfbf22a8-600e-4cbd-a986-53846ab58d02","Type":"ContainerDied","Data":"f6aa55b50e871993f97da2007ea80c850e9d63ebc1fd3f200c62a2561b136a34"} Oct 30 16:58:35.438617 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:35.438584 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4196ea9-8vvfn" Oct 30 16:58:35.462102 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:35.462081 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfbf22a8-600e-4cbd-a986-53846ab58d02-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfbf22a8-600e-4cbd-a986-53846ab58d02-maas-api-openshift-ca\") pod \"dfbf22a8-600e-4cbd-a986-53846ab58d02\" (UID: \"dfbf22a8-600e-4cbd-a986-53846ab58d02\") " Oct 30 16:58:35.462231 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:35.462138 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfbf22a8-600e-4cbd-a986-53846ab58d02-kube-api-access-9zpkr\" (UniqueName: \"kubernetes.io/projected/dfbf22a8-600e-4cbd-a986-53846ab58d02-kube-api-access-9zpkr\") pod \"dfbf22a8-600e-4cbd-a986-53846ab58d02\" (UID: \"dfbf22a8-600e-4cbd-a986-53846ab58d02\") " Oct 30 16:58:35.462307 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:35.462268 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dfbf22a8-600e-4cbd-a986-53846ab58d02-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfbf22a8-600e-4cbd-a986-53846ab58d02-maas-api-xks-ca\") pod \"dfbf22a8-600e-4cbd-a986-53846ab58d02\" (UID: \"dfbf22a8-600e-4cbd-a986-53846ab58d02\") " Oct 30 16:58:35.462397 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:35.462375 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfbf22a8-600e-4cbd-a986-53846ab58d02-maas-api-openshift-ca" pod "dfbf22a8-600e-4cbd-a986-53846ab58d02" (UID: "dfbf22a8-600e-4cbd-a986-53846ab58d02"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:58:35.462621 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:35.462598 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfbf22a8-600e-4cbd-a986-53846ab58d02-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:58:35.464339 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:35.464320 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfbf22a8-600e-4cbd-a986-53846ab58d02-kube-api-access-9zpkr" pod "dfbf22a8-600e-4cbd-a986-53846ab58d02" (UID: "dfbf22a8-600e-4cbd-a986-53846ab58d02"). InnerVolumeSpecName "kube-api-access-9zpkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:58:35.464339 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:35.464324 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfbf22a8-600e-4cbd-a986-53846ab58d02-maas-api-xks-ca" pod "dfbf22a8-600e-4cbd-a986-53846ab58d02" (UID: "dfbf22a8-600e-4cbd-a986-53846ab58d02"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:58:35.564369 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:35.564289 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfbf22a8-600e-4cbd-a986-53846ab58d02-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:58:35.564369 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:35.564319 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zpkr\" (UniqueName: \"kubernetes.io/projected/dfbf22a8-600e-4cbd-a986-53846ab58d02-kube-api-access-9zpkr\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:58:36.261126 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:36.261094 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4196ea9-8vvfn" Oct 30 16:58:36.261126 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:36.261117 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4196ea9-8vvfn" event={"ID":"dfbf22a8-600e-4cbd-a986-53846ab58d02","Type":"ContainerDied","Data":"7246d558703738355ec952fe6ae93ca99fab4eec0a44dce50f8e5aa2473019ba"} Oct 30 16:58:36.261351 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:36.261199 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7246d558703738355ec952fe6ae93ca99fab4eec0a44dce50f8e5aa2473019ba" Oct 30 16:58:38.317239 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:38.317205 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 16:58:42.569511 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:42.569469 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b90ec4e-w76hj"] Oct 30 16:58:42.570112 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:42.570097 4493 cpu_manager.go:414] "containerMap: removing container" podUID="dfbf22a8-600e-4cbd-a986-53846ab58d02" containerName="revoke-keys" Oct 30 16:58:42.570154 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:42.570113 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfbf22a8-600e-4cbd-a986-53846ab58d02" containerName="revoke-keys" Oct 30 16:58:42.570221 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:42.570211 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfbf22a8-600e-4cbd-a986-53846ab58d02" containerName="revoke-keys" Oct 30 16:58:42.573962 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:42.573945 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b90ec4e-w76hj" Oct 30 16:58:42.596636 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:42.596611 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b90ec4e-w76hj"] Oct 30 16:58:42.603411 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:42.603381 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 16:58:42.603565 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:42.603414 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:58:42.603565 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:42.603476 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:58:42.603684 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:42.603578 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:58:42.680202 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:42.680174 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/993cb549-cec9-4159-a0b2-a023e2cd26c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b90ec4e-w76hj\" (UID: \"993cb549-cec9-4159-a0b2-a023e2cd26c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b90ec4e-w76hj" Oct 30 16:58:42.680352 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:42.680313 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr2tj\" (UniqueName: \"kubernetes.io/projected/993cb549-cec9-4159-a0b2-a023e2cd26c8-kube-api-access-vr2tj\") pod \"maas-api-revoke-sub-3b90ec4e-w76hj\" (UID: \"993cb549-cec9-4159-a0b2-a023e2cd26c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b90ec4e-w76hj" Oct 30 16:58:42.680471 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:42.680456 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/993cb549-cec9-4159-a0b2-a023e2cd26c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b90ec4e-w76hj\" (UID: \"993cb549-cec9-4159-a0b2-a023e2cd26c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b90ec4e-w76hj" Oct 30 16:58:42.781675 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:42.781639 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr2tj\" (UniqueName: \"kubernetes.io/projected/993cb549-cec9-4159-a0b2-a023e2cd26c8-kube-api-access-vr2tj\") pod \"maas-api-revoke-sub-3b90ec4e-w76hj\" (UID: \"993cb549-cec9-4159-a0b2-a023e2cd26c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b90ec4e-w76hj" Oct 30 16:58:42.781876 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:42.781718 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/993cb549-cec9-4159-a0b2-a023e2cd26c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b90ec4e-w76hj\" (UID: \"993cb549-cec9-4159-a0b2-a023e2cd26c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b90ec4e-w76hj" Oct 30 16:58:42.781876 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:42.781814 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/993cb549-cec9-4159-a0b2-a023e2cd26c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b90ec4e-w76hj\" (UID: \"993cb549-cec9-4159-a0b2-a023e2cd26c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b90ec4e-w76hj" Oct 30 16:58:42.782459 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:42.782435 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/993cb549-cec9-4159-a0b2-a023e2cd26c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b90ec4e-w76hj\" (UID: \"993cb549-cec9-4159-a0b2-a023e2cd26c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b90ec4e-w76hj" Oct 30 16:58:42.784272 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:42.784249 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/993cb549-cec9-4159-a0b2-a023e2cd26c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b90ec4e-w76hj\" (UID: \"993cb549-cec9-4159-a0b2-a023e2cd26c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b90ec4e-w76hj" Oct 30 16:58:42.805028 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:42.804996 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr2tj\" (UniqueName: \"kubernetes.io/projected/993cb549-cec9-4159-a0b2-a023e2cd26c8-kube-api-access-vr2tj\") pod \"maas-api-revoke-sub-3b90ec4e-w76hj\" (UID: \"993cb549-cec9-4159-a0b2-a023e2cd26c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b90ec4e-w76hj" Oct 30 16:58:42.887847 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:42.887757 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b90ec4e-w76hj" Oct 30 16:58:43.210825 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:43.210779 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b90ec4e-w76hj"] Oct 30 16:58:43.213043 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:58:43.213011 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod993cb549_cec9_4159_a0b2_a023e2cd26c8.slice/crio-1d2ec7e1c83fa2d8209e2ebf61ed69656ade4d1315313374c9d08f14e86825c0 WatchSource:0}: Error finding container 1d2ec7e1c83fa2d8209e2ebf61ed69656ade4d1315313374c9d08f14e86825c0: Status 404 returned error can't find the container with id 1d2ec7e1c83fa2d8209e2ebf61ed69656ade4d1315313374c9d08f14e86825c0 Oct 30 16:58:43.321359 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:43.321323 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b90ec4e-w76hj" event={"ID":"993cb549-cec9-4159-a0b2-a023e2cd26c8","Type":"ContainerStarted","Data":"1d2ec7e1c83fa2d8209e2ebf61ed69656ade4d1315313374c9d08f14e86825c0"} Oct 30 16:58:44.332381 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:44.332346 4493 generic.go:356] "Generic (PLEG): container finished" podID="993cb549-cec9-4159-a0b2-a023e2cd26c8" containerID="e76985d62da397e085713d03d351cb09a96ee428ba73ef5e00b42f6b9d58a1fe" exitCode=0 Oct 30 16:58:44.332896 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:44.332436 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b90ec4e-w76hj" event={"ID":"993cb549-cec9-4159-a0b2-a023e2cd26c8","Type":"ContainerDied","Data":"e76985d62da397e085713d03d351cb09a96ee428ba73ef5e00b42f6b9d58a1fe"} Oct 30 16:58:44.401115 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:44.399856 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-cb6251/auth-test-model-cb6251-kserve-75ff8464df-8qjhr"] Oct 30 16:58:45.549892 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:45.549868 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b90ec4e-w76hj" Oct 30 16:58:45.639077 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:45.638996 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/993cb549-cec9-4159-a0b2-a023e2cd26c8-kube-api-access-vr2tj\" (UniqueName: \"kubernetes.io/projected/993cb549-cec9-4159-a0b2-a023e2cd26c8-kube-api-access-vr2tj\") pod \"993cb549-cec9-4159-a0b2-a023e2cd26c8\" (UID: \"993cb549-cec9-4159-a0b2-a023e2cd26c8\") " Oct 30 16:58:45.639249 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:45.639122 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/993cb549-cec9-4159-a0b2-a023e2cd26c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/993cb549-cec9-4159-a0b2-a023e2cd26c8-maas-api-xks-ca\") pod \"993cb549-cec9-4159-a0b2-a023e2cd26c8\" (UID: \"993cb549-cec9-4159-a0b2-a023e2cd26c8\") " Oct 30 16:58:45.639249 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:45.639223 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/993cb549-cec9-4159-a0b2-a023e2cd26c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/993cb549-cec9-4159-a0b2-a023e2cd26c8-maas-api-openshift-ca\") pod \"993cb549-cec9-4159-a0b2-a023e2cd26c8\" (UID: \"993cb549-cec9-4159-a0b2-a023e2cd26c8\") " Oct 30 16:58:45.639867 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:45.639830 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/993cb549-cec9-4159-a0b2-a023e2cd26c8-maas-api-openshift-ca" pod "993cb549-cec9-4159-a0b2-a023e2cd26c8" (UID: "993cb549-cec9-4159-a0b2-a023e2cd26c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:58:45.641629 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:45.641603 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/993cb549-cec9-4159-a0b2-a023e2cd26c8-maas-api-xks-ca" pod "993cb549-cec9-4159-a0b2-a023e2cd26c8" (UID: "993cb549-cec9-4159-a0b2-a023e2cd26c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:58:45.650649 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:45.650626 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/993cb549-cec9-4159-a0b2-a023e2cd26c8-kube-api-access-vr2tj" pod "993cb549-cec9-4159-a0b2-a023e2cd26c8" (UID: "993cb549-cec9-4159-a0b2-a023e2cd26c8"). InnerVolumeSpecName "kube-api-access-vr2tj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:58:45.741582 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:45.741542 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/993cb549-cec9-4159-a0b2-a023e2cd26c8-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:58:45.741582 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:45.741575 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/993cb549-cec9-4159-a0b2-a023e2cd26c8-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:58:45.741797 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:45.741591 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vr2tj\" (UniqueName: \"kubernetes.io/projected/993cb549-cec9-4159-a0b2-a023e2cd26c8-kube-api-access-vr2tj\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:58:46.350049 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:46.350019 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b90ec4e-w76hj" Oct 30 16:58:46.350256 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:46.350012 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b90ec4e-w76hj" event={"ID":"993cb549-cec9-4159-a0b2-a023e2cd26c8","Type":"ContainerDied","Data":"1d2ec7e1c83fa2d8209e2ebf61ed69656ade4d1315313374c9d08f14e86825c0"} Oct 30 16:58:46.350256 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:46.350169 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d2ec7e1c83fa2d8209e2ebf61ed69656ade4d1315313374c9d08f14e86825c0" Oct 30 16:58:53.010224 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:53.010185 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bddff506-spjzp"] Oct 30 16:58:53.015538 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:53.012308 4493 cpu_manager.go:414] "containerMap: removing container" podUID="993cb549-cec9-4159-a0b2-a023e2cd26c8" containerName="revoke-keys" Oct 30 16:58:53.015538 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:53.012338 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="993cb549-cec9-4159-a0b2-a023e2cd26c8" containerName="revoke-keys" Oct 30 16:58:53.015538 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:53.013062 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="993cb549-cec9-4159-a0b2-a023e2cd26c8" containerName="revoke-keys" Oct 30 16:58:53.024916 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:53.024892 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bddff506-spjzp" Oct 30 16:58:53.039485 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:53.039453 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bddff506-spjzp"] Oct 30 16:58:53.041947 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:53.041919 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fbae41c-770a-4146-b445-417b90cde0dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bddff506-spjzp\" (UID: \"0fbae41c-770a-4146-b445-417b90cde0dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bddff506-spjzp" Oct 30 16:58:53.042062 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:53.041983 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4d6w\" (UniqueName: \"kubernetes.io/projected/0fbae41c-770a-4146-b445-417b90cde0dd-kube-api-access-s4d6w\") pod \"maas-api-revoke-sub-bddff506-spjzp\" (UID: \"0fbae41c-770a-4146-b445-417b90cde0dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bddff506-spjzp" Oct 30 16:58:53.042127 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:53.042063 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fbae41c-770a-4146-b445-417b90cde0dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bddff506-spjzp\" (UID: \"0fbae41c-770a-4146-b445-417b90cde0dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bddff506-spjzp" Oct 30 16:58:53.044488 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:53.044464 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:58:53.044632 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:53.044555 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 16:58:53.044700 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:53.044640 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:58:53.051941 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:53.051923 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:58:53.142981 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:53.142954 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fbae41c-770a-4146-b445-417b90cde0dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bddff506-spjzp\" (UID: \"0fbae41c-770a-4146-b445-417b90cde0dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bddff506-spjzp" Oct 30 16:58:53.143183 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:53.143001 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4d6w\" (UniqueName: \"kubernetes.io/projected/0fbae41c-770a-4146-b445-417b90cde0dd-kube-api-access-s4d6w\") pod \"maas-api-revoke-sub-bddff506-spjzp\" (UID: \"0fbae41c-770a-4146-b445-417b90cde0dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bddff506-spjzp" Oct 30 16:58:53.143183 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:53.143124 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fbae41c-770a-4146-b445-417b90cde0dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bddff506-spjzp\" (UID: \"0fbae41c-770a-4146-b445-417b90cde0dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bddff506-spjzp" Oct 30 16:58:53.143741 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:53.143716 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fbae41c-770a-4146-b445-417b90cde0dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bddff506-spjzp\" (UID: \"0fbae41c-770a-4146-b445-417b90cde0dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bddff506-spjzp" Oct 30 16:58:53.145518 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:53.145496 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fbae41c-770a-4146-b445-417b90cde0dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bddff506-spjzp\" (UID: \"0fbae41c-770a-4146-b445-417b90cde0dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bddff506-spjzp" Oct 30 16:58:53.166285 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:53.166260 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4d6w\" (UniqueName: \"kubernetes.io/projected/0fbae41c-770a-4146-b445-417b90cde0dd-kube-api-access-s4d6w\") pod \"maas-api-revoke-sub-bddff506-spjzp\" (UID: \"0fbae41c-770a-4146-b445-417b90cde0dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bddff506-spjzp" Oct 30 16:58:53.346376 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:53.343031 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bddff506-spjzp" Oct 30 16:58:53.722751 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:53.722723 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bddff506-spjzp"] Oct 30 16:58:53.724433 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:58:53.724405 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fbae41c_770a_4146_b445_417b90cde0dd.slice/crio-771170ff870cc90fa45b8bc4b6b08f1c7a83b0eca15af6ffe2b49819561f7953 WatchSource:0}: Error finding container 771170ff870cc90fa45b8bc4b6b08f1c7a83b0eca15af6ffe2b49819561f7953: Status 404 returned error can't find the container with id 771170ff870cc90fa45b8bc4b6b08f1c7a83b0eca15af6ffe2b49819561f7953 Oct 30 16:58:54.487831 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:54.481923 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bddff506-spjzp" event={"ID":"0fbae41c-770a-4146-b445-417b90cde0dd","Type":"ContainerStarted","Data":"6ec5b3263054683739d362a78303564ef55d57aaa8cd44fc650107acb9c5462e"} Oct 30 16:58:54.487831 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:54.481988 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bddff506-spjzp" event={"ID":"0fbae41c-770a-4146-b445-417b90cde0dd","Type":"ContainerStarted","Data":"771170ff870cc90fa45b8bc4b6b08f1c7a83b0eca15af6ffe2b49819561f7953"} Oct 30 16:58:54.533816 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:54.532094 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bddff506-spjzp" podStartSLOduration=2.532073059 podStartE2EDuration="2.532073059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:58:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:58:54.530256949 +0000 UTC m=+2161.757432049" watchObservedRunningTime="2026-10-30 16:58:54.532073059 +0000 UTC m=+2161.759248160" Oct 30 16:58:55.491385 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:55.491353 4493 generic.go:356] "Generic (PLEG): container finished" podID="0fbae41c-770a-4146-b445-417b90cde0dd" containerID="6ec5b3263054683739d362a78303564ef55d57aaa8cd44fc650107acb9c5462e" exitCode=0 Oct 30 16:58:55.491858 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:55.491440 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bddff506-spjzp" event={"ID":"0fbae41c-770a-4146-b445-417b90cde0dd","Type":"ContainerDied","Data":"6ec5b3263054683739d362a78303564ef55d57aaa8cd44fc650107acb9c5462e"} Oct 30 16:58:55.641555 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:55.641505 4493 scope.go:122] "RemoveContainer" containerID="30ac8d1067882e11748c33eed3e082385242dda3da430b44c34f171e93b871e5" Oct 30 16:58:55.659313 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:55.659292 4493 scope.go:122] "RemoveContainer" containerID="1bbc78bc0fbd07544e6f4249197814e29b45d18d629b2ce3e99c3c8b7bbfff9d" Oct 30 16:58:55.670696 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:55.670675 4493 scope.go:122] "RemoveContainer" containerID="ae277907eb181d5e66fc9fc8d4da642b180d1c3de3647b4dde66f4606b5929aa" Oct 30 16:58:55.680853 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:55.680831 4493 scope.go:122] "RemoveContainer" containerID="f144b4b78e6d9b509eb2c59152e5dff55c008cf44c4a4137fd1ca3a1d0f2e1e2" Oct 30 16:58:55.690175 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:55.690154 4493 scope.go:122] "RemoveContainer" containerID="377f66ea46fca1cbf97d20ea487e29a6565e3ec4cf8129a6c04dc66fb9c90fe1" Oct 30 16:58:55.699826 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:55.699805 4493 scope.go:122] "RemoveContainer" containerID="8ea8e13c6746cdcfca00954449d7cd0e3a6fb9522cdea3d461c0feb2dbff7215" Oct 30 16:58:55.708469 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:55.708452 4493 scope.go:122] "RemoveContainer" containerID="2199581244ee161b1e01b4c37ff3cc5670d63a92b07af4c2109783355d3a386e" Oct 30 16:58:55.717340 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:55.717323 4493 scope.go:122] "RemoveContainer" containerID="83a0f46347960874008431b4961415c78ceb5d3ad125210f1ffddb1bd60fe499" Oct 30 16:58:55.725742 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:55.725721 4493 scope.go:122] "RemoveContainer" containerID="6948ebcb153ac6b2fdda4dc668d4e58105654d92d5208842b48ce0d28db5e3a3" Oct 30 16:58:55.734237 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:55.734209 4493 scope.go:122] "RemoveContainer" containerID="d29fd64292595d492d11f0c631add78c2ede13e2c346fcea1efbbaba254324ba" Oct 30 16:58:56.715324 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:56.715296 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bddff506-spjzp" Oct 30 16:58:56.830579 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:56.830486 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fbae41c-770a-4146-b445-417b90cde0dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fbae41c-770a-4146-b445-417b90cde0dd-maas-api-openshift-ca\") pod \"0fbae41c-770a-4146-b445-417b90cde0dd\" (UID: \"0fbae41c-770a-4146-b445-417b90cde0dd\") " Oct 30 16:58:56.830579 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:56.830552 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fbae41c-770a-4146-b445-417b90cde0dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fbae41c-770a-4146-b445-417b90cde0dd-maas-api-xks-ca\") pod \"0fbae41c-770a-4146-b445-417b90cde0dd\" (UID: \"0fbae41c-770a-4146-b445-417b90cde0dd\") " Oct 30 16:58:56.830821 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:56.830605 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fbae41c-770a-4146-b445-417b90cde0dd-kube-api-access-s4d6w\" (UniqueName: \"kubernetes.io/projected/0fbae41c-770a-4146-b445-417b90cde0dd-kube-api-access-s4d6w\") pod \"0fbae41c-770a-4146-b445-417b90cde0dd\" (UID: \"0fbae41c-770a-4146-b445-417b90cde0dd\") " Oct 30 16:58:56.831205 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:56.831165 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fbae41c-770a-4146-b445-417b90cde0dd-maas-api-openshift-ca" pod "0fbae41c-770a-4146-b445-417b90cde0dd" (UID: "0fbae41c-770a-4146-b445-417b90cde0dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:58:56.831492 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:56.831475 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fbae41c-770a-4146-b445-417b90cde0dd-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:58:56.840218 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:56.840191 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fbae41c-770a-4146-b445-417b90cde0dd-maas-api-xks-ca" pod "0fbae41c-770a-4146-b445-417b90cde0dd" (UID: "0fbae41c-770a-4146-b445-417b90cde0dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:58:56.840348 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:56.840215 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fbae41c-770a-4146-b445-417b90cde0dd-kube-api-access-s4d6w" pod "0fbae41c-770a-4146-b445-417b90cde0dd" (UID: "0fbae41c-770a-4146-b445-417b90cde0dd"). InnerVolumeSpecName "kube-api-access-s4d6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:58:56.932447 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:56.932414 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fbae41c-770a-4146-b445-417b90cde0dd-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:58:56.932447 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:56.932448 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4d6w\" (UniqueName: \"kubernetes.io/projected/0fbae41c-770a-4146-b445-417b90cde0dd-kube-api-access-s4d6w\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:58:57.509041 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:57.509013 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bddff506-spjzp" Oct 30 16:58:57.509234 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:57.509046 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bddff506-spjzp" event={"ID":"0fbae41c-770a-4146-b445-417b90cde0dd","Type":"ContainerDied","Data":"771170ff870cc90fa45b8bc4b6b08f1c7a83b0eca15af6ffe2b49819561f7953"} Oct 30 16:58:57.509234 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:57.509105 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="771170ff870cc90fa45b8bc4b6b08f1c7a83b0eca15af6ffe2b49819561f7953" Oct 30 16:58:59.430902 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:59.430862 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-cb6251/auth-test-model-cb6251-kserve-75ff8464df-8qjhr" podUID="b506c0ec-f4f5-44aa-a8b7-0a5280586ddd" containerName="main" containerID="cri-o://290d464e21b1acd504667699a8af4bf91f1cea8f47392fcf33f642712825fe8f" gracePeriod=45 Oct 30 16:58:59.699948 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:59.699883 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-cb6251/auth-test-model-cb6251-kserve-75ff8464df-8qjhr" Oct 30 16:58:59.769560 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:59.769522 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b506c0ec-f4f5-44aa-a8b7-0a5280586ddd-kube-api-access-kcmkt\" (UniqueName: \"kubernetes.io/projected/b506c0ec-f4f5-44aa-a8b7-0a5280586ddd-kube-api-access-kcmkt\") pod \"b506c0ec-f4f5-44aa-a8b7-0a5280586ddd\" (UID: \"b506c0ec-f4f5-44aa-a8b7-0a5280586ddd\") " Oct 30 16:58:59.769725 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:59.769649 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b506c0ec-f4f5-44aa-a8b7-0a5280586ddd-tls-certs\" (UniqueName: \"kubernetes.io/secret/b506c0ec-f4f5-44aa-a8b7-0a5280586ddd-tls-certs\") pod \"b506c0ec-f4f5-44aa-a8b7-0a5280586ddd\" (UID: \"b506c0ec-f4f5-44aa-a8b7-0a5280586ddd\") " Oct 30 16:58:59.769779 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:59.769726 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b506c0ec-f4f5-44aa-a8b7-0a5280586ddd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b506c0ec-f4f5-44aa-a8b7-0a5280586ddd-dshm\") pod \"b506c0ec-f4f5-44aa-a8b7-0a5280586ddd\" (UID: \"b506c0ec-f4f5-44aa-a8b7-0a5280586ddd\") " Oct 30 16:58:59.769916 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:59.769889 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b506c0ec-f4f5-44aa-a8b7-0a5280586ddd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b506c0ec-f4f5-44aa-a8b7-0a5280586ddd-model-cache\") pod \"b506c0ec-f4f5-44aa-a8b7-0a5280586ddd\" (UID: \"b506c0ec-f4f5-44aa-a8b7-0a5280586ddd\") " Oct 30 16:58:59.769977 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:59.769941 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b506c0ec-f4f5-44aa-a8b7-0a5280586ddd-home\" (UniqueName: \"kubernetes.io/empty-dir/b506c0ec-f4f5-44aa-a8b7-0a5280586ddd-home\") pod \"b506c0ec-f4f5-44aa-a8b7-0a5280586ddd\" (UID: \"b506c0ec-f4f5-44aa-a8b7-0a5280586ddd\") " Oct 30 16:58:59.770038 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:59.769992 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b506c0ec-f4f5-44aa-a8b7-0a5280586ddd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b506c0ec-f4f5-44aa-a8b7-0a5280586ddd-tmp-dir\") pod \"b506c0ec-f4f5-44aa-a8b7-0a5280586ddd\" (UID: \"b506c0ec-f4f5-44aa-a8b7-0a5280586ddd\") " Oct 30 16:58:59.770349 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:59.770268 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b506c0ec-f4f5-44aa-a8b7-0a5280586ddd-model-cache" pod "b506c0ec-f4f5-44aa-a8b7-0a5280586ddd" (UID: "b506c0ec-f4f5-44aa-a8b7-0a5280586ddd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:58:59.770349 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:59.770316 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b506c0ec-f4f5-44aa-a8b7-0a5280586ddd-home" pod "b506c0ec-f4f5-44aa-a8b7-0a5280586ddd" (UID: "b506c0ec-f4f5-44aa-a8b7-0a5280586ddd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:58:59.770560 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:59.770505 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b506c0ec-f4f5-44aa-a8b7-0a5280586ddd-tmp-dir" pod "b506c0ec-f4f5-44aa-a8b7-0a5280586ddd" (UID: "b506c0ec-f4f5-44aa-a8b7-0a5280586ddd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:58:59.770838 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:59.770812 4493 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b506c0ec-f4f5-44aa-a8b7-0a5280586ddd-model-cache\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:58:59.770940 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:59.770841 4493 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b506c0ec-f4f5-44aa-a8b7-0a5280586ddd-home\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:58:59.770940 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:59.770854 4493 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b506c0ec-f4f5-44aa-a8b7-0a5280586ddd-tmp-dir\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:58:59.772229 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:59.772198 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b506c0ec-f4f5-44aa-a8b7-0a5280586ddd-dshm" pod "b506c0ec-f4f5-44aa-a8b7-0a5280586ddd" (UID: "b506c0ec-f4f5-44aa-a8b7-0a5280586ddd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:58:59.772451 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:59.772427 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b506c0ec-f4f5-44aa-a8b7-0a5280586ddd-kube-api-access-kcmkt" pod "b506c0ec-f4f5-44aa-a8b7-0a5280586ddd" (UID: "b506c0ec-f4f5-44aa-a8b7-0a5280586ddd"). InnerVolumeSpecName "kube-api-access-kcmkt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:58:59.773116 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:59.773097 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b506c0ec-f4f5-44aa-a8b7-0a5280586ddd-tls-certs" pod "b506c0ec-f4f5-44aa-a8b7-0a5280586ddd" (UID: "b506c0ec-f4f5-44aa-a8b7-0a5280586ddd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:58:59.872091 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:59.872056 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcmkt\" (UniqueName: \"kubernetes.io/projected/b506c0ec-f4f5-44aa-a8b7-0a5280586ddd-kube-api-access-kcmkt\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:58:59.872091 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:59.872094 4493 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b506c0ec-f4f5-44aa-a8b7-0a5280586ddd-tls-certs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:58:59.872293 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:58:59.872108 4493 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b506c0ec-f4f5-44aa-a8b7-0a5280586ddd-dshm\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:59:00.341305 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:59:00.341274 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 16:59:00.531483 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:00.531448 4493 generic.go:356] "Generic (PLEG): container finished" podID="b506c0ec-f4f5-44aa-a8b7-0a5280586ddd" containerID="290d464e21b1acd504667699a8af4bf91f1cea8f47392fcf33f642712825fe8f" exitCode=0 Oct 30 16:59:00.532057 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:00.531497 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-cb6251/auth-test-model-cb6251-kserve-75ff8464df-8qjhr" Oct 30 16:59:00.532057 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:00.531535 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-cb6251/auth-test-model-cb6251-kserve-75ff8464df-8qjhr" event={"ID":"b506c0ec-f4f5-44aa-a8b7-0a5280586ddd","Type":"ContainerDied","Data":"290d464e21b1acd504667699a8af4bf91f1cea8f47392fcf33f642712825fe8f"} Oct 30 16:59:00.532057 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:00.531593 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-cb6251/auth-test-model-cb6251-kserve-75ff8464df-8qjhr" event={"ID":"b506c0ec-f4f5-44aa-a8b7-0a5280586ddd","Type":"ContainerDied","Data":"69a3c956d3b61ffa856da72a0a3e71b55f25c7d07e0b9b3e3c9145f3d5d492ca"} Oct 30 16:59:00.532057 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:00.531617 4493 scope.go:122] "RemoveContainer" containerID="290d464e21b1acd504667699a8af4bf91f1cea8f47392fcf33f642712825fe8f" Oct 30 16:59:00.542980 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:00.542966 4493 scope.go:122] "RemoveContainer" containerID="290d464e21b1acd504667699a8af4bf91f1cea8f47392fcf33f642712825fe8f" Oct 30 16:59:00.543242 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:59:00.543225 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"290d464e21b1acd504667699a8af4bf91f1cea8f47392fcf33f642712825fe8f\": container with ID starting with 290d464e21b1acd504667699a8af4bf91f1cea8f47392fcf33f642712825fe8f not found: ID does not exist" containerID="290d464e21b1acd504667699a8af4bf91f1cea8f47392fcf33f642712825fe8f" Oct 30 16:59:00.543285 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:00.543250 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"290d464e21b1acd504667699a8af4bf91f1cea8f47392fcf33f642712825fe8f"} err="failed to get container status \"290d464e21b1acd504667699a8af4bf91f1cea8f47392fcf33f642712825fe8f\": rpc error: code = NotFound desc = could not find container \"290d464e21b1acd504667699a8af4bf91f1cea8f47392fcf33f642712825fe8f\": container with ID starting with 290d464e21b1acd504667699a8af4bf91f1cea8f47392fcf33f642712825fe8f not found: ID does not exist" Oct 30 16:59:00.632515 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:00.632409 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-cb6251/auth-test-model-cb6251-kserve-75ff8464df-8qjhr"] Oct 30 16:59:00.633469 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:00.633423 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-cb6251/auth-test-model-cb6251-kserve-75ff8464df-8qjhr"] Oct 30 16:59:00.934996 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:00.934959 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b506c0ec-f4f5-44aa-a8b7-0a5280586ddd" path="/var/lib/kubelet/pods/b506c0ec-f4f5-44aa-a8b7-0a5280586ddd/volumes" Oct 30 16:59:14.346869 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:14.346828 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60250cfb-qq4l5"] Oct 30 16:59:14.347763 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:14.347737 4493 cpu_manager.go:414] "containerMap: removing container" podUID="b506c0ec-f4f5-44aa-a8b7-0a5280586ddd" containerName="main" Oct 30 16:59:14.347763 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:14.347762 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b506c0ec-f4f5-44aa-a8b7-0a5280586ddd" containerName="main" Oct 30 16:59:14.347973 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:14.347814 4493 cpu_manager.go:414] "containerMap: removing container" podUID="0fbae41c-770a-4146-b445-417b90cde0dd" containerName="revoke-keys" Oct 30 16:59:14.347973 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:14.347822 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fbae41c-770a-4146-b445-417b90cde0dd" containerName="revoke-keys" Oct 30 16:59:14.347973 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:14.347952 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="b506c0ec-f4f5-44aa-a8b7-0a5280586ddd" containerName="main" Oct 30 16:59:14.347973 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:14.347970 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fbae41c-770a-4146-b445-417b90cde0dd" containerName="revoke-keys" Oct 30 16:59:14.353388 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:14.353370 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60250cfb-qq4l5" Oct 30 16:59:14.371450 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:14.371426 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60250cfb-qq4l5"] Oct 30 16:59:14.378465 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:14.378437 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:59:14.378465 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:14.378444 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:59:14.380339 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:14.380317 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 16:59:14.380494 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:14.380345 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:59:14.458071 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:14.458038 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp7gs\" (UniqueName: \"kubernetes.io/projected/ebe98f2d-afe7-4684-a56e-5479ec69a6fa-kube-api-access-pp7gs\") pod \"maas-api-revoke-sub-60250cfb-qq4l5\" (UID: \"ebe98f2d-afe7-4684-a56e-5479ec69a6fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60250cfb-qq4l5" Oct 30 16:59:14.458227 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:14.458086 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebe98f2d-afe7-4684-a56e-5479ec69a6fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60250cfb-qq4l5\" (UID: \"ebe98f2d-afe7-4684-a56e-5479ec69a6fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60250cfb-qq4l5" Oct 30 16:59:14.458227 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:14.458216 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebe98f2d-afe7-4684-a56e-5479ec69a6fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60250cfb-qq4l5\" (UID: \"ebe98f2d-afe7-4684-a56e-5479ec69a6fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60250cfb-qq4l5" Oct 30 16:59:14.559362 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:14.559330 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebe98f2d-afe7-4684-a56e-5479ec69a6fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60250cfb-qq4l5\" (UID: \"ebe98f2d-afe7-4684-a56e-5479ec69a6fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60250cfb-qq4l5" Oct 30 16:59:14.559513 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:14.559450 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp7gs\" (UniqueName: \"kubernetes.io/projected/ebe98f2d-afe7-4684-a56e-5479ec69a6fa-kube-api-access-pp7gs\") pod \"maas-api-revoke-sub-60250cfb-qq4l5\" (UID: \"ebe98f2d-afe7-4684-a56e-5479ec69a6fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60250cfb-qq4l5" Oct 30 16:59:14.559513 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:14.559484 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebe98f2d-afe7-4684-a56e-5479ec69a6fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60250cfb-qq4l5\" (UID: \"ebe98f2d-afe7-4684-a56e-5479ec69a6fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60250cfb-qq4l5" Oct 30 16:59:14.560087 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:14.560069 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebe98f2d-afe7-4684-a56e-5479ec69a6fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60250cfb-qq4l5\" (UID: \"ebe98f2d-afe7-4684-a56e-5479ec69a6fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60250cfb-qq4l5" Oct 30 16:59:14.561748 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:14.561731 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebe98f2d-afe7-4684-a56e-5479ec69a6fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60250cfb-qq4l5\" (UID: \"ebe98f2d-afe7-4684-a56e-5479ec69a6fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60250cfb-qq4l5" Oct 30 16:59:14.584268 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:14.584240 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp7gs\" (UniqueName: \"kubernetes.io/projected/ebe98f2d-afe7-4684-a56e-5479ec69a6fa-kube-api-access-pp7gs\") pod \"maas-api-revoke-sub-60250cfb-qq4l5\" (UID: \"ebe98f2d-afe7-4684-a56e-5479ec69a6fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60250cfb-qq4l5" Oct 30 16:59:14.667488 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:14.667462 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60250cfb-qq4l5" Oct 30 16:59:14.979850 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:14.979825 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60250cfb-qq4l5"] Oct 30 16:59:14.980979 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:59:14.980955 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebe98f2d_afe7_4684_a56e_5479ec69a6fa.slice/crio-c70a0f92b28ae8f357c7bb3d88736b434ab4455c8a9c2f7e4ea117fb45a80008 WatchSource:0}: Error finding container c70a0f92b28ae8f357c7bb3d88736b434ab4455c8a9c2f7e4ea117fb45a80008: Status 404 returned error can't find the container with id c70a0f92b28ae8f357c7bb3d88736b434ab4455c8a9c2f7e4ea117fb45a80008 Oct 30 16:59:15.669155 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:15.669121 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60250cfb-qq4l5" event={"ID":"ebe98f2d-afe7-4684-a56e-5479ec69a6fa","Type":"ContainerStarted","Data":"94d41bea71d99e1e7ab57cd727a4be213eb4d72e6bb769aee6750a5c21837dbe"} Oct 30 16:59:15.669551 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:15.669191 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60250cfb-qq4l5" event={"ID":"ebe98f2d-afe7-4684-a56e-5479ec69a6fa","Type":"ContainerStarted","Data":"c70a0f92b28ae8f357c7bb3d88736b434ab4455c8a9c2f7e4ea117fb45a80008"} Oct 30 16:59:15.712975 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:15.712902 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60250cfb-qq4l5" podStartSLOduration=1.712884538 podStartE2EDuration="1.712884538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:59:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:59:15.707892289 +0000 UTC m=+2182.935067390" watchObservedRunningTime="2026-10-30 16:59:15.712884538 +0000 UTC m=+2182.940059639" Oct 30 16:59:16.410553 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:16.410517 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv"] Oct 30 16:59:16.676971 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:16.676886 4493 generic.go:356] "Generic (PLEG): container finished" podID="ebe98f2d-afe7-4684-a56e-5479ec69a6fa" containerID="94d41bea71d99e1e7ab57cd727a4be213eb4d72e6bb769aee6750a5c21837dbe" exitCode=0 Oct 30 16:59:16.677383 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:16.676972 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60250cfb-qq4l5" event={"ID":"ebe98f2d-afe7-4684-a56e-5479ec69a6fa","Type":"ContainerDied","Data":"94d41bea71d99e1e7ab57cd727a4be213eb4d72e6bb769aee6750a5c21837dbe"} Oct 30 16:59:16.677383 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:16.677171 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv" podUID="8ec886a6-5f5d-4057-a72e-b7886477edaa" containerName="istio-proxy" containerID="cri-o://cc518afd361399d7e1b7a3b7ed48b5c937e735941ca52c08db969755dcb1a7cb" gracePeriod=30 Oct 30 16:59:17.135718 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:17.135626 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-dd715e/auth-test-model-dd715e-kserve-7c54c47c97-c4n77"] Oct 30 16:59:17.885440 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:17.885418 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60250cfb-qq4l5" Oct 30 16:59:18.004769 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:18.004679 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebe98f2d-afe7-4684-a56e-5479ec69a6fa-kube-api-access-pp7gs\" (UniqueName: \"kubernetes.io/projected/ebe98f2d-afe7-4684-a56e-5479ec69a6fa-kube-api-access-pp7gs\") pod \"ebe98f2d-afe7-4684-a56e-5479ec69a6fa\" (UID: \"ebe98f2d-afe7-4684-a56e-5479ec69a6fa\") " Oct 30 16:59:18.004959 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:18.004899 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebe98f2d-afe7-4684-a56e-5479ec69a6fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebe98f2d-afe7-4684-a56e-5479ec69a6fa-maas-api-openshift-ca\") pod \"ebe98f2d-afe7-4684-a56e-5479ec69a6fa\" (UID: \"ebe98f2d-afe7-4684-a56e-5479ec69a6fa\") " Oct 30 16:59:18.005087 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:18.005070 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ebe98f2d-afe7-4684-a56e-5479ec69a6fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebe98f2d-afe7-4684-a56e-5479ec69a6fa-maas-api-xks-ca\") pod \"ebe98f2d-afe7-4684-a56e-5479ec69a6fa\" (UID: \"ebe98f2d-afe7-4684-a56e-5479ec69a6fa\") " Oct 30 16:59:18.005238 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:18.005210 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebe98f2d-afe7-4684-a56e-5479ec69a6fa-maas-api-openshift-ca" pod "ebe98f2d-afe7-4684-a56e-5479ec69a6fa" (UID: "ebe98f2d-afe7-4684-a56e-5479ec69a6fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:59:18.006328 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:18.006304 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebe98f2d-afe7-4684-a56e-5479ec69a6fa-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:59:18.007003 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:18.006982 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebe98f2d-afe7-4684-a56e-5479ec69a6fa-kube-api-access-pp7gs" pod "ebe98f2d-afe7-4684-a56e-5479ec69a6fa" (UID: "ebe98f2d-afe7-4684-a56e-5479ec69a6fa"). InnerVolumeSpecName "kube-api-access-pp7gs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:59:18.007662 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:18.007637 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebe98f2d-afe7-4684-a56e-5479ec69a6fa-maas-api-xks-ca" pod "ebe98f2d-afe7-4684-a56e-5479ec69a6fa" (UID: "ebe98f2d-afe7-4684-a56e-5479ec69a6fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:59:18.108192 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:18.108156 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebe98f2d-afe7-4684-a56e-5479ec69a6fa-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:59:18.108192 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:18.108187 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp7gs\" (UniqueName: \"kubernetes.io/projected/ebe98f2d-afe7-4684-a56e-5479ec69a6fa-kube-api-access-pp7gs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:59:18.693383 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:18.693351 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60250cfb-qq4l5" Oct 30 16:59:18.693557 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:18.693377 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60250cfb-qq4l5" event={"ID":"ebe98f2d-afe7-4684-a56e-5479ec69a6fa","Type":"ContainerDied","Data":"c70a0f92b28ae8f357c7bb3d88736b434ab4455c8a9c2f7e4ea117fb45a80008"} Oct 30 16:59:18.693557 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:18.693432 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c70a0f92b28ae8f357c7bb3d88736b434ab4455c8a9c2f7e4ea117fb45a80008" Oct 30 16:59:21.974434 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:21.974409 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv" Oct 30 16:59:22.069360 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:22.069326 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ec886a6-5f5d-4057-a72e-b7886477edaa-istio-token\" (UniqueName: \"kubernetes.io/projected/8ec886a6-5f5d-4057-a72e-b7886477edaa-istio-token\") pod \"8ec886a6-5f5d-4057-a72e-b7886477edaa\" (UID: \"8ec886a6-5f5d-4057-a72e-b7886477edaa\") " Oct 30 16:59:22.069542 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:22.069374 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ec886a6-5f5d-4057-a72e-b7886477edaa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ec886a6-5f5d-4057-a72e-b7886477edaa-istio-data\") pod \"8ec886a6-5f5d-4057-a72e-b7886477edaa\" (UID: \"8ec886a6-5f5d-4057-a72e-b7886477edaa\") " Oct 30 16:59:22.069542 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:22.069433 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ec886a6-5f5d-4057-a72e-b7886477edaa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec886a6-5f5d-4057-a72e-b7886477edaa-workload-socket\") pod \"8ec886a6-5f5d-4057-a72e-b7886477edaa\" (UID: \"8ec886a6-5f5d-4057-a72e-b7886477edaa\") " Oct 30 16:59:22.069668 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:22.069545 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ec886a6-5f5d-4057-a72e-b7886477edaa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ec886a6-5f5d-4057-a72e-b7886477edaa-istiod-ca-cert\") pod \"8ec886a6-5f5d-4057-a72e-b7886477edaa\" (UID: \"8ec886a6-5f5d-4057-a72e-b7886477edaa\") " Oct 30 16:59:22.069668 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:22.069573 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ec886a6-5f5d-4057-a72e-b7886477edaa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ec886a6-5f5d-4057-a72e-b7886477edaa-workload-certs\") pod \"8ec886a6-5f5d-4057-a72e-b7886477edaa\" (UID: \"8ec886a6-5f5d-4057-a72e-b7886477edaa\") " Oct 30 16:59:22.069668 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:22.069669 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ec886a6-5f5d-4057-a72e-b7886477edaa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ec886a6-5f5d-4057-a72e-b7886477edaa-istio-envoy\") pod \"8ec886a6-5f5d-4057-a72e-b7886477edaa\" (UID: \"8ec886a6-5f5d-4057-a72e-b7886477edaa\") " Oct 30 16:59:22.069997 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:22.069722 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ec886a6-5f5d-4057-a72e-b7886477edaa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec886a6-5f5d-4057-a72e-b7886477edaa-credential-socket\") pod \"8ec886a6-5f5d-4057-a72e-b7886477edaa\" (UID: \"8ec886a6-5f5d-4057-a72e-b7886477edaa\") " Oct 30 16:59:22.069997 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:22.069774 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ec886a6-5f5d-4057-a72e-b7886477edaa-kube-api-access-5fk89\" (UniqueName: \"kubernetes.io/projected/8ec886a6-5f5d-4057-a72e-b7886477edaa-kube-api-access-5fk89\") pod \"8ec886a6-5f5d-4057-a72e-b7886477edaa\" (UID: \"8ec886a6-5f5d-4057-a72e-b7886477edaa\") " Oct 30 16:59:22.069997 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:22.069823 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ec886a6-5f5d-4057-a72e-b7886477edaa-workload-socket" pod "8ec886a6-5f5d-4057-a72e-b7886477edaa" (UID: "8ec886a6-5f5d-4057-a72e-b7886477edaa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:22.069997 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:22.069882 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ec886a6-5f5d-4057-a72e-b7886477edaa-istio-data" pod "8ec886a6-5f5d-4057-a72e-b7886477edaa" (UID: "8ec886a6-5f5d-4057-a72e-b7886477edaa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:22.069997 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:22.069902 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8ec886a6-5f5d-4057-a72e-b7886477edaa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ec886a6-5f5d-4057-a72e-b7886477edaa-istio-podinfo\") pod \"8ec886a6-5f5d-4057-a72e-b7886477edaa\" (UID: \"8ec886a6-5f5d-4057-a72e-b7886477edaa\") " Oct 30 16:59:22.070289 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:22.070058 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ec886a6-5f5d-4057-a72e-b7886477edaa-workload-certs" pod "8ec886a6-5f5d-4057-a72e-b7886477edaa" (UID: "8ec886a6-5f5d-4057-a72e-b7886477edaa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:22.070289 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:22.070101 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ec886a6-5f5d-4057-a72e-b7886477edaa-credential-socket" pod "8ec886a6-5f5d-4057-a72e-b7886477edaa" (UID: "8ec886a6-5f5d-4057-a72e-b7886477edaa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:22.070390 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:22.070358 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ec886a6-5f5d-4057-a72e-b7886477edaa-istiod-ca-cert" pod "8ec886a6-5f5d-4057-a72e-b7886477edaa" (UID: "8ec886a6-5f5d-4057-a72e-b7886477edaa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:59:22.070999 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:22.070593 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ec886a6-5f5d-4057-a72e-b7886477edaa-istio-data\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:59:22.070999 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:22.070622 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec886a6-5f5d-4057-a72e-b7886477edaa-workload-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:59:22.070999 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:22.070638 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ec886a6-5f5d-4057-a72e-b7886477edaa-istiod-ca-cert\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:59:22.070999 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:22.070655 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ec886a6-5f5d-4057-a72e-b7886477edaa-workload-certs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:59:22.070999 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:22.070668 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec886a6-5f5d-4057-a72e-b7886477edaa-credential-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:59:22.072897 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:22.072763 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ec886a6-5f5d-4057-a72e-b7886477edaa-istio-envoy" pod "8ec886a6-5f5d-4057-a72e-b7886477edaa" (UID: "8ec886a6-5f5d-4057-a72e-b7886477edaa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:22.073308 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:22.073284 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ec886a6-5f5d-4057-a72e-b7886477edaa-istio-token" pod "8ec886a6-5f5d-4057-a72e-b7886477edaa" (UID: "8ec886a6-5f5d-4057-a72e-b7886477edaa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:59:22.073500 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:22.073471 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ec886a6-5f5d-4057-a72e-b7886477edaa-kube-api-access-5fk89" pod "8ec886a6-5f5d-4057-a72e-b7886477edaa" (UID: "8ec886a6-5f5d-4057-a72e-b7886477edaa"). InnerVolumeSpecName "kube-api-access-5fk89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:59:22.073704 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:22.073685 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ec886a6-5f5d-4057-a72e-b7886477edaa-istio-podinfo" pod "8ec886a6-5f5d-4057-a72e-b7886477edaa" (UID: "8ec886a6-5f5d-4057-a72e-b7886477edaa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:59:22.172009 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:22.171981 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ec886a6-5f5d-4057-a72e-b7886477edaa-istio-podinfo\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:59:22.172009 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:22.172006 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ec886a6-5f5d-4057-a72e-b7886477edaa-istio-token\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:59:22.172009 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:22.172014 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ec886a6-5f5d-4057-a72e-b7886477edaa-istio-envoy\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:59:22.172238 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:22.172022 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fk89\" (UniqueName: \"kubernetes.io/projected/8ec886a6-5f5d-4057-a72e-b7886477edaa-kube-api-access-5fk89\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:59:22.727887 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:22.727844 4493 generic.go:356] "Generic (PLEG): container finished" podID="8ec886a6-5f5d-4057-a72e-b7886477edaa" containerID="cc518afd361399d7e1b7a3b7ed48b5c937e735941ca52c08db969755dcb1a7cb" exitCode=0 Oct 30 16:59:22.728071 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:22.727906 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv" Oct 30 16:59:22.728071 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:22.727905 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv" event={"ID":"8ec886a6-5f5d-4057-a72e-b7886477edaa","Type":"ContainerDied","Data":"cc518afd361399d7e1b7a3b7ed48b5c937e735941ca52c08db969755dcb1a7cb"} Oct 30 16:59:22.728071 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:22.727980 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv" event={"ID":"8ec886a6-5f5d-4057-a72e-b7886477edaa","Type":"ContainerDied","Data":"a37151411ea57e5e91817de0d62e06675831478f26df6e27be3a03809614c71e"} Oct 30 16:59:22.728071 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:22.728011 4493 scope.go:122] "RemoveContainer" containerID="cc518afd361399d7e1b7a3b7ed48b5c937e735941ca52c08db969755dcb1a7cb" Oct 30 16:59:22.739462 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:22.739444 4493 scope.go:122] "RemoveContainer" containerID="cc518afd361399d7e1b7a3b7ed48b5c937e735941ca52c08db969755dcb1a7cb" Oct 30 16:59:22.739758 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:59:22.739739 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc518afd361399d7e1b7a3b7ed48b5c937e735941ca52c08db969755dcb1a7cb\": container with ID starting with cc518afd361399d7e1b7a3b7ed48b5c937e735941ca52c08db969755dcb1a7cb not found: ID does not exist" containerID="cc518afd361399d7e1b7a3b7ed48b5c937e735941ca52c08db969755dcb1a7cb" Oct 30 16:59:22.739875 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:22.739764 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc518afd361399d7e1b7a3b7ed48b5c937e735941ca52c08db969755dcb1a7cb"} err="failed to get container status \"cc518afd361399d7e1b7a3b7ed48b5c937e735941ca52c08db969755dcb1a7cb\": rpc error: code = NotFound desc = could not find container \"cc518afd361399d7e1b7a3b7ed48b5c937e735941ca52c08db969755dcb1a7cb\": container with ID starting with cc518afd361399d7e1b7a3b7ed48b5c937e735941ca52c08db969755dcb1a7cb not found: ID does not exist" Oct 30 16:59:22.799447 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:22.799421 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv"] Oct 30 16:59:22.811513 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:22.811483 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-tndhv"] Oct 30 16:59:22.935947 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:22.935911 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ec886a6-5f5d-4057-a72e-b7886477edaa" path="/var/lib/kubelet/pods/8ec886a6-5f5d-4057-a72e-b7886477edaa/volumes" Oct 30 16:59:24.706980 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:24.706941 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d85db049-snhzz"] Oct 30 16:59:24.707511 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:24.707498 4493 cpu_manager.go:414] "containerMap: removing container" podUID="ebe98f2d-afe7-4684-a56e-5479ec69a6fa" containerName="revoke-keys" Oct 30 16:59:24.707556 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:24.707512 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebe98f2d-afe7-4684-a56e-5479ec69a6fa" containerName="revoke-keys" Oct 30 16:59:24.707595 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:24.707559 4493 cpu_manager.go:414] "containerMap: removing container" podUID="8ec886a6-5f5d-4057-a72e-b7886477edaa" containerName="istio-proxy" Oct 30 16:59:24.707595 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:24.707564 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ec886a6-5f5d-4057-a72e-b7886477edaa" containerName="istio-proxy" Oct 30 16:59:24.707661 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:24.707639 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ec886a6-5f5d-4057-a72e-b7886477edaa" containerName="istio-proxy" Oct 30 16:59:24.707661 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:24.707648 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebe98f2d-afe7-4684-a56e-5479ec69a6fa" containerName="revoke-keys" Oct 30 16:59:24.712659 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:24.712639 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d85db049-snhzz" Oct 30 16:59:24.733352 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:24.733328 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d85db049-snhzz"] Oct 30 16:59:24.741067 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:24.741038 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:59:24.741067 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:24.741060 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:59:24.741257 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:24.741060 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:59:24.746974 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:24.746954 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 16:59:24.804232 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:24.804195 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d72gx\" (UniqueName: \"kubernetes.io/projected/540a4a45-3790-4207-b753-ce03bafbe504-kube-api-access-d72gx\") pod \"maas-api-revoke-sub-d85db049-snhzz\" (UID: \"540a4a45-3790-4207-b753-ce03bafbe504\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d85db049-snhzz" Oct 30 16:59:24.804385 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:24.804369 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/540a4a45-3790-4207-b753-ce03bafbe504-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d85db049-snhzz\" (UID: \"540a4a45-3790-4207-b753-ce03bafbe504\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d85db049-snhzz" Oct 30 16:59:24.804511 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:24.804479 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/540a4a45-3790-4207-b753-ce03bafbe504-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d85db049-snhzz\" (UID: \"540a4a45-3790-4207-b753-ce03bafbe504\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d85db049-snhzz" Oct 30 16:59:24.905180 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:24.905148 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/540a4a45-3790-4207-b753-ce03bafbe504-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d85db049-snhzz\" (UID: \"540a4a45-3790-4207-b753-ce03bafbe504\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d85db049-snhzz" Oct 30 16:59:24.905366 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:24.905216 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/540a4a45-3790-4207-b753-ce03bafbe504-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d85db049-snhzz\" (UID: \"540a4a45-3790-4207-b753-ce03bafbe504\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d85db049-snhzz" Oct 30 16:59:24.905366 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:24.905333 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d72gx\" (UniqueName: \"kubernetes.io/projected/540a4a45-3790-4207-b753-ce03bafbe504-kube-api-access-d72gx\") pod \"maas-api-revoke-sub-d85db049-snhzz\" (UID: \"540a4a45-3790-4207-b753-ce03bafbe504\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d85db049-snhzz" Oct 30 16:59:24.905816 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:24.905777 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/540a4a45-3790-4207-b753-ce03bafbe504-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d85db049-snhzz\" (UID: \"540a4a45-3790-4207-b753-ce03bafbe504\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d85db049-snhzz" Oct 30 16:59:24.907694 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:24.907665 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/540a4a45-3790-4207-b753-ce03bafbe504-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d85db049-snhzz\" (UID: \"540a4a45-3790-4207-b753-ce03bafbe504\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d85db049-snhzz" Oct 30 16:59:24.929145 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:24.929119 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d72gx\" (UniqueName: \"kubernetes.io/projected/540a4a45-3790-4207-b753-ce03bafbe504-kube-api-access-d72gx\") pod \"maas-api-revoke-sub-d85db049-snhzz\" (UID: \"540a4a45-3790-4207-b753-ce03bafbe504\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d85db049-snhzz" Oct 30 16:59:25.033864 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:25.026764 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d85db049-snhzz" Oct 30 16:59:25.379625 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:25.379598 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d85db049-snhzz"] Oct 30 16:59:25.380412 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:59:25.380386 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod540a4a45_3790_4207_b753_ce03bafbe504.slice/crio-b9047f64461cb6c15cf6b6cfa6065a5fb0d4c924efc4989916b2ffe71b2650ed WatchSource:0}: Error finding container b9047f64461cb6c15cf6b6cfa6065a5fb0d4c924efc4989916b2ffe71b2650ed: Status 404 returned error can't find the container with id b9047f64461cb6c15cf6b6cfa6065a5fb0d4c924efc4989916b2ffe71b2650ed Oct 30 16:59:25.756074 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:25.756034 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d85db049-snhzz" event={"ID":"540a4a45-3790-4207-b753-ce03bafbe504","Type":"ContainerStarted","Data":"b9047f64461cb6c15cf6b6cfa6065a5fb0d4c924efc4989916b2ffe71b2650ed"} Oct 30 16:59:26.765088 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:26.765003 4493 generic.go:356] "Generic (PLEG): container finished" podID="540a4a45-3790-4207-b753-ce03bafbe504" containerID="d41ea7dcfc2e4df9b104a2de4077ce6800523b4a23fd2430458a813de2b1137a" exitCode=0 Oct 30 16:59:26.765467 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:26.765087 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d85db049-snhzz" event={"ID":"540a4a45-3790-4207-b753-ce03bafbe504","Type":"ContainerDied","Data":"d41ea7dcfc2e4df9b104a2de4077ce6800523b4a23fd2430458a813de2b1137a"} Oct 30 16:59:27.938919 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:27.938894 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d85db049-snhzz" Oct 30 16:59:28.046235 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:28.046158 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/540a4a45-3790-4207-b753-ce03bafbe504-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/540a4a45-3790-4207-b753-ce03bafbe504-maas-api-xks-ca\") pod \"540a4a45-3790-4207-b753-ce03bafbe504\" (UID: \"540a4a45-3790-4207-b753-ce03bafbe504\") " Oct 30 16:59:28.046382 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:28.046320 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/540a4a45-3790-4207-b753-ce03bafbe504-kube-api-access-d72gx\" (UniqueName: \"kubernetes.io/projected/540a4a45-3790-4207-b753-ce03bafbe504-kube-api-access-d72gx\") pod \"540a4a45-3790-4207-b753-ce03bafbe504\" (UID: \"540a4a45-3790-4207-b753-ce03bafbe504\") " Oct 30 16:59:28.046461 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:28.046447 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/540a4a45-3790-4207-b753-ce03bafbe504-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/540a4a45-3790-4207-b753-ce03bafbe504-maas-api-openshift-ca\") pod \"540a4a45-3790-4207-b753-ce03bafbe504\" (UID: \"540a4a45-3790-4207-b753-ce03bafbe504\") " Oct 30 16:59:28.046900 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:28.046830 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/540a4a45-3790-4207-b753-ce03bafbe504-maas-api-openshift-ca" pod "540a4a45-3790-4207-b753-ce03bafbe504" (UID: "540a4a45-3790-4207-b753-ce03bafbe504"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:59:28.047243 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:28.047222 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/540a4a45-3790-4207-b753-ce03bafbe504-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:59:28.048471 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:28.048437 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/540a4a45-3790-4207-b753-ce03bafbe504-maas-api-xks-ca" pod "540a4a45-3790-4207-b753-ce03bafbe504" (UID: "540a4a45-3790-4207-b753-ce03bafbe504"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:59:28.048614 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:28.048594 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/540a4a45-3790-4207-b753-ce03bafbe504-kube-api-access-d72gx" pod "540a4a45-3790-4207-b753-ce03bafbe504" (UID: "540a4a45-3790-4207-b753-ce03bafbe504"). InnerVolumeSpecName "kube-api-access-d72gx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:59:28.148050 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:28.148020 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d72gx\" (UniqueName: \"kubernetes.io/projected/540a4a45-3790-4207-b753-ce03bafbe504-kube-api-access-d72gx\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:59:28.148050 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:28.148046 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/540a4a45-3790-4207-b753-ce03bafbe504-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:59:28.780530 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:28.780499 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d85db049-snhzz" Oct 30 16:59:28.780705 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:28.780537 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d85db049-snhzz" event={"ID":"540a4a45-3790-4207-b753-ce03bafbe504","Type":"ContainerDied","Data":"b9047f64461cb6c15cf6b6cfa6065a5fb0d4c924efc4989916b2ffe71b2650ed"} Oct 30 16:59:28.780705 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:28.780581 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9047f64461cb6c15cf6b6cfa6065a5fb0d4c924efc4989916b2ffe71b2650ed" Oct 30 16:59:30.341295 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:59:30.341258 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 16:59:31.417230 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:31.417179 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6w2sv"] Oct 30 16:59:31.417759 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:31.417393 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6w2sv" podUID="59ae5769-a6cf-444d-947e-313ff39785a5" containerName="istio-proxy" containerID="cri-o://03d8bac861db98541fa5cdd6bcd270caad21afee80b1a02209b5fc450831a391" gracePeriod=30 Oct 30 16:59:32.012365 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:32.012326 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd"] Oct 30 16:59:32.012581 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:32.012532 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd" podUID="948c9256-224e-472e-8ccf-5f6d761b5e0c" containerName="istio-proxy" containerID="cri-o://20026c763422eeba9d8b3b052bf64a433460a11aa284fce30f7d4d83533f6ff3" gracePeriod=30 Oct 30 16:59:32.025130 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:32.025095 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp"] Oct 30 16:59:32.025364 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:32.025331 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp" podUID="8c88b7e3-1f93-456a-87ed-4894a1de4b6c" containerName="istio-proxy" containerID="cri-o://d85e60dc92c35d3c7d597c161adb0d1f2ebfd6787730288e39a4bc07579f4dd4" gracePeriod=30 Oct 30 16:59:32.181432 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:32.181373 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-dd715e/auth-test-model-dd715e-kserve-7c54c47c97-c4n77" podUID="67c90611-0953-4152-8696-695093024e17" containerName="main" containerID="cri-o://0d2995d84e2b4bb0f2235dfd3d63669d3037d390ec87d502ec20af9da9c2190c" gracePeriod=45 Oct 30 16:59:32.214172 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:32.214136 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-dd715e/auth-test-model-dd715e-kserve-7c54c47c97-c4n77" podUID="67c90611-0953-4152-8696-695093024e17" containerName="main" probeResult="failure" output="Get \"https://10.134.0.192:8000/ready\": dial tcp 10.134.0.192:8000: connect: connection refused" Oct 30 16:59:32.462233 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:32.462209 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-dd715e/auth-test-model-dd715e-kserve-7c54c47c97-c4n77" Oct 30 16:59:32.502089 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:32.502058 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67c90611-0953-4152-8696-695093024e17-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67c90611-0953-4152-8696-695093024e17-model-cache\") pod \"67c90611-0953-4152-8696-695093024e17\" (UID: \"67c90611-0953-4152-8696-695093024e17\") " Oct 30 16:59:32.502246 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:32.502127 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67c90611-0953-4152-8696-695093024e17-kube-api-access-djzbp\" (UniqueName: \"kubernetes.io/projected/67c90611-0953-4152-8696-695093024e17-kube-api-access-djzbp\") pod \"67c90611-0953-4152-8696-695093024e17\" (UID: \"67c90611-0953-4152-8696-695093024e17\") " Oct 30 16:59:32.502246 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:32.502177 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67c90611-0953-4152-8696-695093024e17-tls-certs\" (UniqueName: \"kubernetes.io/secret/67c90611-0953-4152-8696-695093024e17-tls-certs\") pod \"67c90611-0953-4152-8696-695093024e17\" (UID: \"67c90611-0953-4152-8696-695093024e17\") " Oct 30 16:59:32.502246 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:32.502219 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67c90611-0953-4152-8696-695093024e17-dshm\" (UniqueName: \"kubernetes.io/empty-dir/67c90611-0953-4152-8696-695093024e17-dshm\") pod \"67c90611-0953-4152-8696-695093024e17\" (UID: \"67c90611-0953-4152-8696-695093024e17\") " Oct 30 16:59:32.502436 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:32.502298 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67c90611-0953-4152-8696-695093024e17-model-cache" pod "67c90611-0953-4152-8696-695093024e17" (UID: "67c90611-0953-4152-8696-695093024e17"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:32.502436 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:32.502363 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67c90611-0953-4152-8696-695093024e17-home\" (UniqueName: \"kubernetes.io/empty-dir/67c90611-0953-4152-8696-695093024e17-home\") pod \"67c90611-0953-4152-8696-695093024e17\" (UID: \"67c90611-0953-4152-8696-695093024e17\") " Oct 30 16:59:32.502550 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:32.502477 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67c90611-0953-4152-8696-695093024e17-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67c90611-0953-4152-8696-695093024e17-tmp-dir\") pod \"67c90611-0953-4152-8696-695093024e17\" (UID: \"67c90611-0953-4152-8696-695093024e17\") " Oct 30 16:59:32.502626 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:32.502598 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67c90611-0953-4152-8696-695093024e17-home" pod "67c90611-0953-4152-8696-695093024e17" (UID: "67c90611-0953-4152-8696-695093024e17"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:32.502749 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:32.502690 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67c90611-0953-4152-8696-695093024e17-tmp-dir" pod "67c90611-0953-4152-8696-695093024e17" (UID: "67c90611-0953-4152-8696-695093024e17"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:32.503016 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:32.502996 4493 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67c90611-0953-4152-8696-695093024e17-home\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:59:32.503123 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:32.503041 4493 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67c90611-0953-4152-8696-695093024e17-tmp-dir\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:59:32.503123 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:32.503053 4493 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67c90611-0953-4152-8696-695093024e17-model-cache\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:59:32.505453 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:32.505426 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67c90611-0953-4152-8696-695093024e17-kube-api-access-djzbp" pod "67c90611-0953-4152-8696-695093024e17" (UID: "67c90611-0953-4152-8696-695093024e17"). InnerVolumeSpecName "kube-api-access-djzbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:59:32.505562 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:32.505509 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67c90611-0953-4152-8696-695093024e17-dshm" pod "67c90611-0953-4152-8696-695093024e17" (UID: "67c90611-0953-4152-8696-695093024e17"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:32.506168 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:32.506141 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67c90611-0953-4152-8696-695093024e17-tls-certs" pod "67c90611-0953-4152-8696-695093024e17" (UID: "67c90611-0953-4152-8696-695093024e17"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:59:32.604839 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:32.604714 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djzbp\" (UniqueName: \"kubernetes.io/projected/67c90611-0953-4152-8696-695093024e17-kube-api-access-djzbp\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:59:32.604839 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:32.604754 4493 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67c90611-0953-4152-8696-695093024e17-tls-certs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:59:32.604839 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:32.604768 4493 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67c90611-0953-4152-8696-695093024e17-dshm\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:59:32.827364 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:32.827328 4493 generic.go:356] "Generic (PLEG): container finished" podID="67c90611-0953-4152-8696-695093024e17" containerID="0d2995d84e2b4bb0f2235dfd3d63669d3037d390ec87d502ec20af9da9c2190c" exitCode=0 Oct 30 16:59:32.827575 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:32.827389 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-dd715e/auth-test-model-dd715e-kserve-7c54c47c97-c4n77" Oct 30 16:59:32.827575 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:32.827415 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-dd715e/auth-test-model-dd715e-kserve-7c54c47c97-c4n77" event={"ID":"67c90611-0953-4152-8696-695093024e17","Type":"ContainerDied","Data":"0d2995d84e2b4bb0f2235dfd3d63669d3037d390ec87d502ec20af9da9c2190c"} Oct 30 16:59:32.827575 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:32.827464 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-dd715e/auth-test-model-dd715e-kserve-7c54c47c97-c4n77" event={"ID":"67c90611-0953-4152-8696-695093024e17","Type":"ContainerDied","Data":"b831b171af94b33e2002b098954f41d46029d2c238d4d53f99e0cf6269e5eedc"} Oct 30 16:59:32.827575 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:32.827488 4493 scope.go:122] "RemoveContainer" containerID="0d2995d84e2b4bb0f2235dfd3d63669d3037d390ec87d502ec20af9da9c2190c" Oct 30 16:59:32.839603 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:32.839585 4493 scope.go:122] "RemoveContainer" containerID="0d2995d84e2b4bb0f2235dfd3d63669d3037d390ec87d502ec20af9da9c2190c" Oct 30 16:59:32.839937 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:59:32.839917 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d2995d84e2b4bb0f2235dfd3d63669d3037d390ec87d502ec20af9da9c2190c\": container with ID starting with 0d2995d84e2b4bb0f2235dfd3d63669d3037d390ec87d502ec20af9da9c2190c not found: ID does not exist" containerID="0d2995d84e2b4bb0f2235dfd3d63669d3037d390ec87d502ec20af9da9c2190c" Oct 30 16:59:32.839997 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:32.839947 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d2995d84e2b4bb0f2235dfd3d63669d3037d390ec87d502ec20af9da9c2190c"} err="failed to get container status \"0d2995d84e2b4bb0f2235dfd3d63669d3037d390ec87d502ec20af9da9c2190c\": rpc error: code = NotFound desc = could not find container \"0d2995d84e2b4bb0f2235dfd3d63669d3037d390ec87d502ec20af9da9c2190c\": container with ID starting with 0d2995d84e2b4bb0f2235dfd3d63669d3037d390ec87d502ec20af9da9c2190c not found: ID does not exist" Oct 30 16:59:32.880243 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:32.880159 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-dd715e/auth-test-model-dd715e-kserve-7c54c47c97-c4n77"] Oct 30 16:59:32.883550 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:32.883519 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-dd715e/auth-test-model-dd715e-kserve-7c54c47c97-c4n77"] Oct 30 16:59:32.934557 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:32.934525 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67c90611-0953-4152-8696-695093024e17" path="/var/lib/kubelet/pods/67c90611-0953-4152-8696-695093024e17/volumes" Oct 30 16:59:36.694694 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:36.694668 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6w2sv" Oct 30 16:59:36.760028 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:36.759949 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59ae5769-a6cf-444d-947e-313ff39785a5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59ae5769-a6cf-444d-947e-313ff39785a5-workload-certs\") pod \"59ae5769-a6cf-444d-947e-313ff39785a5\" (UID: \"59ae5769-a6cf-444d-947e-313ff39785a5\") " Oct 30 16:59:36.760028 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:36.760006 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59ae5769-a6cf-444d-947e-313ff39785a5-istio-token\" (UniqueName: \"kubernetes.io/projected/59ae5769-a6cf-444d-947e-313ff39785a5-istio-token\") pod \"59ae5769-a6cf-444d-947e-313ff39785a5\" (UID: \"59ae5769-a6cf-444d-947e-313ff39785a5\") " Oct 30 16:59:36.760263 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:36.760043 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59ae5769-a6cf-444d-947e-313ff39785a5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59ae5769-a6cf-444d-947e-313ff39785a5-credential-socket\") pod \"59ae5769-a6cf-444d-947e-313ff39785a5\" (UID: \"59ae5769-a6cf-444d-947e-313ff39785a5\") " Oct 30 16:59:36.760263 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:36.760075 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59ae5769-a6cf-444d-947e-313ff39785a5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59ae5769-a6cf-444d-947e-313ff39785a5-workload-socket\") pod \"59ae5769-a6cf-444d-947e-313ff39785a5\" (UID: \"59ae5769-a6cf-444d-947e-313ff39785a5\") " Oct 30 16:59:36.760263 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:36.760107 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59ae5769-a6cf-444d-947e-313ff39785a5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59ae5769-a6cf-444d-947e-313ff39785a5-istiod-ca-cert\") pod \"59ae5769-a6cf-444d-947e-313ff39785a5\" (UID: \"59ae5769-a6cf-444d-947e-313ff39785a5\") " Oct 30 16:59:36.760263 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:36.760135 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59ae5769-a6cf-444d-947e-313ff39785a5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59ae5769-a6cf-444d-947e-313ff39785a5-istio-data\") pod \"59ae5769-a6cf-444d-947e-313ff39785a5\" (UID: \"59ae5769-a6cf-444d-947e-313ff39785a5\") " Oct 30 16:59:36.760263 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:36.760195 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59ae5769-a6cf-444d-947e-313ff39785a5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59ae5769-a6cf-444d-947e-313ff39785a5-istio-envoy\") pod \"59ae5769-a6cf-444d-947e-313ff39785a5\" (UID: \"59ae5769-a6cf-444d-947e-313ff39785a5\") " Oct 30 16:59:36.760519 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:36.760301 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/59ae5769-a6cf-444d-947e-313ff39785a5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59ae5769-a6cf-444d-947e-313ff39785a5-istio-podinfo\") pod \"59ae5769-a6cf-444d-947e-313ff39785a5\" (UID: \"59ae5769-a6cf-444d-947e-313ff39785a5\") " Oct 30 16:59:36.760519 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:36.760278 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59ae5769-a6cf-444d-947e-313ff39785a5-workload-certs" pod "59ae5769-a6cf-444d-947e-313ff39785a5" (UID: "59ae5769-a6cf-444d-947e-313ff39785a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:36.760519 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:36.760382 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59ae5769-a6cf-444d-947e-313ff39785a5-kube-api-access-zs98z\" (UniqueName: \"kubernetes.io/projected/59ae5769-a6cf-444d-947e-313ff39785a5-kube-api-access-zs98z\") pod \"59ae5769-a6cf-444d-947e-313ff39785a5\" (UID: \"59ae5769-a6cf-444d-947e-313ff39785a5\") " Oct 30 16:59:36.760663 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:36.760562 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59ae5769-a6cf-444d-947e-313ff39785a5-workload-socket" pod "59ae5769-a6cf-444d-947e-313ff39785a5" (UID: "59ae5769-a6cf-444d-947e-313ff39785a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:36.760663 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:36.760614 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59ae5769-a6cf-444d-947e-313ff39785a5-istiod-ca-cert" pod "59ae5769-a6cf-444d-947e-313ff39785a5" (UID: "59ae5769-a6cf-444d-947e-313ff39785a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:59:36.760989 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:36.760964 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59ae5769-a6cf-444d-947e-313ff39785a5-istio-data" pod "59ae5769-a6cf-444d-947e-313ff39785a5" (UID: "59ae5769-a6cf-444d-947e-313ff39785a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:36.761088 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:36.760985 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59ae5769-a6cf-444d-947e-313ff39785a5-credential-socket" pod "59ae5769-a6cf-444d-947e-313ff39785a5" (UID: "59ae5769-a6cf-444d-947e-313ff39785a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:36.761198 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:36.761180 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59ae5769-a6cf-444d-947e-313ff39785a5-workload-certs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:59:36.761258 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:36.761204 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59ae5769-a6cf-444d-947e-313ff39785a5-credential-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:59:36.761258 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:36.761218 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59ae5769-a6cf-444d-947e-313ff39785a5-workload-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:59:36.761258 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:36.761234 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59ae5769-a6cf-444d-947e-313ff39785a5-istiod-ca-cert\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:59:36.761258 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:36.761248 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59ae5769-a6cf-444d-947e-313ff39785a5-istio-data\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:59:36.763272 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:36.763244 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/59ae5769-a6cf-444d-947e-313ff39785a5-istio-podinfo" pod "59ae5769-a6cf-444d-947e-313ff39785a5" (UID: "59ae5769-a6cf-444d-947e-313ff39785a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:59:36.763272 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:36.763259 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59ae5769-a6cf-444d-947e-313ff39785a5-kube-api-access-zs98z" pod "59ae5769-a6cf-444d-947e-313ff39785a5" (UID: "59ae5769-a6cf-444d-947e-313ff39785a5"). InnerVolumeSpecName "kube-api-access-zs98z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:59:36.763591 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:36.763560 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59ae5769-a6cf-444d-947e-313ff39785a5-istio-token" pod "59ae5769-a6cf-444d-947e-313ff39785a5" (UID: "59ae5769-a6cf-444d-947e-313ff39785a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:59:36.764010 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:36.763993 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59ae5769-a6cf-444d-947e-313ff39785a5-istio-envoy" pod "59ae5769-a6cf-444d-947e-313ff39785a5" (UID: "59ae5769-a6cf-444d-947e-313ff39785a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:36.863117 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:36.863083 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59ae5769-a6cf-444d-947e-313ff39785a5-istio-token\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:59:36.863117 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:36.863118 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59ae5769-a6cf-444d-947e-313ff39785a5-istio-envoy\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:59:36.863117 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:36.863133 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59ae5769-a6cf-444d-947e-313ff39785a5-istio-podinfo\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:59:36.863446 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:36.863147 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zs98z\" (UniqueName: \"kubernetes.io/projected/59ae5769-a6cf-444d-947e-313ff39785a5-kube-api-access-zs98z\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:59:36.878231 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:36.878196 4493 generic.go:356] "Generic (PLEG): container finished" podID="59ae5769-a6cf-444d-947e-313ff39785a5" containerID="03d8bac861db98541fa5cdd6bcd270caad21afee80b1a02209b5fc450831a391" exitCode=0 Oct 30 16:59:36.878370 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:36.878269 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6w2sv" Oct 30 16:59:36.878370 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:36.878285 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6w2sv" event={"ID":"59ae5769-a6cf-444d-947e-313ff39785a5","Type":"ContainerDied","Data":"03d8bac861db98541fa5cdd6bcd270caad21afee80b1a02209b5fc450831a391"} Oct 30 16:59:36.878441 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:36.878381 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6w2sv" event={"ID":"59ae5769-a6cf-444d-947e-313ff39785a5","Type":"ContainerDied","Data":"0a7e7a1c47b9e844d6ad020ce2a8d7735cff95aa2283cafe73407182be41223d"} Oct 30 16:59:36.878441 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:36.878406 4493 scope.go:122] "RemoveContainer" containerID="03d8bac861db98541fa5cdd6bcd270caad21afee80b1a02209b5fc450831a391" Oct 30 16:59:36.889230 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:36.889212 4493 scope.go:122] "RemoveContainer" containerID="03d8bac861db98541fa5cdd6bcd270caad21afee80b1a02209b5fc450831a391" Oct 30 16:59:36.889498 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:59:36.889479 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03d8bac861db98541fa5cdd6bcd270caad21afee80b1a02209b5fc450831a391\": container with ID starting with 03d8bac861db98541fa5cdd6bcd270caad21afee80b1a02209b5fc450831a391 not found: ID does not exist" containerID="03d8bac861db98541fa5cdd6bcd270caad21afee80b1a02209b5fc450831a391" Oct 30 16:59:36.889561 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:36.889510 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03d8bac861db98541fa5cdd6bcd270caad21afee80b1a02209b5fc450831a391"} err="failed to get container status \"03d8bac861db98541fa5cdd6bcd270caad21afee80b1a02209b5fc450831a391\": rpc error: code = NotFound desc = could not find container \"03d8bac861db98541fa5cdd6bcd270caad21afee80b1a02209b5fc450831a391\": container with ID starting with 03d8bac861db98541fa5cdd6bcd270caad21afee80b1a02209b5fc450831a391 not found: ID does not exist" Oct 30 16:59:36.935923 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:36.935894 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6w2sv"] Oct 30 16:59:36.935923 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:36.935927 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-6w2sv"] Oct 30 16:59:37.305811 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:37.305767 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd" Oct 30 16:59:37.371805 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:37.370396 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/948c9256-224e-472e-8ccf-5f6d761b5e0c-istio-token\" (UniqueName: \"kubernetes.io/projected/948c9256-224e-472e-8ccf-5f6d761b5e0c-istio-token\") pod \"948c9256-224e-472e-8ccf-5f6d761b5e0c\" (UID: \"948c9256-224e-472e-8ccf-5f6d761b5e0c\") " Oct 30 16:59:37.371805 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:37.370455 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/948c9256-224e-472e-8ccf-5f6d761b5e0c-kube-api-access-chff7\" (UniqueName: \"kubernetes.io/projected/948c9256-224e-472e-8ccf-5f6d761b5e0c-kube-api-access-chff7\") pod \"948c9256-224e-472e-8ccf-5f6d761b5e0c\" (UID: \"948c9256-224e-472e-8ccf-5f6d761b5e0c\") " Oct 30 16:59:37.371805 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:37.370551 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/948c9256-224e-472e-8ccf-5f6d761b5e0c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/948c9256-224e-472e-8ccf-5f6d761b5e0c-istio-data\") pod \"948c9256-224e-472e-8ccf-5f6d761b5e0c\" (UID: \"948c9256-224e-472e-8ccf-5f6d761b5e0c\") " Oct 30 16:59:37.371805 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:37.370614 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/948c9256-224e-472e-8ccf-5f6d761b5e0c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/948c9256-224e-472e-8ccf-5f6d761b5e0c-istio-envoy\") pod \"948c9256-224e-472e-8ccf-5f6d761b5e0c\" (UID: \"948c9256-224e-472e-8ccf-5f6d761b5e0c\") " Oct 30 16:59:37.371805 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:37.370674 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/948c9256-224e-472e-8ccf-5f6d761b5e0c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/948c9256-224e-472e-8ccf-5f6d761b5e0c-workload-socket\") pod \"948c9256-224e-472e-8ccf-5f6d761b5e0c\" (UID: \"948c9256-224e-472e-8ccf-5f6d761b5e0c\") " Oct 30 16:59:37.371805 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:37.370729 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/948c9256-224e-472e-8ccf-5f6d761b5e0c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/948c9256-224e-472e-8ccf-5f6d761b5e0c-workload-certs\") pod \"948c9256-224e-472e-8ccf-5f6d761b5e0c\" (UID: \"948c9256-224e-472e-8ccf-5f6d761b5e0c\") " Oct 30 16:59:37.371805 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:37.370765 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/948c9256-224e-472e-8ccf-5f6d761b5e0c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/948c9256-224e-472e-8ccf-5f6d761b5e0c-istiod-ca-cert\") pod \"948c9256-224e-472e-8ccf-5f6d761b5e0c\" (UID: \"948c9256-224e-472e-8ccf-5f6d761b5e0c\") " Oct 30 16:59:37.371805 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:37.370881 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/948c9256-224e-472e-8ccf-5f6d761b5e0c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/948c9256-224e-472e-8ccf-5f6d761b5e0c-istio-podinfo\") pod \"948c9256-224e-472e-8ccf-5f6d761b5e0c\" (UID: \"948c9256-224e-472e-8ccf-5f6d761b5e0c\") " Oct 30 16:59:37.371805 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:37.370972 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/948c9256-224e-472e-8ccf-5f6d761b5e0c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/948c9256-224e-472e-8ccf-5f6d761b5e0c-credential-socket\") pod \"948c9256-224e-472e-8ccf-5f6d761b5e0c\" (UID: \"948c9256-224e-472e-8ccf-5f6d761b5e0c\") " Oct 30 16:59:37.372402 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:37.372181 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/948c9256-224e-472e-8ccf-5f6d761b5e0c-credential-socket" pod "948c9256-224e-472e-8ccf-5f6d761b5e0c" (UID: "948c9256-224e-472e-8ccf-5f6d761b5e0c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:37.374365 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:37.374302 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/948c9256-224e-472e-8ccf-5f6d761b5e0c-workload-socket" pod "948c9256-224e-472e-8ccf-5f6d761b5e0c" (UID: "948c9256-224e-472e-8ccf-5f6d761b5e0c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:37.374500 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:37.374384 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/948c9256-224e-472e-8ccf-5f6d761b5e0c-istio-data" pod "948c9256-224e-472e-8ccf-5f6d761b5e0c" (UID: "948c9256-224e-472e-8ccf-5f6d761b5e0c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:37.375078 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:37.374650 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/948c9256-224e-472e-8ccf-5f6d761b5e0c-istiod-ca-cert" pod "948c9256-224e-472e-8ccf-5f6d761b5e0c" (UID: "948c9256-224e-472e-8ccf-5f6d761b5e0c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:59:37.375078 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:37.375012 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/948c9256-224e-472e-8ccf-5f6d761b5e0c-workload-certs" pod "948c9256-224e-472e-8ccf-5f6d761b5e0c" (UID: "948c9256-224e-472e-8ccf-5f6d761b5e0c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:37.377356 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:37.377317 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/948c9256-224e-472e-8ccf-5f6d761b5e0c-istio-podinfo" pod "948c9256-224e-472e-8ccf-5f6d761b5e0c" (UID: "948c9256-224e-472e-8ccf-5f6d761b5e0c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:59:37.377693 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:37.377671 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/948c9256-224e-472e-8ccf-5f6d761b5e0c-istio-envoy" pod "948c9256-224e-472e-8ccf-5f6d761b5e0c" (UID: "948c9256-224e-472e-8ccf-5f6d761b5e0c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:37.378203 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:37.378184 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp" Oct 30 16:59:37.378616 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:37.378599 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/948c9256-224e-472e-8ccf-5f6d761b5e0c-istio-token" pod "948c9256-224e-472e-8ccf-5f6d761b5e0c" (UID: "948c9256-224e-472e-8ccf-5f6d761b5e0c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:59:37.378856 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:37.378829 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/948c9256-224e-472e-8ccf-5f6d761b5e0c-kube-api-access-chff7" pod "948c9256-224e-472e-8ccf-5f6d761b5e0c" (UID: "948c9256-224e-472e-8ccf-5f6d761b5e0c"). InnerVolumeSpecName "kube-api-access-chff7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:59:37.472468 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:37.472430 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c88b7e3-1f93-456a-87ed-4894a1de4b6c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c88b7e3-1f93-456a-87ed-4894a1de4b6c-istiod-ca-cert\") pod \"8c88b7e3-1f93-456a-87ed-4894a1de4b6c\" (UID: \"8c88b7e3-1f93-456a-87ed-4894a1de4b6c\") " Oct 30 16:59:37.472634 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:37.472483 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c88b7e3-1f93-456a-87ed-4894a1de4b6c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c88b7e3-1f93-456a-87ed-4894a1de4b6c-workload-certs\") pod \"8c88b7e3-1f93-456a-87ed-4894a1de4b6c\" (UID: \"8c88b7e3-1f93-456a-87ed-4894a1de4b6c\") " Oct 30 16:59:37.472634 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:37.472506 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c88b7e3-1f93-456a-87ed-4894a1de4b6c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c88b7e3-1f93-456a-87ed-4894a1de4b6c-credential-socket\") pod \"8c88b7e3-1f93-456a-87ed-4894a1de4b6c\" (UID: \"8c88b7e3-1f93-456a-87ed-4894a1de4b6c\") " Oct 30 16:59:37.472634 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:37.472535 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c88b7e3-1f93-456a-87ed-4894a1de4b6c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c88b7e3-1f93-456a-87ed-4894a1de4b6c-istio-data\") pod \"8c88b7e3-1f93-456a-87ed-4894a1de4b6c\" (UID: \"8c88b7e3-1f93-456a-87ed-4894a1de4b6c\") " Oct 30 16:59:37.472634 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:37.472579 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c88b7e3-1f93-456a-87ed-4894a1de4b6c-istio-token\" (UniqueName: \"kubernetes.io/projected/8c88b7e3-1f93-456a-87ed-4894a1de4b6c-istio-token\") pod \"8c88b7e3-1f93-456a-87ed-4894a1de4b6c\" (UID: \"8c88b7e3-1f93-456a-87ed-4894a1de4b6c\") " Oct 30 16:59:37.472909 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:37.472634 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c88b7e3-1f93-456a-87ed-4894a1de4b6c-kube-api-access-ctj54\" (UniqueName: \"kubernetes.io/projected/8c88b7e3-1f93-456a-87ed-4894a1de4b6c-kube-api-access-ctj54\") pod \"8c88b7e3-1f93-456a-87ed-4894a1de4b6c\" (UID: \"8c88b7e3-1f93-456a-87ed-4894a1de4b6c\") " Oct 30 16:59:37.472909 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:37.472736 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c88b7e3-1f93-456a-87ed-4894a1de4b6c-workload-certs" pod "8c88b7e3-1f93-456a-87ed-4894a1de4b6c" (UID: "8c88b7e3-1f93-456a-87ed-4894a1de4b6c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:37.472909 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:37.472800 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c88b7e3-1f93-456a-87ed-4894a1de4b6c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c88b7e3-1f93-456a-87ed-4894a1de4b6c-istio-envoy\") pod \"8c88b7e3-1f93-456a-87ed-4894a1de4b6c\" (UID: \"8c88b7e3-1f93-456a-87ed-4894a1de4b6c\") " Oct 30 16:59:37.472909 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:37.472833 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c88b7e3-1f93-456a-87ed-4894a1de4b6c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c88b7e3-1f93-456a-87ed-4894a1de4b6c-workload-socket\") pod \"8c88b7e3-1f93-456a-87ed-4894a1de4b6c\" (UID: \"8c88b7e3-1f93-456a-87ed-4894a1de4b6c\") " Oct 30 16:59:37.472909 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:37.472866 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c88b7e3-1f93-456a-87ed-4894a1de4b6c-credential-socket" pod "8c88b7e3-1f93-456a-87ed-4894a1de4b6c" (UID: "8c88b7e3-1f93-456a-87ed-4894a1de4b6c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:37.472909 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:37.472884 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8c88b7e3-1f93-456a-87ed-4894a1de4b6c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c88b7e3-1f93-456a-87ed-4894a1de4b6c-istio-podinfo\") pod \"8c88b7e3-1f93-456a-87ed-4894a1de4b6c\" (UID: \"8c88b7e3-1f93-456a-87ed-4894a1de4b6c\") " Oct 30 16:59:37.473294 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:37.473133 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c88b7e3-1f93-456a-87ed-4894a1de4b6c-workload-socket" pod "8c88b7e3-1f93-456a-87ed-4894a1de4b6c" (UID: "8c88b7e3-1f93-456a-87ed-4894a1de4b6c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:37.473294 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:37.473228 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c88b7e3-1f93-456a-87ed-4894a1de4b6c-istio-data" pod "8c88b7e3-1f93-456a-87ed-4894a1de4b6c" (UID: "8c88b7e3-1f93-456a-87ed-4894a1de4b6c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:37.473433 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:37.473411 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c88b7e3-1f93-456a-87ed-4894a1de4b6c-istiod-ca-cert" pod "8c88b7e3-1f93-456a-87ed-4894a1de4b6c" (UID: "8c88b7e3-1f93-456a-87ed-4894a1de4b6c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:59:37.473974 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:37.473949 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/948c9256-224e-472e-8ccf-5f6d761b5e0c-istio-data\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:59:37.474082 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:37.473978 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/948c9256-224e-472e-8ccf-5f6d761b5e0c-istio-envoy\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:59:37.474082 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:37.473993 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c88b7e3-1f93-456a-87ed-4894a1de4b6c-istiod-ca-cert\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:59:37.474082 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:37.474006 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c88b7e3-1f93-456a-87ed-4894a1de4b6c-workload-certs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:59:37.474082 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:37.474018 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c88b7e3-1f93-456a-87ed-4894a1de4b6c-credential-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:59:37.474082 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:37.474030 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/948c9256-224e-472e-8ccf-5f6d761b5e0c-workload-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:59:37.474082 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:37.474043 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c88b7e3-1f93-456a-87ed-4894a1de4b6c-istio-data\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:59:37.474082 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:37.474055 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/948c9256-224e-472e-8ccf-5f6d761b5e0c-workload-certs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:59:37.474082 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:37.474067 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/948c9256-224e-472e-8ccf-5f6d761b5e0c-istiod-ca-cert\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:59:37.474082 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:37.474079 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/948c9256-224e-472e-8ccf-5f6d761b5e0c-istio-podinfo\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:59:37.474562 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:37.474092 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/948c9256-224e-472e-8ccf-5f6d761b5e0c-credential-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:59:37.474562 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:37.474107 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c88b7e3-1f93-456a-87ed-4894a1de4b6c-workload-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:59:37.474562 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:37.474120 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/948c9256-224e-472e-8ccf-5f6d761b5e0c-istio-token\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:59:37.474562 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:37.474132 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chff7\" (UniqueName: \"kubernetes.io/projected/948c9256-224e-472e-8ccf-5f6d761b5e0c-kube-api-access-chff7\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:59:37.475740 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:37.475711 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c88b7e3-1f93-456a-87ed-4894a1de4b6c-kube-api-access-ctj54" pod "8c88b7e3-1f93-456a-87ed-4894a1de4b6c" (UID: "8c88b7e3-1f93-456a-87ed-4894a1de4b6c"). InnerVolumeSpecName "kube-api-access-ctj54". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:59:37.476126 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:37.476106 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c88b7e3-1f93-456a-87ed-4894a1de4b6c-istio-token" pod "8c88b7e3-1f93-456a-87ed-4894a1de4b6c" (UID: "8c88b7e3-1f93-456a-87ed-4894a1de4b6c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:59:37.476226 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:37.476171 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c88b7e3-1f93-456a-87ed-4894a1de4b6c-istio-envoy" pod "8c88b7e3-1f93-456a-87ed-4894a1de4b6c" (UID: "8c88b7e3-1f93-456a-87ed-4894a1de4b6c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:37.476332 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:37.476314 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8c88b7e3-1f93-456a-87ed-4894a1de4b6c-istio-podinfo" pod "8c88b7e3-1f93-456a-87ed-4894a1de4b6c" (UID: "8c88b7e3-1f93-456a-87ed-4894a1de4b6c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:59:37.575416 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:37.575328 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c88b7e3-1f93-456a-87ed-4894a1de4b6c-istio-envoy\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:59:37.575416 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:37.575359 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c88b7e3-1f93-456a-87ed-4894a1de4b6c-istio-podinfo\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:59:37.575416 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:37.575368 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c88b7e3-1f93-456a-87ed-4894a1de4b6c-istio-token\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:59:37.575416 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:37.575378 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctj54\" (UniqueName: \"kubernetes.io/projected/8c88b7e3-1f93-456a-87ed-4894a1de4b6c-kube-api-access-ctj54\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:59:37.885913 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:37.885816 4493 generic.go:356] "Generic (PLEG): container finished" podID="8c88b7e3-1f93-456a-87ed-4894a1de4b6c" containerID="d85e60dc92c35d3c7d597c161adb0d1f2ebfd6787730288e39a4bc07579f4dd4" exitCode=0 Oct 30 16:59:37.885913 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:37.885864 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp" Oct 30 16:59:37.886410 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:37.885903 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp" event={"ID":"8c88b7e3-1f93-456a-87ed-4894a1de4b6c","Type":"ContainerDied","Data":"d85e60dc92c35d3c7d597c161adb0d1f2ebfd6787730288e39a4bc07579f4dd4"} Oct 30 16:59:37.886410 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:37.885977 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp" event={"ID":"8c88b7e3-1f93-456a-87ed-4894a1de4b6c","Type":"ContainerDied","Data":"6ba30ce32cfcdf975d31f13c1a73a941dc7208179031cee1146651eb0c09c9cf"} Oct 30 16:59:37.886410 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:37.886009 4493 scope.go:122] "RemoveContainer" containerID="d85e60dc92c35d3c7d597c161adb0d1f2ebfd6787730288e39a4bc07579f4dd4" Oct 30 16:59:37.887413 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:37.887395 4493 generic.go:356] "Generic (PLEG): container finished" podID="948c9256-224e-472e-8ccf-5f6d761b5e0c" containerID="20026c763422eeba9d8b3b052bf64a433460a11aa284fce30f7d4d83533f6ff3" exitCode=0 Oct 30 16:59:37.887503 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:37.887438 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd" Oct 30 16:59:37.887503 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:37.887485 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd" event={"ID":"948c9256-224e-472e-8ccf-5f6d761b5e0c","Type":"ContainerDied","Data":"20026c763422eeba9d8b3b052bf64a433460a11aa284fce30f7d4d83533f6ff3"} Oct 30 16:59:37.887646 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:37.887580 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd" event={"ID":"948c9256-224e-472e-8ccf-5f6d761b5e0c","Type":"ContainerDied","Data":"cb0e9104c15fe47243eb83ae544cb5ded24b45a80ac5f764d39b67248533dcca"} Oct 30 16:59:37.897337 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:37.897229 4493 scope.go:122] "RemoveContainer" containerID="d85e60dc92c35d3c7d597c161adb0d1f2ebfd6787730288e39a4bc07579f4dd4" Oct 30 16:59:37.897562 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:59:37.897534 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d85e60dc92c35d3c7d597c161adb0d1f2ebfd6787730288e39a4bc07579f4dd4\": container with ID starting with d85e60dc92c35d3c7d597c161adb0d1f2ebfd6787730288e39a4bc07579f4dd4 not found: ID does not exist" containerID="d85e60dc92c35d3c7d597c161adb0d1f2ebfd6787730288e39a4bc07579f4dd4" Oct 30 16:59:37.897665 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:37.897574 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d85e60dc92c35d3c7d597c161adb0d1f2ebfd6787730288e39a4bc07579f4dd4"} err="failed to get container status \"d85e60dc92c35d3c7d597c161adb0d1f2ebfd6787730288e39a4bc07579f4dd4\": rpc error: code = NotFound desc = could not find container \"d85e60dc92c35d3c7d597c161adb0d1f2ebfd6787730288e39a4bc07579f4dd4\": container with ID starting with d85e60dc92c35d3c7d597c161adb0d1f2ebfd6787730288e39a4bc07579f4dd4 not found: ID does not exist" Oct 30 16:59:37.897665 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:37.897597 4493 scope.go:122] "RemoveContainer" containerID="20026c763422eeba9d8b3b052bf64a433460a11aa284fce30f7d4d83533f6ff3" Oct 30 16:59:37.907920 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:37.907901 4493 scope.go:122] "RemoveContainer" containerID="20026c763422eeba9d8b3b052bf64a433460a11aa284fce30f7d4d83533f6ff3" Oct 30 16:59:37.908227 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:59:37.908202 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20026c763422eeba9d8b3b052bf64a433460a11aa284fce30f7d4d83533f6ff3\": container with ID starting with 20026c763422eeba9d8b3b052bf64a433460a11aa284fce30f7d4d83533f6ff3 not found: ID does not exist" containerID="20026c763422eeba9d8b3b052bf64a433460a11aa284fce30f7d4d83533f6ff3" Oct 30 16:59:37.908314 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:37.908232 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20026c763422eeba9d8b3b052bf64a433460a11aa284fce30f7d4d83533f6ff3"} err="failed to get container status \"20026c763422eeba9d8b3b052bf64a433460a11aa284fce30f7d4d83533f6ff3\": rpc error: code = NotFound desc = could not find container \"20026c763422eeba9d8b3b052bf64a433460a11aa284fce30f7d4d83533f6ff3\": container with ID starting with 20026c763422eeba9d8b3b052bf64a433460a11aa284fce30f7d4d83533f6ff3 not found: ID does not exist" Oct 30 16:59:37.941376 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:37.941345 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd"] Oct 30 16:59:37.952587 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:37.952547 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-4l9fd"] Oct 30 16:59:37.986727 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:37.986693 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp"] Oct 30 16:59:37.991963 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:37.991937 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-86nzp"] Oct 30 16:59:38.933582 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:38.933551 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59ae5769-a6cf-444d-947e-313ff39785a5" path="/var/lib/kubelet/pods/59ae5769-a6cf-444d-947e-313ff39785a5/volumes" Oct 30 16:59:38.934097 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:38.934082 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c88b7e3-1f93-456a-87ed-4894a1de4b6c" path="/var/lib/kubelet/pods/8c88b7e3-1f93-456a-87ed-4894a1de4b6c/volumes" Oct 30 16:59:38.934556 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:38.934543 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="948c9256-224e-472e-8ccf-5f6d761b5e0c" path="/var/lib/kubelet/pods/948c9256-224e-472e-8ccf-5f6d761b5e0c/volumes" Oct 30 16:59:39.425821 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:39.425754 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 16:59:40.474506 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:40.474476 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7437-e427f8"] Oct 30 16:59:40.475585 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:40.475565 4493 cpu_manager.go:414] "containerMap: removing container" podUID="59ae5769-a6cf-444d-947e-313ff39785a5" containerName="istio-proxy" Oct 30 16:59:40.475685 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:40.475676 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59ae5769-a6cf-444d-947e-313ff39785a5" containerName="istio-proxy" Oct 30 16:59:40.475748 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:40.475741 4493 cpu_manager.go:414] "containerMap: removing container" podUID="948c9256-224e-472e-8ccf-5f6d761b5e0c" containerName="istio-proxy" Oct 30 16:59:40.475838 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:40.475825 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="948c9256-224e-472e-8ccf-5f6d761b5e0c" containerName="istio-proxy" Oct 30 16:59:40.475928 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:40.475919 4493 cpu_manager.go:414] "containerMap: removing container" podUID="67c90611-0953-4152-8696-695093024e17" containerName="main" Oct 30 16:59:40.475999 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:40.475990 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67c90611-0953-4152-8696-695093024e17" containerName="main" Oct 30 16:59:40.476089 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:40.476080 4493 cpu_manager.go:414] "containerMap: removing container" podUID="540a4a45-3790-4207-b753-ce03bafbe504" containerName="revoke-keys" Oct 30 16:59:40.476159 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:40.476150 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="540a4a45-3790-4207-b753-ce03bafbe504" containerName="revoke-keys" Oct 30 16:59:40.476253 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:40.476244 4493 cpu_manager.go:414] "containerMap: removing container" podUID="8c88b7e3-1f93-456a-87ed-4894a1de4b6c" containerName="istio-proxy" Oct 30 16:59:40.476325 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:40.476314 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c88b7e3-1f93-456a-87ed-4894a1de4b6c" containerName="istio-proxy" Oct 30 16:59:40.476539 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:40.476529 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="540a4a45-3790-4207-b753-ce03bafbe504" containerName="revoke-keys" Oct 30 16:59:40.476622 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:40.476614 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c88b7e3-1f93-456a-87ed-4894a1de4b6c" containerName="istio-proxy" Oct 30 16:59:40.476697 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:40.476688 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="59ae5769-a6cf-444d-947e-313ff39785a5" containerName="istio-proxy" Oct 30 16:59:40.476764 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:40.476756 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="948c9256-224e-472e-8ccf-5f6d761b5e0c" containerName="istio-proxy" Oct 30 16:59:40.476853 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:40.476844 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="67c90611-0953-4152-8696-695093024e17" containerName="main" Oct 30 16:59:40.486137 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:40.486115 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7437-e427f8" Oct 30 16:59:40.503804 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:40.501376 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7437-e427f8"] Oct 30 16:59:40.509193 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:40.507809 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-qn5rq\"" Oct 30 16:59:40.631721 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:40.631680 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxlq7\" (UniqueName: \"kubernetes.io/projected/f6746d01-affd-44d9-aa03-a0743bef04c4-kube-api-access-rxlq7\") pod \"test-curl-7437-e427f8\" (UID: \"f6746d01-affd-44d9-aa03-a0743bef04c4\") " pod="opendatahub/test-curl-7437-e427f8" Oct 30 16:59:40.733736 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:40.733648 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxlq7\" (UniqueName: \"kubernetes.io/projected/f6746d01-affd-44d9-aa03-a0743bef04c4-kube-api-access-rxlq7\") pod \"test-curl-7437-e427f8\" (UID: \"f6746d01-affd-44d9-aa03-a0743bef04c4\") " pod="opendatahub/test-curl-7437-e427f8" Oct 30 16:59:40.756764 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:40.756720 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxlq7\" (UniqueName: \"kubernetes.io/projected/f6746d01-affd-44d9-aa03-a0743bef04c4-kube-api-access-rxlq7\") pod \"test-curl-7437-e427f8\" (UID: \"f6746d01-affd-44d9-aa03-a0743bef04c4\") " pod="opendatahub/test-curl-7437-e427f8" Oct 30 16:59:40.819048 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:40.819011 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7437-e427f8" Oct 30 16:59:41.136035 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:41.135992 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7437-e427f8"] Oct 30 16:59:41.138834 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:59:41.138767 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf6746d01_affd_44d9_aa03_a0743bef04c4.slice/crio-716946d5cd5477ad8866822fc976432dda5c0f697daec061492b1a22804cc0a3 WatchSource:0}: Error finding container 716946d5cd5477ad8866822fc976432dda5c0f697daec061492b1a22804cc0a3: Status 404 returned error can't find the container with id 716946d5cd5477ad8866822fc976432dda5c0f697daec061492b1a22804cc0a3 Oct 30 16:59:41.140920 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:41.140901 4493 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 16:59:41.929266 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:41.929230 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7437-e427f8" event={"ID":"f6746d01-affd-44d9-aa03-a0743bef04c4","Type":"ContainerStarted","Data":"716946d5cd5477ad8866822fc976432dda5c0f697daec061492b1a22804cc0a3"} Oct 30 16:59:42.939573 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:42.939537 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7437-e427f8" event={"ID":"f6746d01-affd-44d9-aa03-a0743bef04c4","Type":"ContainerStarted","Data":"71608a966e0c6520184ac4d6cdb3bc5bff30e1d3abf1ae2d36715d902dd3c93d"} Oct 30 16:59:42.977159 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:42.977111 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7437-e427f8" podStartSLOduration=2.054799696 podStartE2EDuration="2.977098684s" totalImagesPullingTime="922.298988ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:59:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:59:42.976705622 +0000 UTC m=+2210.203880722" watchObservedRunningTime="2026-10-30 16:59:42.977098684 +0000 UTC m=+2210.204273800" Oct 30 16:59:43.892809 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:43.883794 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7437-e427f8"] Oct 30 16:59:43.963017 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:43.962922 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7437-e427f8"] Oct 30 16:59:44.079683 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:44.079650 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7437-52d944"] Oct 30 16:59:44.080185 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:44.080170 4493 cpu_manager.go:414] "containerMap: removing container" podUID="f6746d01-affd-44d9-aa03-a0743bef04c4" containerName="test-curl-7437-e427f8" Oct 30 16:59:44.080238 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:44.080187 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6746d01-affd-44d9-aa03-a0743bef04c4" containerName="test-curl-7437-e427f8" Oct 30 16:59:44.080304 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:44.080295 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6746d01-affd-44d9-aa03-a0743bef04c4" containerName="test-curl-7437-e427f8" Oct 30 16:59:44.084042 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:44.084022 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7437-52d944" Oct 30 16:59:44.104125 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:44.104086 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7437-52d944"] Oct 30 16:59:44.126013 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:44.125975 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-629kq\" (UniqueName: \"kubernetes.io/projected/6754b735-afa1-4ef2-a341-022d4f1d8c3e-kube-api-access-629kq\") pod \"test-curl-7437-52d944\" (UID: \"6754b735-afa1-4ef2-a341-022d4f1d8c3e\") " pod="opendatahub/test-curl-7437-52d944" Oct 30 16:59:44.227409 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:44.227320 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-629kq\" (UniqueName: \"kubernetes.io/projected/6754b735-afa1-4ef2-a341-022d4f1d8c3e-kube-api-access-629kq\") pod \"test-curl-7437-52d944\" (UID: \"6754b735-afa1-4ef2-a341-022d4f1d8c3e\") " pod="opendatahub/test-curl-7437-52d944" Oct 30 16:59:44.251176 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:44.251141 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-629kq\" (UniqueName: \"kubernetes.io/projected/6754b735-afa1-4ef2-a341-022d4f1d8c3e-kube-api-access-629kq\") pod \"test-curl-7437-52d944\" (UID: \"6754b735-afa1-4ef2-a341-022d4f1d8c3e\") " pod="opendatahub/test-curl-7437-52d944" Oct 30 16:59:44.398579 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:44.398548 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7437-52d944" Oct 30 16:59:44.719560 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:44.710811 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7437-52d944"] Oct 30 16:59:44.722314 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:59:44.722285 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6754b735_afa1_4ef2_a341_022d4f1d8c3e.slice/crio-6b00a8b2d29c88578774a7f21662c17edce4786e4154dd47907793897359f2a6 WatchSource:0}: Error finding container 6b00a8b2d29c88578774a7f21662c17edce4786e4154dd47907793897359f2a6: Status 404 returned error can't find the container with id 6b00a8b2d29c88578774a7f21662c17edce4786e4154dd47907793897359f2a6 Oct 30 16:59:44.972907 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:44.972820 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7437-52d944" event={"ID":"6754b735-afa1-4ef2-a341-022d4f1d8c3e","Type":"ContainerStarted","Data":"ff78602d38ac0726dabb6a610d4f898908c23a92aee4a0fc0e75f97add003a67"} Oct 30 16:59:44.972907 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:44.972873 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7437-52d944" event={"ID":"6754b735-afa1-4ef2-a341-022d4f1d8c3e","Type":"ContainerStarted","Data":"6b00a8b2d29c88578774a7f21662c17edce4786e4154dd47907793897359f2a6"} Oct 30 16:59:44.973420 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:44.972909 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7437-e427f8" podUID="f6746d01-affd-44d9-aa03-a0743bef04c4" containerName="test-curl-7437-e427f8" containerID="cri-o://71608a966e0c6520184ac4d6cdb3bc5bff30e1d3abf1ae2d36715d902dd3c93d" gracePeriod=2 Oct 30 16:59:45.013367 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:45.013319 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7437-52d944" podStartSLOduration=1.0133069 podStartE2EDuration="1.0133069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:59:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:59:45.010252763 +0000 UTC m=+2212.237427864" watchObservedRunningTime="2026-10-30 16:59:45.0133069 +0000 UTC m=+2212.240482000" Oct 30 16:59:45.836939 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:45.836730 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7437-52d944"] Oct 30 16:59:45.841586 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:45.841555 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7437-52d944"] Oct 30 16:59:46.002648 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:46.002613 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7437-a963c2"] Oct 30 16:59:46.003191 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:46.003176 4493 cpu_manager.go:414] "containerMap: removing container" podUID="6754b735-afa1-4ef2-a341-022d4f1d8c3e" containerName="test-curl-7437-52d944" Oct 30 16:59:46.003244 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:46.003193 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6754b735-afa1-4ef2-a341-022d4f1d8c3e" containerName="test-curl-7437-52d944" Oct 30 16:59:46.003282 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:46.003274 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="6754b735-afa1-4ef2-a341-022d4f1d8c3e" containerName="test-curl-7437-52d944" Oct 30 16:59:46.006895 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:46.006875 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7437-a963c2" Oct 30 16:59:46.026812 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:46.026725 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7437-a963c2"] Oct 30 16:59:46.045956 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:46.045932 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7gcw\" (UniqueName: \"kubernetes.io/projected/5cb6d557-8502-4b75-85a5-215564eb9566-kube-api-access-t7gcw\") pod \"test-curl-7437-a963c2\" (UID: \"5cb6d557-8502-4b75-85a5-215564eb9566\") " pod="opendatahub/test-curl-7437-a963c2" Oct 30 16:59:46.146946 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:46.146909 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7gcw\" (UniqueName: \"kubernetes.io/projected/5cb6d557-8502-4b75-85a5-215564eb9566-kube-api-access-t7gcw\") pod \"test-curl-7437-a963c2\" (UID: \"5cb6d557-8502-4b75-85a5-215564eb9566\") " pod="opendatahub/test-curl-7437-a963c2" Oct 30 16:59:46.188851 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:46.188820 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7gcw\" (UniqueName: \"kubernetes.io/projected/5cb6d557-8502-4b75-85a5-215564eb9566-kube-api-access-t7gcw\") pod \"test-curl-7437-a963c2\" (UID: \"5cb6d557-8502-4b75-85a5-215564eb9566\") " pod="opendatahub/test-curl-7437-a963c2" Oct 30 16:59:46.319359 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:46.319278 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7437-a963c2" Oct 30 16:59:46.628965 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:46.628941 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7437-a963c2"] Oct 30 16:59:46.630712 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:59:46.630683 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5cb6d557_8502_4b75_85a5_215564eb9566.slice/crio-7d98777201a2fbb8db76049a57947cfbf7db678e85fb059a6365f51224880f93 WatchSource:0}: Error finding container 7d98777201a2fbb8db76049a57947cfbf7db678e85fb059a6365f51224880f93: Status 404 returned error can't find the container with id 7d98777201a2fbb8db76049a57947cfbf7db678e85fb059a6365f51224880f93 Oct 30 16:59:46.988346 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:46.988300 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7437-52d944" podUID="6754b735-afa1-4ef2-a341-022d4f1d8c3e" containerName="test-curl-7437-52d944" containerID="cri-o://ff78602d38ac0726dabb6a610d4f898908c23a92aee4a0fc0e75f97add003a67" gracePeriod=2 Oct 30 16:59:46.988473 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:46.988441 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7437-a963c2" event={"ID":"5cb6d557-8502-4b75-85a5-215564eb9566","Type":"ContainerStarted","Data":"46d2c8ccfe0f717a6096d6ecb8c4f33652164435ebe1eb5fd5a9261623caf960"} Oct 30 16:59:46.988613 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:46.988492 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7437-a963c2" event={"ID":"5cb6d557-8502-4b75-85a5-215564eb9566","Type":"ContainerStarted","Data":"7d98777201a2fbb8db76049a57947cfbf7db678e85fb059a6365f51224880f93"} Oct 30 16:59:47.028257 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:47.028214 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7437-a963c2" podStartSLOduration=2.028199821 podStartE2EDuration="2.028199821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:59:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:59:47.028077049 +0000 UTC m=+2214.255252153" watchObservedRunningTime="2026-10-30 16:59:47.028199821 +0000 UTC m=+2214.255374948" Oct 30 16:59:47.182836 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:47.182765 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7437-e427f8" Oct 30 16:59:47.259916 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:47.259834 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6746d01-affd-44d9-aa03-a0743bef04c4-kube-api-access-rxlq7\" (UniqueName: \"kubernetes.io/projected/f6746d01-affd-44d9-aa03-a0743bef04c4-kube-api-access-rxlq7\") pod \"f6746d01-affd-44d9-aa03-a0743bef04c4\" (UID: \"f6746d01-affd-44d9-aa03-a0743bef04c4\") " Oct 30 16:59:47.262110 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:47.262079 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6746d01-affd-44d9-aa03-a0743bef04c4-kube-api-access-rxlq7" pod "f6746d01-affd-44d9-aa03-a0743bef04c4" (UID: "f6746d01-affd-44d9-aa03-a0743bef04c4"). InnerVolumeSpecName "kube-api-access-rxlq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:59:47.361834 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:47.361777 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxlq7\" (UniqueName: \"kubernetes.io/projected/f6746d01-affd-44d9-aa03-a0743bef04c4-kube-api-access-rxlq7\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:59:47.988162 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:47.988131 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7437-a963c2"] Oct 30 16:59:47.993072 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:47.993040 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7437-a963c2"] Oct 30 16:59:48.000115 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:48.000075 4493 generic.go:356] "Generic (PLEG): container finished" podID="f6746d01-affd-44d9-aa03-a0743bef04c4" containerID="71608a966e0c6520184ac4d6cdb3bc5bff30e1d3abf1ae2d36715d902dd3c93d" exitCode=137 Oct 30 16:59:48.001926 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:48.001190 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7437-e427f8" Oct 30 16:59:48.006898 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:48.006879 4493 scope.go:122] "RemoveContainer" containerID="71608a966e0c6520184ac4d6cdb3bc5bff30e1d3abf1ae2d36715d902dd3c93d" Oct 30 16:59:48.030741 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:59:48.027029 4493 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7437-e427f8\" is forbidden: User \"system:node:ip-10-0-132-180.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-180.ec2.internal' and this object" podUID="f6746d01-affd-44d9-aa03-a0743bef04c4" pod="opendatahub/test-curl-7437-e427f8" Oct 30 16:59:48.030741 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:48.028325 4493 scope.go:122] "RemoveContainer" containerID="71608a966e0c6520184ac4d6cdb3bc5bff30e1d3abf1ae2d36715d902dd3c93d" Oct 30 16:59:48.030741 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:59:48.028619 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71608a966e0c6520184ac4d6cdb3bc5bff30e1d3abf1ae2d36715d902dd3c93d\": container with ID starting with 71608a966e0c6520184ac4d6cdb3bc5bff30e1d3abf1ae2d36715d902dd3c93d not found: ID does not exist" containerID="71608a966e0c6520184ac4d6cdb3bc5bff30e1d3abf1ae2d36715d902dd3c93d" Oct 30 16:59:48.030741 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:48.028642 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71608a966e0c6520184ac4d6cdb3bc5bff30e1d3abf1ae2d36715d902dd3c93d"} err="failed to get container status \"71608a966e0c6520184ac4d6cdb3bc5bff30e1d3abf1ae2d36715d902dd3c93d\": rpc error: code = NotFound desc = could not find container \"71608a966e0c6520184ac4d6cdb3bc5bff30e1d3abf1ae2d36715d902dd3c93d\": container with ID starting with 71608a966e0c6520184ac4d6cdb3bc5bff30e1d3abf1ae2d36715d902dd3c93d not found: ID does not exist" Oct 30 16:59:48.933869 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:48.933831 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6746d01-affd-44d9-aa03-a0743bef04c4" path="/var/lib/kubelet/pods/f6746d01-affd-44d9-aa03-a0743bef04c4/volumes" Oct 30 16:59:49.008020 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:49.007970 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7437-a963c2" podUID="5cb6d557-8502-4b75-85a5-215564eb9566" containerName="test-curl-7437-a963c2" containerID="cri-o://46d2c8ccfe0f717a6096d6ecb8c4f33652164435ebe1eb5fd5a9261623caf960" gracePeriod=2 Oct 30 16:59:49.223697 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:49.223668 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7437-52d944" Oct 30 16:59:49.282371 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:49.282336 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6754b735-afa1-4ef2-a341-022d4f1d8c3e-kube-api-access-629kq\" (UniqueName: \"kubernetes.io/projected/6754b735-afa1-4ef2-a341-022d4f1d8c3e-kube-api-access-629kq\") pod \"6754b735-afa1-4ef2-a341-022d4f1d8c3e\" (UID: \"6754b735-afa1-4ef2-a341-022d4f1d8c3e\") " Oct 30 16:59:49.284605 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:49.284578 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6754b735-afa1-4ef2-a341-022d4f1d8c3e-kube-api-access-629kq" pod "6754b735-afa1-4ef2-a341-022d4f1d8c3e" (UID: "6754b735-afa1-4ef2-a341-022d4f1d8c3e"). InnerVolumeSpecName "kube-api-access-629kq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:59:49.384031 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:49.384001 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-629kq\" (UniqueName: \"kubernetes.io/projected/6754b735-afa1-4ef2-a341-022d4f1d8c3e-kube-api-access-629kq\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:59:50.016060 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:50.016026 4493 generic.go:356] "Generic (PLEG): container finished" podID="6754b735-afa1-4ef2-a341-022d4f1d8c3e" containerID="ff78602d38ac0726dabb6a610d4f898908c23a92aee4a0fc0e75f97add003a67" exitCode=137 Oct 30 16:59:50.016224 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:50.016068 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7437-52d944" Oct 30 16:59:50.016224 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:50.016130 4493 scope.go:122] "RemoveContainer" containerID="ff78602d38ac0726dabb6a610d4f898908c23a92aee4a0fc0e75f97add003a67" Oct 30 16:59:50.027331 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:50.027311 4493 scope.go:122] "RemoveContainer" containerID="ff78602d38ac0726dabb6a610d4f898908c23a92aee4a0fc0e75f97add003a67" Oct 30 16:59:50.027617 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:59:50.027599 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff78602d38ac0726dabb6a610d4f898908c23a92aee4a0fc0e75f97add003a67\": container with ID starting with ff78602d38ac0726dabb6a610d4f898908c23a92aee4a0fc0e75f97add003a67 not found: ID does not exist" containerID="ff78602d38ac0726dabb6a610d4f898908c23a92aee4a0fc0e75f97add003a67" Oct 30 16:59:50.027707 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:50.027624 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff78602d38ac0726dabb6a610d4f898908c23a92aee4a0fc0e75f97add003a67"} err="failed to get container status \"ff78602d38ac0726dabb6a610d4f898908c23a92aee4a0fc0e75f97add003a67\": rpc error: code = NotFound desc = could not find container \"ff78602d38ac0726dabb6a610d4f898908c23a92aee4a0fc0e75f97add003a67\": container with ID starting with ff78602d38ac0726dabb6a610d4f898908c23a92aee4a0fc0e75f97add003a67 not found: ID does not exist" Oct 30 16:59:50.037705 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:59:50.037674 4493 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7437-52d944\" is forbidden: User \"system:node:ip-10-0-132-180.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-180.ec2.internal' and this object" podUID="6754b735-afa1-4ef2-a341-022d4f1d8c3e" pod="opendatahub/test-curl-7437-52d944" Oct 30 16:59:50.068531 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:50.068494 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7437-6729a0"] Oct 30 16:59:50.069099 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:50.069084 4493 cpu_manager.go:414] "containerMap: removing container" podUID="5cb6d557-8502-4b75-85a5-215564eb9566" containerName="test-curl-7437-a963c2" Oct 30 16:59:50.069163 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:50.069101 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cb6d557-8502-4b75-85a5-215564eb9566" containerName="test-curl-7437-a963c2" Oct 30 16:59:50.069223 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:50.069214 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cb6d557-8502-4b75-85a5-215564eb9566" containerName="test-curl-7437-a963c2" Oct 30 16:59:50.073683 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:50.073666 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7437-6729a0" Oct 30 16:59:50.090703 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:50.090676 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5bkk\" (UniqueName: \"kubernetes.io/projected/69503235-3784-4e31-a129-4a987692346f-kube-api-access-f5bkk\") pod \"test-curl-7437-6729a0\" (UID: \"69503235-3784-4e31-a129-4a987692346f\") " pod="opendatahub/test-curl-7437-6729a0" Oct 30 16:59:50.093639 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:50.093615 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7437-6729a0"] Oct 30 16:59:50.124961 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:59:50.124887 4493 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7437-52d944\" is forbidden: User \"system:node:ip-10-0-132-180.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-180.ec2.internal' and this object" podUID="6754b735-afa1-4ef2-a341-022d4f1d8c3e" pod="opendatahub/test-curl-7437-52d944" Oct 30 16:59:50.191969 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:50.191938 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5bkk\" (UniqueName: \"kubernetes.io/projected/69503235-3784-4e31-a129-4a987692346f-kube-api-access-f5bkk\") pod \"test-curl-7437-6729a0\" (UID: \"69503235-3784-4e31-a129-4a987692346f\") " pod="opendatahub/test-curl-7437-6729a0" Oct 30 16:59:50.216077 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:50.216046 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5bkk\" (UniqueName: \"kubernetes.io/projected/69503235-3784-4e31-a129-4a987692346f-kube-api-access-f5bkk\") pod \"test-curl-7437-6729a0\" (UID: \"69503235-3784-4e31-a129-4a987692346f\") " pod="opendatahub/test-curl-7437-6729a0" Oct 30 16:59:50.386779 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:50.386698 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7437-6729a0" Oct 30 16:59:50.691638 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:50.691611 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7437-6729a0"] Oct 30 16:59:50.692361 ip-10-0-132-180 kubenswrapper[4493]: W1030 16:59:50.692331 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod69503235_3784_4e31_a129_4a987692346f.slice/crio-47e9ebb6a7defe456399aafb8a4de66fb9c0cdb6fe859436518d42bea8fac5ed WatchSource:0}: Error finding container 47e9ebb6a7defe456399aafb8a4de66fb9c0cdb6fe859436518d42bea8fac5ed: Status 404 returned error can't find the container with id 47e9ebb6a7defe456399aafb8a4de66fb9c0cdb6fe859436518d42bea8fac5ed Oct 30 16:59:50.936558 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:50.936522 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6754b735-afa1-4ef2-a341-022d4f1d8c3e" path="/var/lib/kubelet/pods/6754b735-afa1-4ef2-a341-022d4f1d8c3e/volumes" Oct 30 16:59:51.027923 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:51.027893 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7437-6729a0" event={"ID":"69503235-3784-4e31-a129-4a987692346f","Type":"ContainerStarted","Data":"9356f271f270e211c28b42e22258ee071c4095303991ae20613981c6e3d2a5a1"} Oct 30 16:59:51.028032 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:51.027940 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7437-6729a0" event={"ID":"69503235-3784-4e31-a129-4a987692346f","Type":"ContainerStarted","Data":"47e9ebb6a7defe456399aafb8a4de66fb9c0cdb6fe859436518d42bea8fac5ed"} Oct 30 16:59:51.067204 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:51.067160 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7437-6729a0" podStartSLOduration=1.067147163 podStartE2EDuration="1.067147163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:59:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:59:51.066202739 +0000 UTC m=+2218.293377841" watchObservedRunningTime="2026-10-30 16:59:51.067147163 +0000 UTC m=+2218.294322309" Oct 30 16:59:51.205096 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:51.205070 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7437-a963c2" Oct 30 16:59:51.304310 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:51.304219 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cb6d557-8502-4b75-85a5-215564eb9566-kube-api-access-t7gcw\" (UniqueName: \"kubernetes.io/projected/5cb6d557-8502-4b75-85a5-215564eb9566-kube-api-access-t7gcw\") pod \"5cb6d557-8502-4b75-85a5-215564eb9566\" (UID: \"5cb6d557-8502-4b75-85a5-215564eb9566\") " Oct 30 16:59:51.307319 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:51.307287 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cb6d557-8502-4b75-85a5-215564eb9566-kube-api-access-t7gcw" pod "5cb6d557-8502-4b75-85a5-215564eb9566" (UID: "5cb6d557-8502-4b75-85a5-215564eb9566"). InnerVolumeSpecName "kube-api-access-t7gcw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:59:51.405983 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:51.405948 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7gcw\" (UniqueName: \"kubernetes.io/projected/5cb6d557-8502-4b75-85a5-215564eb9566-kube-api-access-t7gcw\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:59:51.949627 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:51.949586 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7437-6729a0"] Oct 30 16:59:51.956289 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:51.956261 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7437-6729a0"] Oct 30 16:59:52.066020 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:52.065988 4493 generic.go:356] "Generic (PLEG): container finished" podID="5cb6d557-8502-4b75-85a5-215564eb9566" containerID="46d2c8ccfe0f717a6096d6ecb8c4f33652164435ebe1eb5fd5a9261623caf960" exitCode=137 Oct 30 16:59:52.066181 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:52.066045 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7437-a963c2" Oct 30 16:59:52.066181 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:52.066093 4493 scope.go:122] "RemoveContainer" containerID="46d2c8ccfe0f717a6096d6ecb8c4f33652164435ebe1eb5fd5a9261623caf960" Oct 30 16:59:52.079513 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:52.079494 4493 scope.go:122] "RemoveContainer" containerID="46d2c8ccfe0f717a6096d6ecb8c4f33652164435ebe1eb5fd5a9261623caf960" Oct 30 16:59:52.079907 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:59:52.079887 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46d2c8ccfe0f717a6096d6ecb8c4f33652164435ebe1eb5fd5a9261623caf960\": container with ID starting with 46d2c8ccfe0f717a6096d6ecb8c4f33652164435ebe1eb5fd5a9261623caf960 not found: ID does not exist" containerID="46d2c8ccfe0f717a6096d6ecb8c4f33652164435ebe1eb5fd5a9261623caf960" Oct 30 16:59:52.079963 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:52.079927 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46d2c8ccfe0f717a6096d6ecb8c4f33652164435ebe1eb5fd5a9261623caf960"} err="failed to get container status \"46d2c8ccfe0f717a6096d6ecb8c4f33652164435ebe1eb5fd5a9261623caf960\": rpc error: code = NotFound desc = could not find container \"46d2c8ccfe0f717a6096d6ecb8c4f33652164435ebe1eb5fd5a9261623caf960\": container with ID starting with 46d2c8ccfe0f717a6096d6ecb8c4f33652164435ebe1eb5fd5a9261623caf960 not found: ID does not exist" Oct 30 16:59:52.088311 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:59:52.088275 4493 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7437-a963c2\" is forbidden: User \"system:node:ip-10-0-132-180.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-180.ec2.internal' and this object" podUID="5cb6d557-8502-4b75-85a5-215564eb9566" pod="opendatahub/test-curl-7437-a963c2" Oct 30 16:59:52.936209 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:52.936173 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cb6d557-8502-4b75-85a5-215564eb9566" path="/var/lib/kubelet/pods/5cb6d557-8502-4b75-85a5-215564eb9566/volumes" Oct 30 16:59:52.943281 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:59:52.943247 4493 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7437-a963c2\" is forbidden: User \"system:node:ip-10-0-132-180.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-180.ec2.internal' and this object" podUID="5cb6d557-8502-4b75-85a5-215564eb9566" pod="opendatahub/test-curl-7437-a963c2" Oct 30 16:59:53.075068 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:53.075031 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7437-6729a0" podUID="69503235-3784-4e31-a129-4a987692346f" containerName="test-curl-7437-6729a0" containerID="cri-o://9356f271f270e211c28b42e22258ee071c4095303991ae20613981c6e3d2a5a1" gracePeriod=2 Oct 30 16:59:55.379174 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:55.379148 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7437-6729a0" Oct 30 16:59:55.465774 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:55.465739 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69503235-3784-4e31-a129-4a987692346f-kube-api-access-f5bkk\" (UniqueName: \"kubernetes.io/projected/69503235-3784-4e31-a129-4a987692346f-kube-api-access-f5bkk\") pod \"69503235-3784-4e31-a129-4a987692346f\" (UID: \"69503235-3784-4e31-a129-4a987692346f\") " Oct 30 16:59:55.468130 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:55.468103 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69503235-3784-4e31-a129-4a987692346f-kube-api-access-f5bkk" pod "69503235-3784-4e31-a129-4a987692346f" (UID: "69503235-3784-4e31-a129-4a987692346f"). InnerVolumeSpecName "kube-api-access-f5bkk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:59:55.567678 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:55.567631 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5bkk\" (UniqueName: \"kubernetes.io/projected/69503235-3784-4e31-a129-4a987692346f-kube-api-access-f5bkk\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 16:59:55.885108 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:55.885075 4493 scope.go:122] "RemoveContainer" containerID="666ecdb459cd1a0cb5c487b2898b809c3deb46b0a3906e96f35969fadae0d5e1" Oct 30 16:59:55.895409 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:55.895386 4493 scope.go:122] "RemoveContainer" containerID="66303bba1c7588d7534fbf8b6b9566a9ef094e75f33d0c858ee9708fa27714d1" Oct 30 16:59:56.130565 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:56.130531 4493 generic.go:356] "Generic (PLEG): container finished" podID="69503235-3784-4e31-a129-4a987692346f" containerID="9356f271f270e211c28b42e22258ee071c4095303991ae20613981c6e3d2a5a1" exitCode=137 Oct 30 16:59:56.130772 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:56.130578 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7437-6729a0" Oct 30 16:59:56.130772 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:56.130636 4493 scope.go:122] "RemoveContainer" containerID="9356f271f270e211c28b42e22258ee071c4095303991ae20613981c6e3d2a5a1" Oct 30 16:59:56.142375 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:56.142351 4493 scope.go:122] "RemoveContainer" containerID="9356f271f270e211c28b42e22258ee071c4095303991ae20613981c6e3d2a5a1" Oct 30 16:59:56.142688 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:59:56.142664 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9356f271f270e211c28b42e22258ee071c4095303991ae20613981c6e3d2a5a1\": container with ID starting with 9356f271f270e211c28b42e22258ee071c4095303991ae20613981c6e3d2a5a1 not found: ID does not exist" containerID="9356f271f270e211c28b42e22258ee071c4095303991ae20613981c6e3d2a5a1" Oct 30 16:59:56.142747 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:56.142702 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9356f271f270e211c28b42e22258ee071c4095303991ae20613981c6e3d2a5a1"} err="failed to get container status \"9356f271f270e211c28b42e22258ee071c4095303991ae20613981c6e3d2a5a1\": rpc error: code = NotFound desc = could not find container \"9356f271f270e211c28b42e22258ee071c4095303991ae20613981c6e3d2a5a1\": container with ID starting with 9356f271f270e211c28b42e22258ee071c4095303991ae20613981c6e3d2a5a1 not found: ID does not exist" Oct 30 16:59:56.154142 ip-10-0-132-180 kubenswrapper[4493]: E1030 16:59:56.154107 4493 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7437-6729a0\" is forbidden: User \"system:node:ip-10-0-132-180.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-180.ec2.internal' and this object" podUID="69503235-3784-4e31-a129-4a987692346f" pod="opendatahub/test-curl-7437-6729a0" Oct 30 16:59:56.936064 ip-10-0-132-180 kubenswrapper[4493]: I1030 16:59:56.936027 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69503235-3784-4e31-a129-4a987692346f" path="/var/lib/kubelet/pods/69503235-3784-4e31-a129-4a987692346f/volumes" Oct 30 17:00:00.279344 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.279305 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889660-2bb5p"] Oct 30 17:00:00.280224 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.280202 4493 cpu_manager.go:414] "containerMap: removing container" podUID="69503235-3784-4e31-a129-4a987692346f" containerName="test-curl-7437-6729a0" Oct 30 17:00:00.280379 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.280227 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69503235-3784-4e31-a129-4a987692346f" containerName="test-curl-7437-6729a0" Oct 30 17:00:00.280442 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.280396 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="69503235-3784-4e31-a129-4a987692346f" containerName="test-curl-7437-6729a0" Oct 30 17:00:00.286167 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.286132 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-06e7f6-29889660-xnlfg"] Oct 30 17:00:00.286329 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.286273 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889660-2bb5p" Oct 30 17:00:00.290804 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.290762 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889660-n8scx"] Oct 30 17:00:00.290940 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.290914 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-06e7f6-29889660-xnlfg" Oct 30 17:00:00.295440 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.295408 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889660-2rqc5"] Oct 30 17:00:00.295581 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.295492 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889660-n8scx" Oct 30 17:00:00.299649 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.299623 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889660-bwdvj"] Oct 30 17:00:00.299778 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.299747 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889660-2rqc5" Oct 30 17:00:00.302199 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.302162 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:00:00.304756 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.304740 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889660-bwdvj" Oct 30 17:00:00.314724 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.309922 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889660-2rqc5"] Oct 30 17:00:00.314724 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.313942 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 17:00:00.323268 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.322663 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-06e7f6-29889660-xnlfg"] Oct 30 17:00:00.325586 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.325548 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:00:00.338417 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.337736 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889660-n8scx"] Oct 30 17:00:00.389837 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.362030 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vstst\" (UniqueName: \"kubernetes.io/projected/d2b861e7-c17b-44a9-86c8-8102ecc2691c-kube-api-access-vstst\") pod \"maas-api-key-cleanup-29889660-bwdvj\" (UID: \"d2b861e7-c17b-44a9-86c8-8102ecc2691c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889660-bwdvj" Oct 30 17:00:00.389837 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.379219 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30b24448-c388-4366-a826-fddc6d987015-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-420ff3-29889660-2bb5p\" (UID: \"30b24448-c388-4366-a826-fddc6d987015\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889660-2bb5p" Oct 30 17:00:00.389837 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.379354 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js2nh\" (UniqueName: \"kubernetes.io/projected/708fd15b-ac7c-42e1-aa73-584566905947-kube-api-access-js2nh\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-06e7f6-29889660-xnlfg\" (UID: \"708fd15b-ac7c-42e1-aa73-584566905947\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-06e7f6-29889660-xnlfg" Oct 30 17:00:00.389837 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.379393 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbc8t\" (UniqueName: \"kubernetes.io/projected/3576003e-ed26-4e09-aef4-1f9bddc69a5d-kube-api-access-sbc8t\") pod \"maas-api-key-cleanup-e2e-worker-w1-a77b48-29889660-n8scx\" (UID: \"3576003e-ed26-4e09-aef4-1f9bddc69a5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889660-n8scx" Oct 30 17:00:00.389837 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.379482 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nn5v\" (UniqueName: \"kubernetes.io/projected/30b24448-c388-4366-a826-fddc6d987015-kube-api-access-7nn5v\") pod \"maas-api-key-cleanup-e2e-worker-w4-420ff3-29889660-2bb5p\" (UID: \"30b24448-c388-4366-a826-fddc6d987015\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889660-2bb5p" Oct 30 17:00:00.389837 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.379594 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/708fd15b-ac7c-42e1-aa73-584566905947-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-06e7f6-29889660-xnlfg\" (UID: \"708fd15b-ac7c-42e1-aa73-584566905947\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-06e7f6-29889660-xnlfg" Oct 30 17:00:00.389837 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.379632 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3576003e-ed26-4e09-aef4-1f9bddc69a5d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-a77b48-29889660-n8scx\" (UID: \"3576003e-ed26-4e09-aef4-1f9bddc69a5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889660-n8scx" Oct 30 17:00:00.389837 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.379664 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69e0b1a3-2a21-4412-9495-29669c26929b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f3503b-29889660-2rqc5\" (UID: \"69e0b1a3-2a21-4412-9495-29669c26929b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889660-2rqc5" Oct 30 17:00:00.389837 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.379750 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drc2n\" (UniqueName: \"kubernetes.io/projected/69e0b1a3-2a21-4412-9495-29669c26929b-kube-api-access-drc2n\") pod \"maas-api-key-cleanup-e2e-worker-w0-f3503b-29889660-2rqc5\" (UID: \"69e0b1a3-2a21-4412-9495-29669c26929b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889660-2rqc5" Oct 30 17:00:00.389837 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.379833 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2b861e7-c17b-44a9-86c8-8102ecc2691c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889660-bwdvj\" (UID: \"d2b861e7-c17b-44a9-86c8-8102ecc2691c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889660-bwdvj" Oct 30 17:00:00.389837 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.380935 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889660-2bb5p"] Oct 30 17:00:00.406055 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:00:00.396593 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 17:00:00.406055 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.399811 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-deb721-29889660-r4c5w"] Oct 30 17:00:00.406055 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.405836 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-deb721-29889660-r4c5w" Oct 30 17:00:00.420381 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.412018 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cf4621-29889660-xbjgx"] Oct 30 17:00:00.426844 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.426818 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889660-bwdvj"] Oct 30 17:00:00.427748 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.427026 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889660-27gc7"] Oct 30 17:00:00.427748 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.427702 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cf4621-29889660-xbjgx" Oct 30 17:00:00.442328 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.442099 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-deb721-29889660-r4c5w"] Oct 30 17:00:00.442328 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.442143 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3ca339-29889660-pbd75"] Oct 30 17:00:00.455879 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.447633 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889660-27gc7" Oct 30 17:00:00.460001 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.459046 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889660-q5xx4"] Oct 30 17:00:00.460001 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.459914 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3ca339-29889660-pbd75" Oct 30 17:00:00.475438 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.475414 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889660-q5xx4" Oct 30 17:00:00.475584 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.475326 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cf4621-29889660-xbjgx"] Oct 30 17:00:00.475584 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.475578 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889660-27gc7"] Oct 30 17:00:00.475704 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.475605 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3ca339-29889660-pbd75"] Oct 30 17:00:00.475704 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.475634 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889660-q5xx4"] Oct 30 17:00:00.482567 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.480752 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/708fd15b-ac7c-42e1-aa73-584566905947-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-06e7f6-29889660-xnlfg\" (UID: \"708fd15b-ac7c-42e1-aa73-584566905947\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-06e7f6-29889660-xnlfg" Oct 30 17:00:00.482567 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.480820 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3576003e-ed26-4e09-aef4-1f9bddc69a5d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-a77b48-29889660-n8scx\" (UID: \"3576003e-ed26-4e09-aef4-1f9bddc69a5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889660-n8scx" Oct 30 17:00:00.482567 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.480850 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69e0b1a3-2a21-4412-9495-29669c26929b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f3503b-29889660-2rqc5\" (UID: \"69e0b1a3-2a21-4412-9495-29669c26929b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889660-2rqc5" Oct 30 17:00:00.482567 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.480917 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drc2n\" (UniqueName: \"kubernetes.io/projected/69e0b1a3-2a21-4412-9495-29669c26929b-kube-api-access-drc2n\") pod \"maas-api-key-cleanup-e2e-worker-w0-f3503b-29889660-2rqc5\" (UID: \"69e0b1a3-2a21-4412-9495-29669c26929b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889660-2rqc5" Oct 30 17:00:00.482567 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.480972 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2b861e7-c17b-44a9-86c8-8102ecc2691c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889660-bwdvj\" (UID: \"d2b861e7-c17b-44a9-86c8-8102ecc2691c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889660-bwdvj" Oct 30 17:00:00.482567 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.480997 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vstst\" (UniqueName: \"kubernetes.io/projected/d2b861e7-c17b-44a9-86c8-8102ecc2691c-kube-api-access-vstst\") pod \"maas-api-key-cleanup-29889660-bwdvj\" (UID: \"d2b861e7-c17b-44a9-86c8-8102ecc2691c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889660-bwdvj" Oct 30 17:00:00.482567 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.481039 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30b24448-c388-4366-a826-fddc6d987015-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-420ff3-29889660-2bb5p\" (UID: \"30b24448-c388-4366-a826-fddc6d987015\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889660-2bb5p" Oct 30 17:00:00.482567 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.481146 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js2nh\" (UniqueName: \"kubernetes.io/projected/708fd15b-ac7c-42e1-aa73-584566905947-kube-api-access-js2nh\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-06e7f6-29889660-xnlfg\" (UID: \"708fd15b-ac7c-42e1-aa73-584566905947\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-06e7f6-29889660-xnlfg" Oct 30 17:00:00.482567 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.481179 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbc8t\" (UniqueName: \"kubernetes.io/projected/3576003e-ed26-4e09-aef4-1f9bddc69a5d-kube-api-access-sbc8t\") pod \"maas-api-key-cleanup-e2e-worker-w1-a77b48-29889660-n8scx\" (UID: \"3576003e-ed26-4e09-aef4-1f9bddc69a5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889660-n8scx" Oct 30 17:00:00.482567 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.481218 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccgsx\" (UniqueName: \"kubernetes.io/projected/305e6ca0-4f0e-42e8-ae53-1e495e07a1a3-kube-api-access-ccgsx\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-deb721-29889660-r4c5w\" (UID: \"305e6ca0-4f0e-42e8-ae53-1e495e07a1a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-deb721-29889660-r4c5w" Oct 30 17:00:00.482567 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.481261 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/305e6ca0-4f0e-42e8-ae53-1e495e07a1a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-deb721-29889660-r4c5w\" (UID: \"305e6ca0-4f0e-42e8-ae53-1e495e07a1a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-deb721-29889660-r4c5w" Oct 30 17:00:00.482567 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.481304 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nn5v\" (UniqueName: \"kubernetes.io/projected/30b24448-c388-4366-a826-fddc6d987015-kube-api-access-7nn5v\") pod \"maas-api-key-cleanup-e2e-worker-w4-420ff3-29889660-2bb5p\" (UID: \"30b24448-c388-4366-a826-fddc6d987015\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889660-2bb5p" Oct 30 17:00:00.482567 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.481682 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3576003e-ed26-4e09-aef4-1f9bddc69a5d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-a77b48-29889660-n8scx\" (UID: \"3576003e-ed26-4e09-aef4-1f9bddc69a5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889660-n8scx" Oct 30 17:00:00.482567 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.482191 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69e0b1a3-2a21-4412-9495-29669c26929b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f3503b-29889660-2rqc5\" (UID: \"69e0b1a3-2a21-4412-9495-29669c26929b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889660-2rqc5" Oct 30 17:00:00.482567 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.482438 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30b24448-c388-4366-a826-fddc6d987015-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-420ff3-29889660-2bb5p\" (UID: \"30b24448-c388-4366-a826-fddc6d987015\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889660-2bb5p" Oct 30 17:00:00.483839 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.482890 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2b861e7-c17b-44a9-86c8-8102ecc2691c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889660-bwdvj\" (UID: \"d2b861e7-c17b-44a9-86c8-8102ecc2691c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889660-bwdvj" Oct 30 17:00:00.483839 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.483314 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/708fd15b-ac7c-42e1-aa73-584566905947-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-06e7f6-29889660-xnlfg\" (UID: \"708fd15b-ac7c-42e1-aa73-584566905947\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-06e7f6-29889660-xnlfg" Oct 30 17:00:00.510631 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.510568 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbc8t\" (UniqueName: \"kubernetes.io/projected/3576003e-ed26-4e09-aef4-1f9bddc69a5d-kube-api-access-sbc8t\") pod \"maas-api-key-cleanup-e2e-worker-w1-a77b48-29889660-n8scx\" (UID: \"3576003e-ed26-4e09-aef4-1f9bddc69a5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889660-n8scx" Oct 30 17:00:00.510631 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.510568 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vstst\" (UniqueName: \"kubernetes.io/projected/d2b861e7-c17b-44a9-86c8-8102ecc2691c-kube-api-access-vstst\") pod \"maas-api-key-cleanup-29889660-bwdvj\" (UID: \"d2b861e7-c17b-44a9-86c8-8102ecc2691c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889660-bwdvj" Oct 30 17:00:00.510852 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.510691 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-js2nh\" (UniqueName: \"kubernetes.io/projected/708fd15b-ac7c-42e1-aa73-584566905947-kube-api-access-js2nh\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-06e7f6-29889660-xnlfg\" (UID: \"708fd15b-ac7c-42e1-aa73-584566905947\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-06e7f6-29889660-xnlfg" Oct 30 17:00:00.511327 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.511301 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nn5v\" (UniqueName: \"kubernetes.io/projected/30b24448-c388-4366-a826-fddc6d987015-kube-api-access-7nn5v\") pod \"maas-api-key-cleanup-e2e-worker-w4-420ff3-29889660-2bb5p\" (UID: \"30b24448-c388-4366-a826-fddc6d987015\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889660-2bb5p" Oct 30 17:00:00.512108 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.512084 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drc2n\" (UniqueName: \"kubernetes.io/projected/69e0b1a3-2a21-4412-9495-29669c26929b-kube-api-access-drc2n\") pod \"maas-api-key-cleanup-e2e-worker-w0-f3503b-29889660-2rqc5\" (UID: \"69e0b1a3-2a21-4412-9495-29669c26929b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889660-2rqc5" Oct 30 17:00:00.582488 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.582397 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccgsx\" (UniqueName: \"kubernetes.io/projected/305e6ca0-4f0e-42e8-ae53-1e495e07a1a3-kube-api-access-ccgsx\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-deb721-29889660-r4c5w\" (UID: \"305e6ca0-4f0e-42e8-ae53-1e495e07a1a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-deb721-29889660-r4c5w" Oct 30 17:00:00.582488 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.582443 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/305e6ca0-4f0e-42e8-ae53-1e495e07a1a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-deb721-29889660-r4c5w\" (UID: \"305e6ca0-4f0e-42e8-ae53-1e495e07a1a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-deb721-29889660-r4c5w" Oct 30 17:00:00.582682 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.582641 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32e4a7ff-2f2e-4140-a540-b59282f6a80f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-2d365c-29889660-27gc7\" (UID: \"32e4a7ff-2f2e-4140-a540-b59282f6a80f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889660-27gc7" Oct 30 17:00:00.582918 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.582897 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94917f70-a899-4b27-86a3-7e420b6856ae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-cf4621-29889660-xbjgx\" (UID: \"94917f70-a899-4b27-86a3-7e420b6856ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cf4621-29889660-xbjgx" Oct 30 17:00:00.582988 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.582935 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69skt\" (UniqueName: \"kubernetes.io/projected/32e4a7ff-2f2e-4140-a540-b59282f6a80f-kube-api-access-69skt\") pod \"maas-api-key-cleanup-e2e-worker-w3-2d365c-29889660-27gc7\" (UID: \"32e4a7ff-2f2e-4140-a540-b59282f6a80f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889660-27gc7" Oct 30 17:00:00.583136 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.583117 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8phsr\" (UniqueName: \"kubernetes.io/projected/94917f70-a899-4b27-86a3-7e420b6856ae-kube-api-access-8phsr\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-cf4621-29889660-xbjgx\" (UID: \"94917f70-a899-4b27-86a3-7e420b6856ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cf4621-29889660-xbjgx" Oct 30 17:00:00.583224 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.583152 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/debf5df6-8130-4b62-a23f-74a234816389-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-18519c-29889660-q5xx4\" (UID: \"debf5df6-8130-4b62-a23f-74a234816389\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889660-q5xx4" Oct 30 17:00:00.583273 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.583232 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/305e6ca0-4f0e-42e8-ae53-1e495e07a1a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-deb721-29889660-r4c5w\" (UID: \"305e6ca0-4f0e-42e8-ae53-1e495e07a1a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-deb721-29889660-r4c5w" Oct 30 17:00:00.583330 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.583309 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8757e465-e3fa-42aa-8911-67323ce3cc2d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-3ca339-29889660-pbd75\" (UID: \"8757e465-e3fa-42aa-8911-67323ce3cc2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3ca339-29889660-pbd75" Oct 30 17:00:00.583408 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.583386 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xp28\" (UniqueName: \"kubernetes.io/projected/8757e465-e3fa-42aa-8911-67323ce3cc2d-kube-api-access-2xp28\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-3ca339-29889660-pbd75\" (UID: \"8757e465-e3fa-42aa-8911-67323ce3cc2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3ca339-29889660-pbd75" Oct 30 17:00:00.583446 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.583426 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mw9jc\" (UniqueName: \"kubernetes.io/projected/debf5df6-8130-4b62-a23f-74a234816389-kube-api-access-mw9jc\") pod \"maas-api-key-cleanup-e2e-worker-w2-18519c-29889660-q5xx4\" (UID: \"debf5df6-8130-4b62-a23f-74a234816389\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889660-q5xx4" Oct 30 17:00:00.604918 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.604889 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889660-2bb5p" Oct 30 17:00:00.605365 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.605335 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccgsx\" (UniqueName: \"kubernetes.io/projected/305e6ca0-4f0e-42e8-ae53-1e495e07a1a3-kube-api-access-ccgsx\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-deb721-29889660-r4c5w\" (UID: \"305e6ca0-4f0e-42e8-ae53-1e495e07a1a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-deb721-29889660-r4c5w" Oct 30 17:00:00.629924 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.629886 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-06e7f6-29889660-xnlfg" Oct 30 17:00:00.664808 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.662842 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889660-n8scx" Oct 30 17:00:00.689942 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.688679 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94917f70-a899-4b27-86a3-7e420b6856ae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-cf4621-29889660-xbjgx\" (UID: \"94917f70-a899-4b27-86a3-7e420b6856ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cf4621-29889660-xbjgx" Oct 30 17:00:00.689942 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.688744 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69skt\" (UniqueName: \"kubernetes.io/projected/32e4a7ff-2f2e-4140-a540-b59282f6a80f-kube-api-access-69skt\") pod \"maas-api-key-cleanup-e2e-worker-w3-2d365c-29889660-27gc7\" (UID: \"32e4a7ff-2f2e-4140-a540-b59282f6a80f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889660-27gc7" Oct 30 17:00:00.689942 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.688870 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8phsr\" (UniqueName: \"kubernetes.io/projected/94917f70-a899-4b27-86a3-7e420b6856ae-kube-api-access-8phsr\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-cf4621-29889660-xbjgx\" (UID: \"94917f70-a899-4b27-86a3-7e420b6856ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cf4621-29889660-xbjgx" Oct 30 17:00:00.689942 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.688902 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/debf5df6-8130-4b62-a23f-74a234816389-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-18519c-29889660-q5xx4\" (UID: \"debf5df6-8130-4b62-a23f-74a234816389\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889660-q5xx4" Oct 30 17:00:00.689942 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.688940 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8757e465-e3fa-42aa-8911-67323ce3cc2d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-3ca339-29889660-pbd75\" (UID: \"8757e465-e3fa-42aa-8911-67323ce3cc2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3ca339-29889660-pbd75" Oct 30 17:00:00.689942 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.688988 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xp28\" (UniqueName: \"kubernetes.io/projected/8757e465-e3fa-42aa-8911-67323ce3cc2d-kube-api-access-2xp28\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-3ca339-29889660-pbd75\" (UID: \"8757e465-e3fa-42aa-8911-67323ce3cc2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3ca339-29889660-pbd75" Oct 30 17:00:00.689942 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.689018 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw9jc\" (UniqueName: \"kubernetes.io/projected/debf5df6-8130-4b62-a23f-74a234816389-kube-api-access-mw9jc\") pod \"maas-api-key-cleanup-e2e-worker-w2-18519c-29889660-q5xx4\" (UID: \"debf5df6-8130-4b62-a23f-74a234816389\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889660-q5xx4" Oct 30 17:00:00.689942 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.689114 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32e4a7ff-2f2e-4140-a540-b59282f6a80f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-2d365c-29889660-27gc7\" (UID: \"32e4a7ff-2f2e-4140-a540-b59282f6a80f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889660-27gc7" Oct 30 17:00:00.690516 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.690058 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32e4a7ff-2f2e-4140-a540-b59282f6a80f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-2d365c-29889660-27gc7\" (UID: \"32e4a7ff-2f2e-4140-a540-b59282f6a80f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889660-27gc7" Oct 30 17:00:00.691699 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.690694 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94917f70-a899-4b27-86a3-7e420b6856ae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-cf4621-29889660-xbjgx\" (UID: \"94917f70-a899-4b27-86a3-7e420b6856ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cf4621-29889660-xbjgx" Oct 30 17:00:00.692153 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.691986 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889660-2rqc5" Oct 30 17:00:00.692417 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.692389 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8757e465-e3fa-42aa-8911-67323ce3cc2d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-3ca339-29889660-pbd75\" (UID: \"8757e465-e3fa-42aa-8911-67323ce3cc2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3ca339-29889660-pbd75" Oct 30 17:00:00.694723 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.693815 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/debf5df6-8130-4b62-a23f-74a234816389-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-18519c-29889660-q5xx4\" (UID: \"debf5df6-8130-4b62-a23f-74a234816389\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889660-q5xx4" Oct 30 17:00:00.719338 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.718910 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889660-bwdvj" Oct 30 17:00:00.719338 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.719276 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69skt\" (UniqueName: \"kubernetes.io/projected/32e4a7ff-2f2e-4140-a540-b59282f6a80f-kube-api-access-69skt\") pod \"maas-api-key-cleanup-e2e-worker-w3-2d365c-29889660-27gc7\" (UID: \"32e4a7ff-2f2e-4140-a540-b59282f6a80f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889660-27gc7" Oct 30 17:00:00.724977 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.724742 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8phsr\" (UniqueName: \"kubernetes.io/projected/94917f70-a899-4b27-86a3-7e420b6856ae-kube-api-access-8phsr\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-cf4621-29889660-xbjgx\" (UID: \"94917f70-a899-4b27-86a3-7e420b6856ae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cf4621-29889660-xbjgx" Oct 30 17:00:00.725479 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.725401 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xp28\" (UniqueName: \"kubernetes.io/projected/8757e465-e3fa-42aa-8911-67323ce3cc2d-kube-api-access-2xp28\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-3ca339-29889660-pbd75\" (UID: \"8757e465-e3fa-42aa-8911-67323ce3cc2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3ca339-29889660-pbd75" Oct 30 17:00:00.732190 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.727610 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-deb721-29889660-r4c5w" Oct 30 17:00:00.732190 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.730815 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mw9jc\" (UniqueName: \"kubernetes.io/projected/debf5df6-8130-4b62-a23f-74a234816389-kube-api-access-mw9jc\") pod \"maas-api-key-cleanup-e2e-worker-w2-18519c-29889660-q5xx4\" (UID: \"debf5df6-8130-4b62-a23f-74a234816389\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889660-q5xx4" Oct 30 17:00:00.777419 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.777324 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cf4621-29889660-xbjgx" Oct 30 17:00:00.788265 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.788185 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889660-27gc7" Oct 30 17:00:00.798420 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.797985 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3ca339-29889660-pbd75" Oct 30 17:00:00.812076 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.811616 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889660-q5xx4" Oct 30 17:00:00.960042 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:00:00.959476 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30b24448_c388_4366_a826_fddc6d987015.slice/crio-9319d4b681a9229e7bd66fe1c9bc254cd8dcda8542601da20cb8641492a6bc93 WatchSource:0}: Error finding container 9319d4b681a9229e7bd66fe1c9bc254cd8dcda8542601da20cb8641492a6bc93: Status 404 returned error can't find the container with id 9319d4b681a9229e7bd66fe1c9bc254cd8dcda8542601da20cb8641492a6bc93 Oct 30 17:00:00.978509 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.973582 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889660-2bb5p"] Oct 30 17:00:00.978509 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:00.973670 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-06e7f6-29889660-xnlfg"] Oct 30 17:00:01.006049 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:00:01.005987 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod708fd15b_ac7c_42e1_aa73_584566905947.slice/crio-061464cc30fa725219aa44658217316477e01e2df8c4393fabc7f6c2786038dc WatchSource:0}: Error finding container 061464cc30fa725219aa44658217316477e01e2df8c4393fabc7f6c2786038dc: Status 404 returned error can't find the container with id 061464cc30fa725219aa44658217316477e01e2df8c4393fabc7f6c2786038dc Oct 30 17:00:01.057864 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:01.057818 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889660-n8scx"] Oct 30 17:00:01.075114 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:01.075078 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889660-2rqc5"] Oct 30 17:00:01.084714 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:00:01.076507 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69e0b1a3_2a21_4412_9495_29669c26929b.slice/crio-c8a4b1421deede0eb76c0c51cdf01331ed55891b16b97d6c2db3c191dad23899 WatchSource:0}: Error finding container c8a4b1421deede0eb76c0c51cdf01331ed55891b16b97d6c2db3c191dad23899: Status 404 returned error can't find the container with id c8a4b1421deede0eb76c0c51cdf01331ed55891b16b97d6c2db3c191dad23899 Oct 30 17:00:01.118575 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:01.118198 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889660-bwdvj"] Oct 30 17:00:01.190468 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:01.190426 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889660-2rqc5" event={"ID":"69e0b1a3-2a21-4412-9495-29669c26929b","Type":"ContainerStarted","Data":"c8a4b1421deede0eb76c0c51cdf01331ed55891b16b97d6c2db3c191dad23899"} Oct 30 17:00:01.192844 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:01.192814 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889660-n8scx" event={"ID":"3576003e-ed26-4e09-aef4-1f9bddc69a5d","Type":"ContainerStarted","Data":"723c19e5f605fe9584308e5434faf39345a8236feab70ed5b74a23ce0af6c295"} Oct 30 17:00:01.195683 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:01.195660 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-06e7f6-29889660-xnlfg" event={"ID":"708fd15b-ac7c-42e1-aa73-584566905947","Type":"ContainerStarted","Data":"061464cc30fa725219aa44658217316477e01e2df8c4393fabc7f6c2786038dc"} Oct 30 17:00:01.196985 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:01.196957 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889660-2bb5p" event={"ID":"30b24448-c388-4366-a826-fddc6d987015","Type":"ContainerStarted","Data":"9319d4b681a9229e7bd66fe1c9bc254cd8dcda8542601da20cb8641492a6bc93"} Oct 30 17:00:01.281900 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:01.281868 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3ca339-29889660-pbd75"] Oct 30 17:00:01.283522 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:00:01.283497 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8757e465_e3fa_42aa_8911_67323ce3cc2d.slice/crio-225f6a0fba5b8ffb5c78f90b2bd0234a21dea0463626c4ad3f5568a34a7994ea WatchSource:0}: Error finding container 225f6a0fba5b8ffb5c78f90b2bd0234a21dea0463626c4ad3f5568a34a7994ea: Status 404 returned error can't find the container with id 225f6a0fba5b8ffb5c78f90b2bd0234a21dea0463626c4ad3f5568a34a7994ea Oct 30 17:00:01.324918 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:01.323819 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-deb721-29889660-r4c5w"] Oct 30 17:00:01.328219 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:00:01.328165 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod305e6ca0_4f0e_42e8_ae53_1e495e07a1a3.slice/crio-2229d1c2e189700f95371353920f53e2a23b15cec004a5547721872fe3ba5fcb WatchSource:0}: Error finding container 2229d1c2e189700f95371353920f53e2a23b15cec004a5547721872fe3ba5fcb: Status 404 returned error can't find the container with id 2229d1c2e189700f95371353920f53e2a23b15cec004a5547721872fe3ba5fcb Oct 30 17:00:01.435345 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:01.432229 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889660-q5xx4"] Oct 30 17:00:01.436484 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:01.436459 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cf4621-29889660-xbjgx"] Oct 30 17:00:01.437998 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:00:01.437967 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddebf5df6_8130_4b62_a23f_74a234816389.slice/crio-e966fa75018290b2a314593f77ed27650e2ef450b933123cc62655b4c3b0d942 WatchSource:0}: Error finding container e966fa75018290b2a314593f77ed27650e2ef450b933123cc62655b4c3b0d942: Status 404 returned error can't find the container with id e966fa75018290b2a314593f77ed27650e2ef450b933123cc62655b4c3b0d942 Oct 30 17:00:01.439006 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:00:01.438968 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94917f70_a899_4b27_86a3_7e420b6856ae.slice/crio-d1c68e868cb326a7a66e25b0d8baffc684ae46bb1483ff5cb00a550251ee4465 WatchSource:0}: Error finding container d1c68e868cb326a7a66e25b0d8baffc684ae46bb1483ff5cb00a550251ee4465: Status 404 returned error can't find the container with id d1c68e868cb326a7a66e25b0d8baffc684ae46bb1483ff5cb00a550251ee4465 Oct 30 17:00:01.447497 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:01.447472 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889660-27gc7"] Oct 30 17:00:01.451873 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:00:01.451846 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32e4a7ff_2f2e_4140_a540_b59282f6a80f.slice/crio-6751069dc69697435c9396c5b4ceba26d1af42926e21981530fcedc76bc53dce WatchSource:0}: Error finding container 6751069dc69697435c9396c5b4ceba26d1af42926e21981530fcedc76bc53dce: Status 404 returned error can't find the container with id 6751069dc69697435c9396c5b4ceba26d1af42926e21981530fcedc76bc53dce Oct 30 17:00:02.224289 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:02.224257 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889660-q5xx4" event={"ID":"debf5df6-8130-4b62-a23f-74a234816389","Type":"ContainerStarted","Data":"fd75105e168ee70cf192f1586682852f98731b2f23a2f3e8e64ad05da2fefe43"} Oct 30 17:00:02.224460 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:02.224309 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889660-q5xx4" event={"ID":"debf5df6-8130-4b62-a23f-74a234816389","Type":"ContainerStarted","Data":"e966fa75018290b2a314593f77ed27650e2ef450b933123cc62655b4c3b0d942"} Oct 30 17:00:02.226445 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:02.226410 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889660-n8scx" event={"ID":"3576003e-ed26-4e09-aef4-1f9bddc69a5d","Type":"ContainerStarted","Data":"800ccbad3f3c052b4cf45a51925c79030746dbb860989a6578de952edf890aff"} Oct 30 17:00:02.228774 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:02.228704 4493 generic.go:356] "Generic (PLEG): container finished" podID="708fd15b-ac7c-42e1-aa73-584566905947" containerID="cfc61a0c4721dcba238e8340c8ffa5e2e75b73f24f547c3cb42f14d3837aa46f" exitCode=60 Oct 30 17:00:02.229221 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:02.229161 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-06e7f6-29889660-xnlfg" event={"ID":"708fd15b-ac7c-42e1-aa73-584566905947","Type":"ContainerDied","Data":"cfc61a0c4721dcba238e8340c8ffa5e2e75b73f24f547c3cb42f14d3837aa46f"} Oct 30 17:00:02.229336 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:02.229297 4493 scope.go:122] "RemoveContainer" containerID="cfc61a0c4721dcba238e8340c8ffa5e2e75b73f24f547c3cb42f14d3837aa46f" Oct 30 17:00:02.231677 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:02.231644 4493 generic.go:356] "Generic (PLEG): container finished" podID="30b24448-c388-4366-a826-fddc6d987015" containerID="27cd65a9da05aaa7e3b584c8ea0e46430beeeae138225e76434d81fa3eed6fca" exitCode=60 Oct 30 17:00:02.231777 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:02.231727 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889660-2bb5p" event={"ID":"30b24448-c388-4366-a826-fddc6d987015","Type":"ContainerDied","Data":"27cd65a9da05aaa7e3b584c8ea0e46430beeeae138225e76434d81fa3eed6fca"} Oct 30 17:00:02.231981 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:02.231964 4493 scope.go:122] "RemoveContainer" containerID="27cd65a9da05aaa7e3b584c8ea0e46430beeeae138225e76434d81fa3eed6fca" Oct 30 17:00:02.242502 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:02.242467 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889660-27gc7" event={"ID":"32e4a7ff-2f2e-4140-a540-b59282f6a80f","Type":"ContainerStarted","Data":"91ee3c21cae4e9fbb17c2bffd1628366e794e23d91555be5ff8e14404f4e0e5c"} Oct 30 17:00:02.242654 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:02.242545 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889660-27gc7" event={"ID":"32e4a7ff-2f2e-4140-a540-b59282f6a80f","Type":"ContainerStarted","Data":"6751069dc69697435c9396c5b4ceba26d1af42926e21981530fcedc76bc53dce"} Oct 30 17:00:02.245740 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:02.245710 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cf4621-29889660-xbjgx" event={"ID":"94917f70-a899-4b27-86a3-7e420b6856ae","Type":"ContainerStarted","Data":"d6bfc3c474a32fe698b18e37f804da3d8f7060a38b84f471d22127014a4e7876"} Oct 30 17:00:02.246012 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:02.245991 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cf4621-29889660-xbjgx" event={"ID":"94917f70-a899-4b27-86a3-7e420b6856ae","Type":"ContainerStarted","Data":"d1c68e868cb326a7a66e25b0d8baffc684ae46bb1483ff5cb00a550251ee4465"} Oct 30 17:00:02.248812 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:02.248365 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889660-2rqc5" event={"ID":"69e0b1a3-2a21-4412-9495-29669c26929b","Type":"ContainerStarted","Data":"a07fc320e23fea9f3ec3ac2a459ab2462e01c6765e0dbc154c3a9328d3993949"} Oct 30 17:00:02.250344 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:02.250300 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889660-bwdvj" event={"ID":"d2b861e7-c17b-44a9-86c8-8102ecc2691c","Type":"ContainerStarted","Data":"e76267dba244225db98803dd5c2a42d2b912c70c5da2a9df4610a263add5de66"} Oct 30 17:00:02.250493 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:02.250366 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889660-bwdvj" event={"ID":"d2b861e7-c17b-44a9-86c8-8102ecc2691c","Type":"ContainerStarted","Data":"3840d760de1a51c118283cf50acccf2b5354bf4ee88302c37cfb8c8f3c8665b5"} Oct 30 17:00:02.254369 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:02.254336 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3ca339-29889660-pbd75" event={"ID":"8757e465-e3fa-42aa-8911-67323ce3cc2d","Type":"ContainerStarted","Data":"724251cb6f48661fee136e28ac34670b0d00c894001e44dd70bfcba5e9646e90"} Oct 30 17:00:02.254492 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:02.254390 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3ca339-29889660-pbd75" event={"ID":"8757e465-e3fa-42aa-8911-67323ce3cc2d","Type":"ContainerStarted","Data":"225f6a0fba5b8ffb5c78f90b2bd0234a21dea0463626c4ad3f5568a34a7994ea"} Oct 30 17:00:02.264146 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:02.264012 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-deb721-29889660-r4c5w" event={"ID":"305e6ca0-4f0e-42e8-ae53-1e495e07a1a3","Type":"ContainerStarted","Data":"68df40959cf9d7f2e1b670063a25d5d17ef7ed9a8acf20bc3842abadb29574b3"} Oct 30 17:00:02.264146 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:02.264082 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-deb721-29889660-r4c5w" event={"ID":"305e6ca0-4f0e-42e8-ae53-1e495e07a1a3","Type":"ContainerStarted","Data":"2229d1c2e189700f95371353920f53e2a23b15cec004a5547721872fe3ba5fcb"} Oct 30 17:00:02.269722 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:02.269664 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889660-q5xx4" podStartSLOduration=2.269646357 podStartE2EDuration="2.269646357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:00:02.268153413 +0000 UTC m=+2229.495328515" watchObservedRunningTime="2026-10-30 17:00:02.269646357 +0000 UTC m=+2229.496821459" Oct 30 17:00:02.422691 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:02.422018 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889660-bwdvj" podStartSLOduration=2.421999544 podStartE2EDuration="2.421999544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:00:02.421254823 +0000 UTC m=+2229.648429930" watchObservedRunningTime="2026-10-30 17:00:02.421999544 +0000 UTC m=+2229.649174645" Oct 30 17:00:02.463614 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:02.463559 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3ca339-29889660-pbd75" podStartSLOduration=2.463543243 podStartE2EDuration="2.463543243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:00:02.461454432 +0000 UTC m=+2229.688629532" watchObservedRunningTime="2026-10-30 17:00:02.463543243 +0000 UTC m=+2229.690718385" Oct 30 17:00:02.510694 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:02.510624 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cf4621-29889660-xbjgx" podStartSLOduration=2.5105990719999998 podStartE2EDuration="2.510599072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:00:02.500185097 +0000 UTC m=+2229.727360196" watchObservedRunningTime="2026-10-30 17:00:02.510599072 +0000 UTC m=+2229.737774172" Oct 30 17:00:02.551173 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:02.551052 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889660-27gc7" podStartSLOduration=2.5510331969999998 podStartE2EDuration="2.551033197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:00:02.54498612 +0000 UTC m=+2229.772161220" watchObservedRunningTime="2026-10-30 17:00:02.551033197 +0000 UTC m=+2229.778208298" Oct 30 17:00:02.588724 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:02.588667 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889660-2rqc5" podStartSLOduration=2.5886543250000003 podStartE2EDuration="2.588654325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:00:02.586768148 +0000 UTC m=+2229.813943239" watchObservedRunningTime="2026-10-30 17:00:02.588654325 +0000 UTC m=+2229.815829426" Oct 30 17:00:02.629410 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:02.629348 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-deb721-29889660-r4c5w" podStartSLOduration=2.62933203 podStartE2EDuration="2.62933203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:00:02.625972032 +0000 UTC m=+2229.853147142" watchObservedRunningTime="2026-10-30 17:00:02.62933203 +0000 UTC m=+2229.856507131" Oct 30 17:00:02.671302 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:02.671251 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889660-n8scx" podStartSLOduration=2.671235952 podStartE2EDuration="2.671235952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:00:02.670767889 +0000 UTC m=+2229.897942990" watchObservedRunningTime="2026-10-30 17:00:02.671235952 +0000 UTC m=+2229.898411210" Oct 30 17:00:03.273412 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:03.273373 4493 generic.go:356] "Generic (PLEG): container finished" podID="32e4a7ff-2f2e-4140-a540-b59282f6a80f" containerID="91ee3c21cae4e9fbb17c2bffd1628366e794e23d91555be5ff8e14404f4e0e5c" exitCode=60 Oct 30 17:00:03.273623 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:03.273456 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889660-27gc7" event={"ID":"32e4a7ff-2f2e-4140-a540-b59282f6a80f","Type":"ContainerDied","Data":"91ee3c21cae4e9fbb17c2bffd1628366e794e23d91555be5ff8e14404f4e0e5c"} Oct 30 17:00:03.273767 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:03.273746 4493 scope.go:122] "RemoveContainer" containerID="91ee3c21cae4e9fbb17c2bffd1628366e794e23d91555be5ff8e14404f4e0e5c" Oct 30 17:00:03.275434 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:03.275410 4493 generic.go:356] "Generic (PLEG): container finished" podID="94917f70-a899-4b27-86a3-7e420b6856ae" containerID="d6bfc3c474a32fe698b18e37f804da3d8f7060a38b84f471d22127014a4e7876" exitCode=60 Oct 30 17:00:03.275569 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:03.275505 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cf4621-29889660-xbjgx" event={"ID":"94917f70-a899-4b27-86a3-7e420b6856ae","Type":"ContainerDied","Data":"d6bfc3c474a32fe698b18e37f804da3d8f7060a38b84f471d22127014a4e7876"} Oct 30 17:00:03.275732 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:03.275712 4493 scope.go:122] "RemoveContainer" containerID="d6bfc3c474a32fe698b18e37f804da3d8f7060a38b84f471d22127014a4e7876" Oct 30 17:00:03.277529 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:03.277372 4493 generic.go:356] "Generic (PLEG): container finished" podID="69e0b1a3-2a21-4412-9495-29669c26929b" containerID="a07fc320e23fea9f3ec3ac2a459ab2462e01c6765e0dbc154c3a9328d3993949" exitCode=60 Oct 30 17:00:03.277529 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:03.277448 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889660-2rqc5" event={"ID":"69e0b1a3-2a21-4412-9495-29669c26929b","Type":"ContainerDied","Data":"a07fc320e23fea9f3ec3ac2a459ab2462e01c6765e0dbc154c3a9328d3993949"} Oct 30 17:00:03.277689 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:03.277674 4493 scope.go:122] "RemoveContainer" containerID="a07fc320e23fea9f3ec3ac2a459ab2462e01c6765e0dbc154c3a9328d3993949" Oct 30 17:00:03.279417 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:03.279398 4493 generic.go:356] "Generic (PLEG): container finished" podID="d2b861e7-c17b-44a9-86c8-8102ecc2691c" containerID="e76267dba244225db98803dd5c2a42d2b912c70c5da2a9df4610a263add5de66" exitCode=60 Oct 30 17:00:03.279524 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:03.279445 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889660-bwdvj" event={"ID":"d2b861e7-c17b-44a9-86c8-8102ecc2691c","Type":"ContainerDied","Data":"e76267dba244225db98803dd5c2a42d2b912c70c5da2a9df4610a263add5de66"} Oct 30 17:00:03.279668 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:03.279652 4493 scope.go:122] "RemoveContainer" containerID="e76267dba244225db98803dd5c2a42d2b912c70c5da2a9df4610a263add5de66" Oct 30 17:00:03.281336 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:03.281259 4493 generic.go:356] "Generic (PLEG): container finished" podID="8757e465-e3fa-42aa-8911-67323ce3cc2d" containerID="724251cb6f48661fee136e28ac34670b0d00c894001e44dd70bfcba5e9646e90" exitCode=60 Oct 30 17:00:03.281465 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:03.281343 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3ca339-29889660-pbd75" event={"ID":"8757e465-e3fa-42aa-8911-67323ce3cc2d","Type":"ContainerDied","Data":"724251cb6f48661fee136e28ac34670b0d00c894001e44dd70bfcba5e9646e90"} Oct 30 17:00:03.281768 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:03.281751 4493 scope.go:122] "RemoveContainer" containerID="724251cb6f48661fee136e28ac34670b0d00c894001e44dd70bfcba5e9646e90" Oct 30 17:00:03.283072 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:03.283051 4493 generic.go:356] "Generic (PLEG): container finished" podID="305e6ca0-4f0e-42e8-ae53-1e495e07a1a3" containerID="68df40959cf9d7f2e1b670063a25d5d17ef7ed9a8acf20bc3842abadb29574b3" exitCode=60 Oct 30 17:00:03.283151 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:03.283131 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-deb721-29889660-r4c5w" event={"ID":"305e6ca0-4f0e-42e8-ae53-1e495e07a1a3","Type":"ContainerDied","Data":"68df40959cf9d7f2e1b670063a25d5d17ef7ed9a8acf20bc3842abadb29574b3"} Oct 30 17:00:03.283374 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:03.283357 4493 scope.go:122] "RemoveContainer" containerID="68df40959cf9d7f2e1b670063a25d5d17ef7ed9a8acf20bc3842abadb29574b3" Oct 30 17:00:03.284829 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:03.284808 4493 generic.go:356] "Generic (PLEG): container finished" podID="debf5df6-8130-4b62-a23f-74a234816389" containerID="fd75105e168ee70cf192f1586682852f98731b2f23a2f3e8e64ad05da2fefe43" exitCode=60 Oct 30 17:00:03.284897 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:03.284838 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889660-q5xx4" event={"ID":"debf5df6-8130-4b62-a23f-74a234816389","Type":"ContainerDied","Data":"fd75105e168ee70cf192f1586682852f98731b2f23a2f3e8e64ad05da2fefe43"} Oct 30 17:00:03.285075 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:03.285059 4493 scope.go:122] "RemoveContainer" containerID="fd75105e168ee70cf192f1586682852f98731b2f23a2f3e8e64ad05da2fefe43" Oct 30 17:00:03.287223 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:03.287176 4493 generic.go:356] "Generic (PLEG): container finished" podID="3576003e-ed26-4e09-aef4-1f9bddc69a5d" containerID="800ccbad3f3c052b4cf45a51925c79030746dbb860989a6578de952edf890aff" exitCode=60 Oct 30 17:00:03.287326 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:03.287254 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889660-n8scx" event={"ID":"3576003e-ed26-4e09-aef4-1f9bddc69a5d","Type":"ContainerDied","Data":"800ccbad3f3c052b4cf45a51925c79030746dbb860989a6578de952edf890aff"} Oct 30 17:00:03.287439 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:03.287423 4493 scope.go:122] "RemoveContainer" containerID="800ccbad3f3c052b4cf45a51925c79030746dbb860989a6578de952edf890aff" Oct 30 17:00:03.289337 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:03.289316 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-06e7f6-29889660-xnlfg" event={"ID":"708fd15b-ac7c-42e1-aa73-584566905947","Type":"ContainerStarted","Data":"17f8574475f7a6f6161c2e4242a25baad23a81e118ec5a36c6edce07c812227f"} Oct 30 17:00:03.291208 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:03.291186 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889660-2bb5p" event={"ID":"30b24448-c388-4366-a826-fddc6d987015","Type":"ContainerStarted","Data":"d6afd5d765ed4eddb74f6ddcf03bf92feb9cc97e5cc86ad086431573231e6798"} Oct 30 17:00:03.567012 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:03.566949 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889660-2bb5p" podStartSLOduration=3.56693592 podStartE2EDuration="3.56693592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:00:03.564484194 +0000 UTC m=+2230.791659293" watchObservedRunningTime="2026-10-30 17:00:03.56693592 +0000 UTC m=+2230.794111078" Oct 30 17:00:03.609693 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:03.609129 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-06e7f6-29889660-xnlfg" podStartSLOduration=3.6091088129999997 podStartE2EDuration="3.609108813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:00:03.603655601 +0000 UTC m=+2230.830830702" watchObservedRunningTime="2026-10-30 17:00:03.609108813 +0000 UTC m=+2230.836283914" Oct 30 17:00:04.332488 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:04.332450 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889660-27gc7" event={"ID":"32e4a7ff-2f2e-4140-a540-b59282f6a80f","Type":"ContainerStarted","Data":"fc9b6a244e2e49368f573f8ca7ad379e0763bf487d931e9f2065bea0930debeb"} Oct 30 17:00:04.342815 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:04.342582 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cf4621-29889660-xbjgx" event={"ID":"94917f70-a899-4b27-86a3-7e420b6856ae","Type":"ContainerStarted","Data":"a566c457f3f5307cfb8f9a252c391174d16572f0e48f0374a787135c5aa8068b"} Oct 30 17:00:04.351828 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:04.351768 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889660-2rqc5" event={"ID":"69e0b1a3-2a21-4412-9495-29669c26929b","Type":"ContainerStarted","Data":"c7ce11dcfd8d29ae18185ed085bc7d9e703f6ee6aad3fb9c2787019d066c904b"} Oct 30 17:00:04.363763 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:04.363073 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889660-bwdvj" event={"ID":"d2b861e7-c17b-44a9-86c8-8102ecc2691c","Type":"ContainerStarted","Data":"c007fa5ebe561c7087a653197706cbd4ac0f5a6926f430415f310890722ca7da"} Oct 30 17:00:04.368843 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:04.368721 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3ca339-29889660-pbd75" event={"ID":"8757e465-e3fa-42aa-8911-67323ce3cc2d","Type":"ContainerStarted","Data":"8167b2ce89adceb8ff7d9e8fc60cffc1c9313f072313ca2e194b325cfcc36301"} Oct 30 17:00:04.373497 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:04.373463 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-deb721-29889660-r4c5w" event={"ID":"305e6ca0-4f0e-42e8-ae53-1e495e07a1a3","Type":"ContainerStarted","Data":"6786e4b80014d402726df19ac276311949615f66eaffdb0c4c3eb599fdc8c303"} Oct 30 17:00:04.379162 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:04.379133 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889660-q5xx4" event={"ID":"debf5df6-8130-4b62-a23f-74a234816389","Type":"ContainerStarted","Data":"bd95e3bf9b31f2a1f8c18846cd593a21771d0a7b9fd8c7d2b5b0b4b9668a6af6"} Oct 30 17:00:04.385043 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:04.385013 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889660-n8scx" event={"ID":"3576003e-ed26-4e09-aef4-1f9bddc69a5d","Type":"ContainerStarted","Data":"878794b8f05ef724134d0f6fcc528cb4e5fb8502b73c05c77823ee57e6f0671a"} Oct 30 17:00:04.388952 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:04.388858 4493 generic.go:356] "Generic (PLEG): container finished" podID="708fd15b-ac7c-42e1-aa73-584566905947" containerID="17f8574475f7a6f6161c2e4242a25baad23a81e118ec5a36c6edce07c812227f" exitCode=60 Oct 30 17:00:04.392055 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:04.391262 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-06e7f6-29889660-xnlfg" event={"ID":"708fd15b-ac7c-42e1-aa73-584566905947","Type":"ContainerDied","Data":"17f8574475f7a6f6161c2e4242a25baad23a81e118ec5a36c6edce07c812227f"} Oct 30 17:00:04.392055 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:04.391330 4493 scope.go:122] "RemoveContainer" containerID="cfc61a0c4721dcba238e8340c8ffa5e2e75b73f24f547c3cb42f14d3837aa46f" Oct 30 17:00:04.392055 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:04.391862 4493 scope.go:122] "RemoveContainer" containerID="17f8574475f7a6f6161c2e4242a25baad23a81e118ec5a36c6edce07c812227f" Oct 30 17:00:04.392611 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:00:04.392565 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-06e7f6-29889660-xnlfg_odh-ai-gateway-infra(708fd15b-ac7c-42e1-aa73-584566905947)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-06e7f6-29889660-xnlfg" podUID="708fd15b-ac7c-42e1-aa73-584566905947" Oct 30 17:00:04.403101 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:04.403070 4493 generic.go:356] "Generic (PLEG): container finished" podID="30b24448-c388-4366-a826-fddc6d987015" containerID="d6afd5d765ed4eddb74f6ddcf03bf92feb9cc97e5cc86ad086431573231e6798" exitCode=60 Oct 30 17:00:04.403340 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:04.403319 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889660-2bb5p" event={"ID":"30b24448-c388-4366-a826-fddc6d987015","Type":"ContainerDied","Data":"d6afd5d765ed4eddb74f6ddcf03bf92feb9cc97e5cc86ad086431573231e6798"} Oct 30 17:00:04.404819 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:04.403869 4493 scope.go:122] "RemoveContainer" containerID="d6afd5d765ed4eddb74f6ddcf03bf92feb9cc97e5cc86ad086431573231e6798" Oct 30 17:00:04.405843 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:00:04.405347 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-420ff3-29889660-2bb5p_odh-ai-gateway-infra(30b24448-c388-4366-a826-fddc6d987015)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889660-2bb5p" podUID="30b24448-c388-4366-a826-fddc6d987015" Oct 30 17:00:04.439488 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:04.439463 4493 scope.go:122] "RemoveContainer" containerID="27cd65a9da05aaa7e3b584c8ea0e46430beeeae138225e76434d81fa3eed6fca" Oct 30 17:00:05.417181 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:05.417144 4493 generic.go:356] "Generic (PLEG): container finished" podID="8757e465-e3fa-42aa-8911-67323ce3cc2d" containerID="8167b2ce89adceb8ff7d9e8fc60cffc1c9313f072313ca2e194b325cfcc36301" exitCode=60 Oct 30 17:00:05.417706 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:05.417230 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3ca339-29889660-pbd75" event={"ID":"8757e465-e3fa-42aa-8911-67323ce3cc2d","Type":"ContainerDied","Data":"8167b2ce89adceb8ff7d9e8fc60cffc1c9313f072313ca2e194b325cfcc36301"} Oct 30 17:00:05.417706 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:05.417304 4493 scope.go:122] "RemoveContainer" containerID="724251cb6f48661fee136e28ac34670b0d00c894001e44dd70bfcba5e9646e90" Oct 30 17:00:05.417706 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:05.417550 4493 scope.go:122] "RemoveContainer" containerID="8167b2ce89adceb8ff7d9e8fc60cffc1c9313f072313ca2e194b325cfcc36301" Oct 30 17:00:05.417948 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:00:05.417922 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-3ca339-29889660-pbd75_odh-ai-gateway-infra(8757e465-e3fa-42aa-8911-67323ce3cc2d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3ca339-29889660-pbd75" podUID="8757e465-e3fa-42aa-8911-67323ce3cc2d" Oct 30 17:00:05.421279 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:05.420733 4493 generic.go:356] "Generic (PLEG): container finished" podID="305e6ca0-4f0e-42e8-ae53-1e495e07a1a3" containerID="6786e4b80014d402726df19ac276311949615f66eaffdb0c4c3eb599fdc8c303" exitCode=60 Oct 30 17:00:05.421279 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:05.420912 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-deb721-29889660-r4c5w" event={"ID":"305e6ca0-4f0e-42e8-ae53-1e495e07a1a3","Type":"ContainerDied","Data":"6786e4b80014d402726df19ac276311949615f66eaffdb0c4c3eb599fdc8c303"} Oct 30 17:00:05.421477 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:05.421414 4493 scope.go:122] "RemoveContainer" containerID="6786e4b80014d402726df19ac276311949615f66eaffdb0c4c3eb599fdc8c303" Oct 30 17:00:05.421751 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:00:05.421720 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-deb721-29889660-r4c5w_odh-ai-gateway-infra(305e6ca0-4f0e-42e8-ae53-1e495e07a1a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-deb721-29889660-r4c5w" podUID="305e6ca0-4f0e-42e8-ae53-1e495e07a1a3" Oct 30 17:00:05.424690 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:05.424666 4493 generic.go:356] "Generic (PLEG): container finished" podID="debf5df6-8130-4b62-a23f-74a234816389" containerID="bd95e3bf9b31f2a1f8c18846cd593a21771d0a7b9fd8c7d2b5b0b4b9668a6af6" exitCode=60 Oct 30 17:00:05.425237 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:05.424707 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889660-q5xx4" event={"ID":"debf5df6-8130-4b62-a23f-74a234816389","Type":"ContainerDied","Data":"bd95e3bf9b31f2a1f8c18846cd593a21771d0a7b9fd8c7d2b5b0b4b9668a6af6"} Oct 30 17:00:05.425237 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:05.425019 4493 scope.go:122] "RemoveContainer" containerID="bd95e3bf9b31f2a1f8c18846cd593a21771d0a7b9fd8c7d2b5b0b4b9668a6af6" Oct 30 17:00:05.425404 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:00:05.425317 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-18519c-29889660-q5xx4_odh-ai-gateway-infra(debf5df6-8130-4b62-a23f-74a234816389)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889660-q5xx4" podUID="debf5df6-8130-4b62-a23f-74a234816389" Oct 30 17:00:05.427599 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:05.427575 4493 generic.go:356] "Generic (PLEG): container finished" podID="3576003e-ed26-4e09-aef4-1f9bddc69a5d" containerID="878794b8f05ef724134d0f6fcc528cb4e5fb8502b73c05c77823ee57e6f0671a" exitCode=60 Oct 30 17:00:05.427695 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:05.427674 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889660-n8scx" event={"ID":"3576003e-ed26-4e09-aef4-1f9bddc69a5d","Type":"ContainerDied","Data":"878794b8f05ef724134d0f6fcc528cb4e5fb8502b73c05c77823ee57e6f0671a"} Oct 30 17:00:05.428740 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:05.428719 4493 scope.go:122] "RemoveContainer" containerID="878794b8f05ef724134d0f6fcc528cb4e5fb8502b73c05c77823ee57e6f0671a" Oct 30 17:00:05.429315 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:00:05.429068 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-a77b48-29889660-n8scx_odh-ai-gateway-infra(3576003e-ed26-4e09-aef4-1f9bddc69a5d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889660-n8scx" podUID="3576003e-ed26-4e09-aef4-1f9bddc69a5d" Oct 30 17:00:05.430626 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:05.430571 4493 scope.go:122] "RemoveContainer" containerID="17f8574475f7a6f6161c2e4242a25baad23a81e118ec5a36c6edce07c812227f" Oct 30 17:00:05.430919 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:00:05.430893 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-06e7f6-29889660-xnlfg_odh-ai-gateway-infra(708fd15b-ac7c-42e1-aa73-584566905947)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-06e7f6-29889660-xnlfg" podUID="708fd15b-ac7c-42e1-aa73-584566905947" Oct 30 17:00:05.432800 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:05.432745 4493 scope.go:122] "RemoveContainer" containerID="d6afd5d765ed4eddb74f6ddcf03bf92feb9cc97e5cc86ad086431573231e6798" Oct 30 17:00:05.433038 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:00:05.433013 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-420ff3-29889660-2bb5p_odh-ai-gateway-infra(30b24448-c388-4366-a826-fddc6d987015)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889660-2bb5p" podUID="30b24448-c388-4366-a826-fddc6d987015" Oct 30 17:00:05.440685 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:05.440664 4493 scope.go:122] "RemoveContainer" containerID="68df40959cf9d7f2e1b670063a25d5d17ef7ed9a8acf20bc3842abadb29574b3" Oct 30 17:00:05.442660 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:05.442637 4493 generic.go:356] "Generic (PLEG): container finished" podID="32e4a7ff-2f2e-4140-a540-b59282f6a80f" containerID="fc9b6a244e2e49368f573f8ca7ad379e0763bf487d931e9f2065bea0930debeb" exitCode=60 Oct 30 17:00:05.442753 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:05.442695 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889660-27gc7" event={"ID":"32e4a7ff-2f2e-4140-a540-b59282f6a80f","Type":"ContainerDied","Data":"fc9b6a244e2e49368f573f8ca7ad379e0763bf487d931e9f2065bea0930debeb"} Oct 30 17:00:05.443011 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:05.442995 4493 scope.go:122] "RemoveContainer" containerID="fc9b6a244e2e49368f573f8ca7ad379e0763bf487d931e9f2065bea0930debeb" Oct 30 17:00:05.443238 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:00:05.443221 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-2d365c-29889660-27gc7_odh-ai-gateway-infra(32e4a7ff-2f2e-4140-a540-b59282f6a80f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889660-27gc7" podUID="32e4a7ff-2f2e-4140-a540-b59282f6a80f" Oct 30 17:00:05.445984 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:05.445766 4493 generic.go:356] "Generic (PLEG): container finished" podID="94917f70-a899-4b27-86a3-7e420b6856ae" containerID="a566c457f3f5307cfb8f9a252c391174d16572f0e48f0374a787135c5aa8068b" exitCode=60 Oct 30 17:00:05.445984 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:05.445841 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cf4621-29889660-xbjgx" event={"ID":"94917f70-a899-4b27-86a3-7e420b6856ae","Type":"ContainerDied","Data":"a566c457f3f5307cfb8f9a252c391174d16572f0e48f0374a787135c5aa8068b"} Oct 30 17:00:05.446148 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:05.446128 4493 scope.go:122] "RemoveContainer" containerID="a566c457f3f5307cfb8f9a252c391174d16572f0e48f0374a787135c5aa8068b" Oct 30 17:00:05.446418 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:00:05.446397 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-cf4621-29889660-xbjgx_odh-ai-gateway-infra(94917f70-a899-4b27-86a3-7e420b6856ae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cf4621-29889660-xbjgx" podUID="94917f70-a899-4b27-86a3-7e420b6856ae" Oct 30 17:00:05.448657 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:05.448634 4493 generic.go:356] "Generic (PLEG): container finished" podID="69e0b1a3-2a21-4412-9495-29669c26929b" containerID="c7ce11dcfd8d29ae18185ed085bc7d9e703f6ee6aad3fb9c2787019d066c904b" exitCode=60 Oct 30 17:00:05.448750 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:05.448713 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889660-2rqc5" event={"ID":"69e0b1a3-2a21-4412-9495-29669c26929b","Type":"ContainerDied","Data":"c7ce11dcfd8d29ae18185ed085bc7d9e703f6ee6aad3fb9c2787019d066c904b"} Oct 30 17:00:05.449101 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:05.449081 4493 scope.go:122] "RemoveContainer" containerID="c7ce11dcfd8d29ae18185ed085bc7d9e703f6ee6aad3fb9c2787019d066c904b" Oct 30 17:00:05.449357 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:00:05.449334 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f3503b-29889660-2rqc5_odh-ai-gateway-infra(69e0b1a3-2a21-4412-9495-29669c26929b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889660-2rqc5" podUID="69e0b1a3-2a21-4412-9495-29669c26929b" Oct 30 17:00:05.458427 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:05.454354 4493 generic.go:356] "Generic (PLEG): container finished" podID="d2b861e7-c17b-44a9-86c8-8102ecc2691c" containerID="c007fa5ebe561c7087a653197706cbd4ac0f5a6926f430415f310890722ca7da" exitCode=60 Oct 30 17:00:05.458427 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:05.454396 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889660-bwdvj" event={"ID":"d2b861e7-c17b-44a9-86c8-8102ecc2691c","Type":"ContainerDied","Data":"c007fa5ebe561c7087a653197706cbd4ac0f5a6926f430415f310890722ca7da"} Oct 30 17:00:05.458427 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:05.454698 4493 scope.go:122] "RemoveContainer" containerID="c007fa5ebe561c7087a653197706cbd4ac0f5a6926f430415f310890722ca7da" Oct 30 17:00:05.458427 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:00:05.454993 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889660-bwdvj_odh-ai-gateway-infra(d2b861e7-c17b-44a9-86c8-8102ecc2691c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889660-bwdvj" podUID="d2b861e7-c17b-44a9-86c8-8102ecc2691c" Oct 30 17:00:05.489850 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:05.489822 4493 scope.go:122] "RemoveContainer" containerID="fd75105e168ee70cf192f1586682852f98731b2f23a2f3e8e64ad05da2fefe43" Oct 30 17:00:05.511103 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:05.510962 4493 scope.go:122] "RemoveContainer" containerID="800ccbad3f3c052b4cf45a51925c79030746dbb860989a6578de952edf890aff" Oct 30 17:00:05.530624 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:05.530598 4493 scope.go:122] "RemoveContainer" containerID="91ee3c21cae4e9fbb17c2bffd1628366e794e23d91555be5ff8e14404f4e0e5c" Oct 30 17:00:05.547728 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:05.547705 4493 scope.go:122] "RemoveContainer" containerID="d6bfc3c474a32fe698b18e37f804da3d8f7060a38b84f471d22127014a4e7876" Oct 30 17:00:05.561547 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:05.561528 4493 scope.go:122] "RemoveContainer" containerID="a07fc320e23fea9f3ec3ac2a459ab2462e01c6765e0dbc154c3a9328d3993949" Oct 30 17:00:05.575457 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:05.575436 4493 scope.go:122] "RemoveContainer" containerID="e76267dba244225db98803dd5c2a42d2b912c70c5da2a9df4610a263add5de66" Oct 30 17:00:06.464976 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:06.464945 4493 scope.go:122] "RemoveContainer" containerID="bd95e3bf9b31f2a1f8c18846cd593a21771d0a7b9fd8c7d2b5b0b4b9668a6af6" Oct 30 17:00:06.465512 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:00:06.465304 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-18519c-29889660-q5xx4_odh-ai-gateway-infra(debf5df6-8130-4b62-a23f-74a234816389)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889660-q5xx4" podUID="debf5df6-8130-4b62-a23f-74a234816389" Oct 30 17:00:06.466968 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:06.466945 4493 scope.go:122] "RemoveContainer" containerID="878794b8f05ef724134d0f6fcc528cb4e5fb8502b73c05c77823ee57e6f0671a" Oct 30 17:00:06.467299 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:00:06.467275 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-a77b48-29889660-n8scx_odh-ai-gateway-infra(3576003e-ed26-4e09-aef4-1f9bddc69a5d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889660-n8scx" podUID="3576003e-ed26-4e09-aef4-1f9bddc69a5d" Oct 30 17:00:06.469017 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:06.468999 4493 scope.go:122] "RemoveContainer" containerID="fc9b6a244e2e49368f573f8ca7ad379e0763bf487d931e9f2065bea0930debeb" Oct 30 17:00:06.469266 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:00:06.469246 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-2d365c-29889660-27gc7_odh-ai-gateway-infra(32e4a7ff-2f2e-4140-a540-b59282f6a80f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889660-27gc7" podUID="32e4a7ff-2f2e-4140-a540-b59282f6a80f" Oct 30 17:00:06.471022 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:06.471006 4493 scope.go:122] "RemoveContainer" containerID="a566c457f3f5307cfb8f9a252c391174d16572f0e48f0374a787135c5aa8068b" Oct 30 17:00:06.471243 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:00:06.471225 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-cf4621-29889660-xbjgx_odh-ai-gateway-infra(94917f70-a899-4b27-86a3-7e420b6856ae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cf4621-29889660-xbjgx" podUID="94917f70-a899-4b27-86a3-7e420b6856ae" Oct 30 17:00:06.472740 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:06.472719 4493 scope.go:122] "RemoveContainer" containerID="c7ce11dcfd8d29ae18185ed085bc7d9e703f6ee6aad3fb9c2787019d066c904b" Oct 30 17:00:06.473045 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:00:06.473022 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f3503b-29889660-2rqc5_odh-ai-gateway-infra(69e0b1a3-2a21-4412-9495-29669c26929b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889660-2rqc5" podUID="69e0b1a3-2a21-4412-9495-29669c26929b" Oct 30 17:00:06.474273 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:06.474256 4493 scope.go:122] "RemoveContainer" containerID="c007fa5ebe561c7087a653197706cbd4ac0f5a6926f430415f310890722ca7da" Oct 30 17:00:06.474491 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:00:06.474471 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889660-bwdvj_odh-ai-gateway-infra(d2b861e7-c17b-44a9-86c8-8102ecc2691c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889660-bwdvj" podUID="d2b861e7-c17b-44a9-86c8-8102ecc2691c" Oct 30 17:00:06.475829 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:06.475811 4493 scope.go:122] "RemoveContainer" containerID="8167b2ce89adceb8ff7d9e8fc60cffc1c9313f072313ca2e194b325cfcc36301" Oct 30 17:00:06.476057 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:00:06.476035 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-3ca339-29889660-pbd75_odh-ai-gateway-infra(8757e465-e3fa-42aa-8911-67323ce3cc2d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3ca339-29889660-pbd75" podUID="8757e465-e3fa-42aa-8911-67323ce3cc2d" Oct 30 17:00:06.477431 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:06.477414 4493 scope.go:122] "RemoveContainer" containerID="6786e4b80014d402726df19ac276311949615f66eaffdb0c4c3eb599fdc8c303" Oct 30 17:00:06.477617 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:00:06.477599 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-deb721-29889660-r4c5w_odh-ai-gateway-infra(305e6ca0-4f0e-42e8-ae53-1e495e07a1a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-deb721-29889660-r4c5w" podUID="305e6ca0-4f0e-42e8-ae53-1e495e07a1a3" Oct 30 17:00:15.926405 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:15.926358 4493 scope.go:122] "RemoveContainer" containerID="fc9b6a244e2e49368f573f8ca7ad379e0763bf487d931e9f2065bea0930debeb" Oct 30 17:00:16.559271 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:16.559241 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889660-27gc7" event={"ID":"32e4a7ff-2f2e-4140-a540-b59282f6a80f","Type":"ContainerStarted","Data":"b774711d25f77af4a973b88c8659f387f35c01cdff93dee1da3030343bfbe1bb"} Oct 30 17:00:16.926051 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:16.926008 4493 scope.go:122] "RemoveContainer" containerID="17f8574475f7a6f6161c2e4242a25baad23a81e118ec5a36c6edce07c812227f" Oct 30 17:00:16.926235 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:16.926214 4493 scope.go:122] "RemoveContainer" containerID="c7ce11dcfd8d29ae18185ed085bc7d9e703f6ee6aad3fb9c2787019d066c904b" Oct 30 17:00:16.926371 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:16.926353 4493 scope.go:122] "RemoveContainer" containerID="6786e4b80014d402726df19ac276311949615f66eaffdb0c4c3eb599fdc8c303" Oct 30 17:00:16.926554 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:16.926534 4493 scope.go:122] "RemoveContainer" containerID="c007fa5ebe561c7087a653197706cbd4ac0f5a6926f430415f310890722ca7da" Oct 30 17:00:16.927471 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:16.927038 4493 scope.go:122] "RemoveContainer" containerID="d6afd5d765ed4eddb74f6ddcf03bf92feb9cc97e5cc86ad086431573231e6798" Oct 30 17:00:16.927579 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:16.927536 4493 scope.go:122] "RemoveContainer" containerID="a566c457f3f5307cfb8f9a252c391174d16572f0e48f0374a787135c5aa8068b" Oct 30 17:00:17.568646 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:17.568618 4493 generic.go:356] "Generic (PLEG): container finished" podID="32e4a7ff-2f2e-4140-a540-b59282f6a80f" containerID="b774711d25f77af4a973b88c8659f387f35c01cdff93dee1da3030343bfbe1bb" exitCode=60 Oct 30 17:00:17.568858 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:17.568693 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889660-27gc7" event={"ID":"32e4a7ff-2f2e-4140-a540-b59282f6a80f","Type":"ContainerDied","Data":"b774711d25f77af4a973b88c8659f387f35c01cdff93dee1da3030343bfbe1bb"} Oct 30 17:00:17.568858 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:17.568752 4493 scope.go:122] "RemoveContainer" containerID="fc9b6a244e2e49368f573f8ca7ad379e0763bf487d931e9f2065bea0930debeb" Oct 30 17:00:17.569161 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:17.569142 4493 scope.go:122] "RemoveContainer" containerID="b774711d25f77af4a973b88c8659f387f35c01cdff93dee1da3030343bfbe1bb" Oct 30 17:00:17.569480 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:00:17.569458 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-2d365c-29889660-27gc7_odh-ai-gateway-infra(32e4a7ff-2f2e-4140-a540-b59282f6a80f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889660-27gc7" podUID="32e4a7ff-2f2e-4140-a540-b59282f6a80f" Oct 30 17:00:17.570894 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:17.570873 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cf4621-29889660-xbjgx" event={"ID":"94917f70-a899-4b27-86a3-7e420b6856ae","Type":"ContainerStarted","Data":"2b4aa6e4729847908fde0f09147a24e0261f28719e982abac4402dab4227d289"} Oct 30 17:00:17.572909 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:17.572890 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889660-bwdvj" event={"ID":"d2b861e7-c17b-44a9-86c8-8102ecc2691c","Type":"ContainerStarted","Data":"009841ac2e12b1156559164414e515ddd239f6195f9e0c5edff5497324efb077"} Oct 30 17:00:17.574966 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:17.574946 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-06e7f6-29889660-xnlfg" event={"ID":"708fd15b-ac7c-42e1-aa73-584566905947","Type":"ContainerStarted","Data":"9573ee826954ef3da683843d876328ed3476a32b1f9614c7c6ed537289fb44c8"} Oct 30 17:00:17.615088 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:17.615061 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889660-27gc7"] Oct 30 17:00:17.926097 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:17.926072 4493 scope.go:122] "RemoveContainer" containerID="878794b8f05ef724134d0f6fcc528cb4e5fb8502b73c05c77823ee57e6f0671a" Oct 30 17:00:17.926272 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:17.926170 4493 scope.go:122] "RemoveContainer" containerID="bd95e3bf9b31f2a1f8c18846cd593a21771d0a7b9fd8c7d2b5b0b4b9668a6af6" Oct 30 17:00:17.926350 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:17.926336 4493 scope.go:122] "RemoveContainer" containerID="8167b2ce89adceb8ff7d9e8fc60cffc1c9313f072313ca2e194b325cfcc36301" Oct 30 17:00:18.586206 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:18.586162 4493 generic.go:356] "Generic (PLEG): container finished" podID="94917f70-a899-4b27-86a3-7e420b6856ae" containerID="2b4aa6e4729847908fde0f09147a24e0261f28719e982abac4402dab4227d289" exitCode=60 Oct 30 17:00:18.586694 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:18.586230 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cf4621-29889660-xbjgx" event={"ID":"94917f70-a899-4b27-86a3-7e420b6856ae","Type":"ContainerDied","Data":"2b4aa6e4729847908fde0f09147a24e0261f28719e982abac4402dab4227d289"} Oct 30 17:00:18.586694 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:18.586288 4493 scope.go:122] "RemoveContainer" containerID="a566c457f3f5307cfb8f9a252c391174d16572f0e48f0374a787135c5aa8068b" Oct 30 17:00:18.586694 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:18.586602 4493 scope.go:122] "RemoveContainer" containerID="2b4aa6e4729847908fde0f09147a24e0261f28719e982abac4402dab4227d289" Oct 30 17:00:18.586921 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:00:18.586896 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-cf4621-29889660-xbjgx_odh-ai-gateway-infra(94917f70-a899-4b27-86a3-7e420b6856ae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cf4621-29889660-xbjgx" podUID="94917f70-a899-4b27-86a3-7e420b6856ae" Oct 30 17:00:18.588459 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:18.588437 4493 generic.go:356] "Generic (PLEG): container finished" podID="69e0b1a3-2a21-4412-9495-29669c26929b" containerID="bc2c447bf4940b326fa42dbe70cc76e8f73c5d613a5bde08c1298e1ff387fa92" exitCode=60 Oct 30 17:00:18.588588 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:18.588508 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889660-2rqc5" event={"ID":"69e0b1a3-2a21-4412-9495-29669c26929b","Type":"ContainerDied","Data":"bc2c447bf4940b326fa42dbe70cc76e8f73c5d613a5bde08c1298e1ff387fa92"} Oct 30 17:00:18.588863 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:18.588847 4493 scope.go:122] "RemoveContainer" containerID="bc2c447bf4940b326fa42dbe70cc76e8f73c5d613a5bde08c1298e1ff387fa92" Oct 30 17:00:18.589089 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:00:18.589066 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f3503b-29889660-2rqc5_odh-ai-gateway-infra(69e0b1a3-2a21-4412-9495-29669c26929b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889660-2rqc5" podUID="69e0b1a3-2a21-4412-9495-29669c26929b" Oct 30 17:00:18.590558 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:18.590535 4493 generic.go:356] "Generic (PLEG): container finished" podID="d2b861e7-c17b-44a9-86c8-8102ecc2691c" containerID="009841ac2e12b1156559164414e515ddd239f6195f9e0c5edff5497324efb077" exitCode=60 Oct 30 17:00:18.590672 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:18.590602 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889660-bwdvj" event={"ID":"d2b861e7-c17b-44a9-86c8-8102ecc2691c","Type":"ContainerDied","Data":"009841ac2e12b1156559164414e515ddd239f6195f9e0c5edff5497324efb077"} Oct 30 17:00:18.590962 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:18.590942 4493 scope.go:122] "RemoveContainer" containerID="009841ac2e12b1156559164414e515ddd239f6195f9e0c5edff5497324efb077" Oct 30 17:00:18.591208 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:00:18.591183 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889660-bwdvj_odh-ai-gateway-infra(d2b861e7-c17b-44a9-86c8-8102ecc2691c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889660-bwdvj" podUID="d2b861e7-c17b-44a9-86c8-8102ecc2691c" Oct 30 17:00:18.592778 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:18.592744 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3ca339-29889660-pbd75" event={"ID":"8757e465-e3fa-42aa-8911-67323ce3cc2d","Type":"ContainerStarted","Data":"7367196d8f068799c39a2bbb2a89f62a33639e402629d5931662a93b00dff52a"} Oct 30 17:00:18.594832 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:18.594811 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-deb721-29889660-r4c5w" event={"ID":"305e6ca0-4f0e-42e8-ae53-1e495e07a1a3","Type":"ContainerDied","Data":"881c4f3291f339c01c011f22f1677675a327399f595820e4ffd65966a94b2117"} Oct 30 17:00:18.594915 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:18.594776 4493 generic.go:356] "Generic (PLEG): container finished" podID="305e6ca0-4f0e-42e8-ae53-1e495e07a1a3" containerID="881c4f3291f339c01c011f22f1677675a327399f595820e4ffd65966a94b2117" exitCode=60 Oct 30 17:00:18.595106 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:18.595088 4493 scope.go:122] "RemoveContainer" containerID="881c4f3291f339c01c011f22f1677675a327399f595820e4ffd65966a94b2117" Oct 30 17:00:18.595291 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:00:18.595273 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-deb721-29889660-r4c5w_odh-ai-gateway-infra(305e6ca0-4f0e-42e8-ae53-1e495e07a1a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-deb721-29889660-r4c5w" podUID="305e6ca0-4f0e-42e8-ae53-1e495e07a1a3" Oct 30 17:00:18.597232 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:18.597208 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889660-n8scx" event={"ID":"3576003e-ed26-4e09-aef4-1f9bddc69a5d","Type":"ContainerStarted","Data":"65dcd70f540cf997da6796d9e56dfe3f324278670f2a812fb2ddf1d7f7a8c463"} Oct 30 17:00:18.599840 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:18.599779 4493 generic.go:356] "Generic (PLEG): container finished" podID="708fd15b-ac7c-42e1-aa73-584566905947" containerID="9573ee826954ef3da683843d876328ed3476a32b1f9614c7c6ed537289fb44c8" exitCode=60 Oct 30 17:00:18.599934 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:18.599838 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-06e7f6-29889660-xnlfg" event={"ID":"708fd15b-ac7c-42e1-aa73-584566905947","Type":"ContainerDied","Data":"9573ee826954ef3da683843d876328ed3476a32b1f9614c7c6ed537289fb44c8"} Oct 30 17:00:18.600234 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:18.600157 4493 scope.go:122] "RemoveContainer" containerID="9573ee826954ef3da683843d876328ed3476a32b1f9614c7c6ed537289fb44c8" Oct 30 17:00:18.600582 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:00:18.600389 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-06e7f6-29889660-xnlfg_odh-ai-gateway-infra(708fd15b-ac7c-42e1-aa73-584566905947)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-06e7f6-29889660-xnlfg" podUID="708fd15b-ac7c-42e1-aa73-584566905947" Oct 30 17:00:18.602195 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:18.602159 4493 generic.go:356] "Generic (PLEG): container finished" podID="30b24448-c388-4366-a826-fddc6d987015" containerID="56c88979c930f81a68d4a9d5049db4859e9cf18055250d9472b655e628e79579" exitCode=60 Oct 30 17:00:18.602195 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:18.602187 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889660-2bb5p" event={"ID":"30b24448-c388-4366-a826-fddc6d987015","Type":"ContainerDied","Data":"56c88979c930f81a68d4a9d5049db4859e9cf18055250d9472b655e628e79579"} Oct 30 17:00:18.602464 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:18.602447 4493 scope.go:122] "RemoveContainer" containerID="56c88979c930f81a68d4a9d5049db4859e9cf18055250d9472b655e628e79579" Oct 30 17:00:18.602645 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:00:18.602628 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-420ff3-29889660-2bb5p_odh-ai-gateway-infra(30b24448-c388-4366-a826-fddc6d987015)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889660-2bb5p" podUID="30b24448-c388-4366-a826-fddc6d987015" Oct 30 17:00:18.605518 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:18.605499 4493 scope.go:122] "RemoveContainer" containerID="c7ce11dcfd8d29ae18185ed085bc7d9e703f6ee6aad3fb9c2787019d066c904b" Oct 30 17:00:18.619606 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:18.619583 4493 scope.go:122] "RemoveContainer" containerID="c007fa5ebe561c7087a653197706cbd4ac0f5a6926f430415f310890722ca7da" Oct 30 17:00:18.634944 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:18.634925 4493 scope.go:122] "RemoveContainer" containerID="6786e4b80014d402726df19ac276311949615f66eaffdb0c4c3eb599fdc8c303" Oct 30 17:00:18.650204 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:18.650180 4493 scope.go:122] "RemoveContainer" containerID="17f8574475f7a6f6161c2e4242a25baad23a81e118ec5a36c6edce07c812227f" Oct 30 17:00:18.666124 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:18.666085 4493 scope.go:122] "RemoveContainer" containerID="d6afd5d765ed4eddb74f6ddcf03bf92feb9cc97e5cc86ad086431573231e6798" Oct 30 17:00:18.678556 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:18.678523 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cf4621-29889660-xbjgx"] Oct 30 17:00:18.732981 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:18.730834 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-06e7f6-29889660-xnlfg"] Oct 30 17:00:18.765101 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:18.765047 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889660-bwdvj"] Oct 30 17:00:18.771576 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:18.771557 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889660-27gc7" Oct 30 17:00:18.889009 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:18.888980 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32e4a7ff-2f2e-4140-a540-b59282f6a80f-kube-api-access-69skt\" (UniqueName: \"kubernetes.io/projected/32e4a7ff-2f2e-4140-a540-b59282f6a80f-kube-api-access-69skt\") pod \"32e4a7ff-2f2e-4140-a540-b59282f6a80f\" (UID: \"32e4a7ff-2f2e-4140-a540-b59282f6a80f\") " Oct 30 17:00:18.889183 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:18.889077 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32e4a7ff-2f2e-4140-a540-b59282f6a80f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32e4a7ff-2f2e-4140-a540-b59282f6a80f-maas-api-service-ca\") pod \"32e4a7ff-2f2e-4140-a540-b59282f6a80f\" (UID: \"32e4a7ff-2f2e-4140-a540-b59282f6a80f\") " Oct 30 17:00:18.889406 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:18.889386 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32e4a7ff-2f2e-4140-a540-b59282f6a80f-maas-api-service-ca" pod "32e4a7ff-2f2e-4140-a540-b59282f6a80f" (UID: "32e4a7ff-2f2e-4140-a540-b59282f6a80f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:00:18.889524 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:18.889512 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32e4a7ff-2f2e-4140-a540-b59282f6a80f-maas-api-service-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:00:18.891368 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:18.891351 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32e4a7ff-2f2e-4140-a540-b59282f6a80f-kube-api-access-69skt" pod "32e4a7ff-2f2e-4140-a540-b59282f6a80f" (UID: "32e4a7ff-2f2e-4140-a540-b59282f6a80f"). InnerVolumeSpecName "kube-api-access-69skt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:18.990837 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:18.990812 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69skt\" (UniqueName: \"kubernetes.io/projected/32e4a7ff-2f2e-4140-a540-b59282f6a80f-kube-api-access-69skt\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:00:19.622469 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:19.622433 4493 generic.go:356] "Generic (PLEG): container finished" podID="8757e465-e3fa-42aa-8911-67323ce3cc2d" containerID="7367196d8f068799c39a2bbb2a89f62a33639e402629d5931662a93b00dff52a" exitCode=60 Oct 30 17:00:19.622845 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:19.622531 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3ca339-29889660-pbd75" event={"ID":"8757e465-e3fa-42aa-8911-67323ce3cc2d","Type":"ContainerDied","Data":"7367196d8f068799c39a2bbb2a89f62a33639e402629d5931662a93b00dff52a"} Oct 30 17:00:19.622845 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:19.622584 4493 scope.go:122] "RemoveContainer" containerID="8167b2ce89adceb8ff7d9e8fc60cffc1c9313f072313ca2e194b325cfcc36301" Oct 30 17:00:19.623049 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:19.623032 4493 scope.go:122] "RemoveContainer" containerID="7367196d8f068799c39a2bbb2a89f62a33639e402629d5931662a93b00dff52a" Oct 30 17:00:19.623380 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:00:19.623357 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-3ca339-29889660-pbd75_odh-ai-gateway-infra(8757e465-e3fa-42aa-8911-67323ce3cc2d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3ca339-29889660-pbd75" podUID="8757e465-e3fa-42aa-8911-67323ce3cc2d" Oct 30 17:00:19.627072 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:19.627045 4493 generic.go:356] "Generic (PLEG): container finished" podID="debf5df6-8130-4b62-a23f-74a234816389" containerID="5fad27c56efb7dc456e61ae0f61003f918513254ffb4b252436d5d8a574e07fe" exitCode=60 Oct 30 17:00:19.627182 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:19.627141 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889660-q5xx4" event={"ID":"debf5df6-8130-4b62-a23f-74a234816389","Type":"ContainerDied","Data":"5fad27c56efb7dc456e61ae0f61003f918513254ffb4b252436d5d8a574e07fe"} Oct 30 17:00:19.627676 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:19.627653 4493 scope.go:122] "RemoveContainer" containerID="5fad27c56efb7dc456e61ae0f61003f918513254ffb4b252436d5d8a574e07fe" Oct 30 17:00:19.628122 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:00:19.627993 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-18519c-29889660-q5xx4_odh-ai-gateway-infra(debf5df6-8130-4b62-a23f-74a234816389)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889660-q5xx4" podUID="debf5df6-8130-4b62-a23f-74a234816389" Oct 30 17:00:19.629653 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:19.629635 4493 generic.go:356] "Generic (PLEG): container finished" podID="3576003e-ed26-4e09-aef4-1f9bddc69a5d" containerID="65dcd70f540cf997da6796d9e56dfe3f324278670f2a812fb2ddf1d7f7a8c463" exitCode=60 Oct 30 17:00:19.629768 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:19.629724 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889660-n8scx" event={"ID":"3576003e-ed26-4e09-aef4-1f9bddc69a5d","Type":"ContainerDied","Data":"65dcd70f540cf997da6796d9e56dfe3f324278670f2a812fb2ddf1d7f7a8c463"} Oct 30 17:00:19.630239 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:19.630208 4493 scope.go:122] "RemoveContainer" containerID="65dcd70f540cf997da6796d9e56dfe3f324278670f2a812fb2ddf1d7f7a8c463" Oct 30 17:00:19.630531 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:00:19.630504 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-a77b48-29889660-n8scx_odh-ai-gateway-infra(3576003e-ed26-4e09-aef4-1f9bddc69a5d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889660-n8scx" podUID="3576003e-ed26-4e09-aef4-1f9bddc69a5d" Oct 30 17:00:19.637339 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:19.637320 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889660-27gc7" Oct 30 17:00:19.638023 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:19.637958 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889660-27gc7" event={"ID":"32e4a7ff-2f2e-4140-a540-b59282f6a80f","Type":"ContainerDied","Data":"6751069dc69697435c9396c5b4ceba26d1af42926e21981530fcedc76bc53dce"} Oct 30 17:00:19.641652 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:19.641082 4493 scope.go:122] "RemoveContainer" containerID="bd95e3bf9b31f2a1f8c18846cd593a21771d0a7b9fd8c7d2b5b0b4b9668a6af6" Oct 30 17:00:19.661535 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:19.661495 4493 scope.go:122] "RemoveContainer" containerID="878794b8f05ef724134d0f6fcc528cb4e5fb8502b73c05c77823ee57e6f0671a" Oct 30 17:00:19.698998 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:19.698536 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889660-2bb5p"] Oct 30 17:00:19.741000 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:19.735655 4493 scope.go:122] "RemoveContainer" containerID="b774711d25f77af4a973b88c8659f387f35c01cdff93dee1da3030343bfbe1bb" Oct 30 17:00:19.754344 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:19.754297 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889660-27gc7"] Oct 30 17:00:19.761913 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:19.761811 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889660-27gc7"] Oct 30 17:00:19.811421 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:19.811388 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3ca339-29889660-pbd75"] Oct 30 17:00:19.873403 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:19.873332 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889660-bwdvj" Oct 30 17:00:19.887513 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:19.887452 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889660-2rqc5"] Oct 30 17:00:19.967807 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:19.962841 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-deb721-29889660-r4c5w"] Oct 30 17:00:20.011570 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.005125 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-06e7f6-29889660-xnlfg" Oct 30 17:00:20.011570 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.005417 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889660-n8scx"] Oct 30 17:00:20.037034 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.036998 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cf4621-29889660-xbjgx" Oct 30 17:00:20.044188 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.041480 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889660-2bb5p" Oct 30 17:00:20.058109 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.058080 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30b24448-c388-4366-a826-fddc6d987015-kube-api-access-7nn5v\" (UniqueName: \"kubernetes.io/projected/30b24448-c388-4366-a826-fddc6d987015-kube-api-access-7nn5v\") pod \"30b24448-c388-4366-a826-fddc6d987015\" (UID: \"30b24448-c388-4366-a826-fddc6d987015\") " Oct 30 17:00:20.058322 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.058305 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2b861e7-c17b-44a9-86c8-8102ecc2691c-kube-api-access-vstst\" (UniqueName: \"kubernetes.io/projected/d2b861e7-c17b-44a9-86c8-8102ecc2691c-kube-api-access-vstst\") pod \"d2b861e7-c17b-44a9-86c8-8102ecc2691c\" (UID: \"d2b861e7-c17b-44a9-86c8-8102ecc2691c\") " Oct 30 17:00:20.058977 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.058957 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2b861e7-c17b-44a9-86c8-8102ecc2691c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2b861e7-c17b-44a9-86c8-8102ecc2691c-maas-api-service-ca\") pod \"d2b861e7-c17b-44a9-86c8-8102ecc2691c\" (UID: \"d2b861e7-c17b-44a9-86c8-8102ecc2691c\") " Oct 30 17:00:20.059148 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.059134 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30b24448-c388-4366-a826-fddc6d987015-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30b24448-c388-4366-a826-fddc6d987015-maas-api-service-ca\") pod \"30b24448-c388-4366-a826-fddc6d987015\" (UID: \"30b24448-c388-4366-a826-fddc6d987015\") " Oct 30 17:00:20.059333 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.059318 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/708fd15b-ac7c-42e1-aa73-584566905947-kube-api-access-js2nh\" (UniqueName: \"kubernetes.io/projected/708fd15b-ac7c-42e1-aa73-584566905947-kube-api-access-js2nh\") pod \"708fd15b-ac7c-42e1-aa73-584566905947\" (UID: \"708fd15b-ac7c-42e1-aa73-584566905947\") " Oct 30 17:00:20.059529 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.059516 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94917f70-a899-4b27-86a3-7e420b6856ae-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94917f70-a899-4b27-86a3-7e420b6856ae-maas-api-service-ca\") pod \"94917f70-a899-4b27-86a3-7e420b6856ae\" (UID: \"94917f70-a899-4b27-86a3-7e420b6856ae\") " Oct 30 17:00:20.059650 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.059638 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/708fd15b-ac7c-42e1-aa73-584566905947-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/708fd15b-ac7c-42e1-aa73-584566905947-maas-api-service-ca\") pod \"708fd15b-ac7c-42e1-aa73-584566905947\" (UID: \"708fd15b-ac7c-42e1-aa73-584566905947\") " Oct 30 17:00:20.059929 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.059896 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94917f70-a899-4b27-86a3-7e420b6856ae-kube-api-access-8phsr\" (UniqueName: \"kubernetes.io/projected/94917f70-a899-4b27-86a3-7e420b6856ae-kube-api-access-8phsr\") pod \"94917f70-a899-4b27-86a3-7e420b6856ae\" (UID: \"94917f70-a899-4b27-86a3-7e420b6856ae\") " Oct 30 17:00:20.061340 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.061311 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2b861e7-c17b-44a9-86c8-8102ecc2691c-maas-api-service-ca" pod "d2b861e7-c17b-44a9-86c8-8102ecc2691c" (UID: "d2b861e7-c17b-44a9-86c8-8102ecc2691c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:00:20.062175 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.062133 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30b24448-c388-4366-a826-fddc6d987015-maas-api-service-ca" pod "30b24448-c388-4366-a826-fddc6d987015" (UID: "30b24448-c388-4366-a826-fddc6d987015"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:00:20.066212 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.066000 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/708fd15b-ac7c-42e1-aa73-584566905947-maas-api-service-ca" pod "708fd15b-ac7c-42e1-aa73-584566905947" (UID: "708fd15b-ac7c-42e1-aa73-584566905947"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:00:20.066735 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.066492 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94917f70-a899-4b27-86a3-7e420b6856ae-maas-api-service-ca" pod "94917f70-a899-4b27-86a3-7e420b6856ae" (UID: "94917f70-a899-4b27-86a3-7e420b6856ae"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:00:20.067940 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.067914 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/708fd15b-ac7c-42e1-aa73-584566905947-kube-api-access-js2nh" pod "708fd15b-ac7c-42e1-aa73-584566905947" (UID: "708fd15b-ac7c-42e1-aa73-584566905947"). InnerVolumeSpecName "kube-api-access-js2nh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:20.074038 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.074003 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30b24448-c388-4366-a826-fddc6d987015-kube-api-access-7nn5v" pod "30b24448-c388-4366-a826-fddc6d987015" (UID: "30b24448-c388-4366-a826-fddc6d987015"). InnerVolumeSpecName "kube-api-access-7nn5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:20.075456 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.075424 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2b861e7-c17b-44a9-86c8-8102ecc2691c-kube-api-access-vstst" pod "d2b861e7-c17b-44a9-86c8-8102ecc2691c" (UID: "d2b861e7-c17b-44a9-86c8-8102ecc2691c"). InnerVolumeSpecName "kube-api-access-vstst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:20.077105 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.077072 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94917f70-a899-4b27-86a3-7e420b6856ae-kube-api-access-8phsr" pod "94917f70-a899-4b27-86a3-7e420b6856ae" (UID: "94917f70-a899-4b27-86a3-7e420b6856ae"). InnerVolumeSpecName "kube-api-access-8phsr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:20.163082 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.161711 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nn5v\" (UniqueName: \"kubernetes.io/projected/30b24448-c388-4366-a826-fddc6d987015-kube-api-access-7nn5v\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:00:20.163082 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.161743 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vstst\" (UniqueName: \"kubernetes.io/projected/d2b861e7-c17b-44a9-86c8-8102ecc2691c-kube-api-access-vstst\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:00:20.163082 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.161760 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2b861e7-c17b-44a9-86c8-8102ecc2691c-maas-api-service-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:00:20.163082 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.161774 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30b24448-c388-4366-a826-fddc6d987015-maas-api-service-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:00:20.163082 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.161808 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-js2nh\" (UniqueName: \"kubernetes.io/projected/708fd15b-ac7c-42e1-aa73-584566905947-kube-api-access-js2nh\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:00:20.163082 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.161820 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94917f70-a899-4b27-86a3-7e420b6856ae-maas-api-service-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:00:20.163082 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.161833 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/708fd15b-ac7c-42e1-aa73-584566905947-maas-api-service-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:00:20.163082 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.161846 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8phsr\" (UniqueName: \"kubernetes.io/projected/94917f70-a899-4b27-86a3-7e420b6856ae-kube-api-access-8phsr\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:00:20.379531 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.379370 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889660-2rqc5" Oct 30 17:00:20.410454 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.410420 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-deb721-29889660-r4c5w" Oct 30 17:00:20.572866 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.562590 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69e0b1a3-2a21-4412-9495-29669c26929b-kube-api-access-drc2n\" (UniqueName: \"kubernetes.io/projected/69e0b1a3-2a21-4412-9495-29669c26929b-kube-api-access-drc2n\") pod \"69e0b1a3-2a21-4412-9495-29669c26929b\" (UID: \"69e0b1a3-2a21-4412-9495-29669c26929b\") " Oct 30 17:00:20.573053 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.572988 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69e0b1a3-2a21-4412-9495-29669c26929b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69e0b1a3-2a21-4412-9495-29669c26929b-maas-api-service-ca\") pod \"69e0b1a3-2a21-4412-9495-29669c26929b\" (UID: \"69e0b1a3-2a21-4412-9495-29669c26929b\") " Oct 30 17:00:20.573429 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.573380 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69e0b1a3-2a21-4412-9495-29669c26929b-maas-api-service-ca" pod "69e0b1a3-2a21-4412-9495-29669c26929b" (UID: "69e0b1a3-2a21-4412-9495-29669c26929b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:00:20.573799 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.573758 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/305e6ca0-4f0e-42e8-ae53-1e495e07a1a3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/305e6ca0-4f0e-42e8-ae53-1e495e07a1a3-maas-api-service-ca\") pod \"305e6ca0-4f0e-42e8-ae53-1e495e07a1a3\" (UID: \"305e6ca0-4f0e-42e8-ae53-1e495e07a1a3\") " Oct 30 17:00:20.574395 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.574279 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/305e6ca0-4f0e-42e8-ae53-1e495e07a1a3-maas-api-service-ca" pod "305e6ca0-4f0e-42e8-ae53-1e495e07a1a3" (UID: "305e6ca0-4f0e-42e8-ae53-1e495e07a1a3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:00:20.574911 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.574892 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/305e6ca0-4f0e-42e8-ae53-1e495e07a1a3-kube-api-access-ccgsx\" (UniqueName: \"kubernetes.io/projected/305e6ca0-4f0e-42e8-ae53-1e495e07a1a3-kube-api-access-ccgsx\") pod \"305e6ca0-4f0e-42e8-ae53-1e495e07a1a3\" (UID: \"305e6ca0-4f0e-42e8-ae53-1e495e07a1a3\") " Oct 30 17:00:20.575587 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.575548 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/305e6ca0-4f0e-42e8-ae53-1e495e07a1a3-maas-api-service-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:00:20.575723 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.575707 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69e0b1a3-2a21-4412-9495-29669c26929b-maas-api-service-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:00:20.587065 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.587015 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/305e6ca0-4f0e-42e8-ae53-1e495e07a1a3-kube-api-access-ccgsx" pod "305e6ca0-4f0e-42e8-ae53-1e495e07a1a3" (UID: "305e6ca0-4f0e-42e8-ae53-1e495e07a1a3"). InnerVolumeSpecName "kube-api-access-ccgsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:20.587207 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.587073 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69e0b1a3-2a21-4412-9495-29669c26929b-kube-api-access-drc2n" pod "69e0b1a3-2a21-4412-9495-29669c26929b" (UID: "69e0b1a3-2a21-4412-9495-29669c26929b"). InnerVolumeSpecName "kube-api-access-drc2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:20.656912 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.656877 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-06e7f6-29889660-xnlfg" event={"ID":"708fd15b-ac7c-42e1-aa73-584566905947","Type":"ContainerDied","Data":"061464cc30fa725219aa44658217316477e01e2df8c4393fabc7f6c2786038dc"} Oct 30 17:00:20.657321 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.656944 4493 scope.go:122] "RemoveContainer" containerID="9573ee826954ef3da683843d876328ed3476a32b1f9614c7c6ed537289fb44c8" Oct 30 17:00:20.657321 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.656886 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-06e7f6-29889660-xnlfg" Oct 30 17:00:20.658927 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.658905 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889660-2bb5p" Oct 30 17:00:20.659069 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.658990 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889660-2bb5p" event={"ID":"30b24448-c388-4366-a826-fddc6d987015","Type":"ContainerDied","Data":"9319d4b681a9229e7bd66fe1c9bc254cd8dcda8542601da20cb8641492a6bc93"} Oct 30 17:00:20.667517 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.665724 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cf4621-29889660-xbjgx" Oct 30 17:00:20.667517 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.665761 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cf4621-29889660-xbjgx" event={"ID":"94917f70-a899-4b27-86a3-7e420b6856ae","Type":"ContainerDied","Data":"d1c68e868cb326a7a66e25b0d8baffc684ae46bb1483ff5cb00a550251ee4465"} Oct 30 17:00:20.667817 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.667610 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889660-2rqc5" Oct 30 17:00:20.667817 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.667612 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889660-2rqc5" event={"ID":"69e0b1a3-2a21-4412-9495-29669c26929b","Type":"ContainerDied","Data":"c8a4b1421deede0eb76c0c51cdf01331ed55891b16b97d6c2db3c191dad23899"} Oct 30 17:00:20.670033 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.669735 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889660-bwdvj" event={"ID":"d2b861e7-c17b-44a9-86c8-8102ecc2691c","Type":"ContainerDied","Data":"3840d760de1a51c118283cf50acccf2b5354bf4ee88302c37cfb8c8f3c8665b5"} Oct 30 17:00:20.670033 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.669835 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889660-bwdvj" Oct 30 17:00:20.673028 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.672938 4493 scope.go:122] "RemoveContainer" containerID="56c88979c930f81a68d4a9d5049db4859e9cf18055250d9472b655e628e79579" Oct 30 17:00:20.677013 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.676964 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccgsx\" (UniqueName: \"kubernetes.io/projected/305e6ca0-4f0e-42e8-ae53-1e495e07a1a3-kube-api-access-ccgsx\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:00:20.677147 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.677018 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drc2n\" (UniqueName: \"kubernetes.io/projected/69e0b1a3-2a21-4412-9495-29669c26929b-kube-api-access-drc2n\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:00:20.678609 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.678585 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-deb721-29889660-r4c5w" event={"ID":"305e6ca0-4f0e-42e8-ae53-1e495e07a1a3","Type":"ContainerDied","Data":"2229d1c2e189700f95371353920f53e2a23b15cec004a5547721872fe3ba5fcb"} Oct 30 17:00:20.678700 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.678674 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-deb721-29889660-r4c5w" Oct 30 17:00:20.692244 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.692223 4493 scope.go:122] "RemoveContainer" containerID="2b4aa6e4729847908fde0f09147a24e0261f28719e982abac4402dab4227d289" Oct 30 17:00:20.705189 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.705162 4493 scope.go:122] "RemoveContainer" containerID="bc2c447bf4940b326fa42dbe70cc76e8f73c5d613a5bde08c1298e1ff387fa92" Oct 30 17:00:20.716617 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.716592 4493 scope.go:122] "RemoveContainer" containerID="009841ac2e12b1156559164414e515ddd239f6195f9e0c5edff5497324efb077" Oct 30 17:00:20.731924 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.731907 4493 scope.go:122] "RemoveContainer" containerID="881c4f3291f339c01c011f22f1677675a327399f595820e4ffd65966a94b2117" Oct 30 17:00:20.734819 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.734694 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889660-2rqc5"] Oct 30 17:00:20.748583 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.743932 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889660-2rqc5"] Oct 30 17:00:20.811926 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.811891 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-deb721-29889660-r4c5w"] Oct 30 17:00:20.814310 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.814284 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-deb721-29889660-r4c5w"] Oct 30 17:00:20.851947 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.851864 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889660-2bb5p"] Oct 30 17:00:20.857898 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.857867 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-420ff3-29889660-2bb5p"] Oct 30 17:00:20.861629 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.861601 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889660-q5xx4"] Oct 30 17:00:20.885594 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.885548 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-06e7f6-29889660-xnlfg"] Oct 30 17:00:20.891692 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.891664 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-06e7f6-29889660-xnlfg"] Oct 30 17:00:20.895523 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.895502 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889660-n8scx" Oct 30 17:00:20.904760 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.904676 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3ca339-29889660-pbd75" Oct 30 17:00:20.923431 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.923403 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889660-bwdvj"] Oct 30 17:00:20.964990 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.960812 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="305e6ca0-4f0e-42e8-ae53-1e495e07a1a3" path="/var/lib/kubelet/pods/305e6ca0-4f0e-42e8-ae53-1e495e07a1a3/volumes" Oct 30 17:00:20.964990 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.961383 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30b24448-c388-4366-a826-fddc6d987015" path="/var/lib/kubelet/pods/30b24448-c388-4366-a826-fddc6d987015/volumes" Oct 30 17:00:20.964990 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.962044 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32e4a7ff-2f2e-4140-a540-b59282f6a80f" path="/var/lib/kubelet/pods/32e4a7ff-2f2e-4140-a540-b59282f6a80f/volumes" Oct 30 17:00:20.964990 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.962959 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69e0b1a3-2a21-4412-9495-29669c26929b" path="/var/lib/kubelet/pods/69e0b1a3-2a21-4412-9495-29669c26929b/volumes" Oct 30 17:00:20.964990 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.963497 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="708fd15b-ac7c-42e1-aa73-584566905947" path="/var/lib/kubelet/pods/708fd15b-ac7c-42e1-aa73-584566905947/volumes" Oct 30 17:00:20.974138 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.971248 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889660-bwdvj"] Oct 30 17:00:20.986610 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.986585 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8757e465-e3fa-42aa-8911-67323ce3cc2d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8757e465-e3fa-42aa-8911-67323ce3cc2d-maas-api-service-ca\") pod \"8757e465-e3fa-42aa-8911-67323ce3cc2d\" (UID: \"8757e465-e3fa-42aa-8911-67323ce3cc2d\") " Oct 30 17:00:20.986733 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.986642 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8757e465-e3fa-42aa-8911-67323ce3cc2d-kube-api-access-2xp28\" (UniqueName: \"kubernetes.io/projected/8757e465-e3fa-42aa-8911-67323ce3cc2d-kube-api-access-2xp28\") pod \"8757e465-e3fa-42aa-8911-67323ce3cc2d\" (UID: \"8757e465-e3fa-42aa-8911-67323ce3cc2d\") " Oct 30 17:00:20.986814 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.986743 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3576003e-ed26-4e09-aef4-1f9bddc69a5d-kube-api-access-sbc8t\" (UniqueName: \"kubernetes.io/projected/3576003e-ed26-4e09-aef4-1f9bddc69a5d-kube-api-access-sbc8t\") pod \"3576003e-ed26-4e09-aef4-1f9bddc69a5d\" (UID: \"3576003e-ed26-4e09-aef4-1f9bddc69a5d\") " Oct 30 17:00:20.986901 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.986889 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3576003e-ed26-4e09-aef4-1f9bddc69a5d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3576003e-ed26-4e09-aef4-1f9bddc69a5d-maas-api-service-ca\") pod \"3576003e-ed26-4e09-aef4-1f9bddc69a5d\" (UID: \"3576003e-ed26-4e09-aef4-1f9bddc69a5d\") " Oct 30 17:00:20.987272 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.986999 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8757e465-e3fa-42aa-8911-67323ce3cc2d-maas-api-service-ca" pod "8757e465-e3fa-42aa-8911-67323ce3cc2d" (UID: "8757e465-e3fa-42aa-8911-67323ce3cc2d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:00:20.987459 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.987391 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8757e465-e3fa-42aa-8911-67323ce3cc2d-maas-api-service-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:00:20.987621 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.987595 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3576003e-ed26-4e09-aef4-1f9bddc69a5d-maas-api-service-ca" pod "3576003e-ed26-4e09-aef4-1f9bddc69a5d" (UID: "3576003e-ed26-4e09-aef4-1f9bddc69a5d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:00:20.992458 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.992432 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3576003e-ed26-4e09-aef4-1f9bddc69a5d-kube-api-access-sbc8t" pod "3576003e-ed26-4e09-aef4-1f9bddc69a5d" (UID: "3576003e-ed26-4e09-aef4-1f9bddc69a5d"). InnerVolumeSpecName "kube-api-access-sbc8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:20.992679 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:20.992657 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8757e465-e3fa-42aa-8911-67323ce3cc2d-kube-api-access-2xp28" pod "8757e465-e3fa-42aa-8911-67323ce3cc2d" (UID: "8757e465-e3fa-42aa-8911-67323ce3cc2d"). InnerVolumeSpecName "kube-api-access-2xp28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:21.000261 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:21.000239 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cf4621-29889660-xbjgx"] Oct 30 17:00:21.009292 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:21.009095 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cf4621-29889660-xbjgx"] Oct 30 17:00:21.045523 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:21.045498 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889660-q5xx4" Oct 30 17:00:21.088229 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:21.088197 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/debf5df6-8130-4b62-a23f-74a234816389-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/debf5df6-8130-4b62-a23f-74a234816389-maas-api-service-ca\") pod \"debf5df6-8130-4b62-a23f-74a234816389\" (UID: \"debf5df6-8130-4b62-a23f-74a234816389\") " Oct 30 17:00:21.088414 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:21.088351 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/debf5df6-8130-4b62-a23f-74a234816389-kube-api-access-mw9jc\" (UniqueName: \"kubernetes.io/projected/debf5df6-8130-4b62-a23f-74a234816389-kube-api-access-mw9jc\") pod \"debf5df6-8130-4b62-a23f-74a234816389\" (UID: \"debf5df6-8130-4b62-a23f-74a234816389\") " Oct 30 17:00:21.088579 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:21.088554 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/debf5df6-8130-4b62-a23f-74a234816389-maas-api-service-ca" pod "debf5df6-8130-4b62-a23f-74a234816389" (UID: "debf5df6-8130-4b62-a23f-74a234816389"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:00:21.088905 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:21.088885 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbc8t\" (UniqueName: \"kubernetes.io/projected/3576003e-ed26-4e09-aef4-1f9bddc69a5d-kube-api-access-sbc8t\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:00:21.088984 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:21.088909 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3576003e-ed26-4e09-aef4-1f9bddc69a5d-maas-api-service-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:00:21.088984 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:21.088923 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/debf5df6-8130-4b62-a23f-74a234816389-maas-api-service-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:00:21.088984 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:21.088936 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xp28\" (UniqueName: \"kubernetes.io/projected/8757e465-e3fa-42aa-8911-67323ce3cc2d-kube-api-access-2xp28\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:00:21.090644 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:21.090620 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/debf5df6-8130-4b62-a23f-74a234816389-kube-api-access-mw9jc" pod "debf5df6-8130-4b62-a23f-74a234816389" (UID: "debf5df6-8130-4b62-a23f-74a234816389"). InnerVolumeSpecName "kube-api-access-mw9jc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:21.190609 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:21.190568 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mw9jc\" (UniqueName: \"kubernetes.io/projected/debf5df6-8130-4b62-a23f-74a234816389-kube-api-access-mw9jc\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:00:21.690835 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:21.690793 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3ca339-29889660-pbd75" event={"ID":"8757e465-e3fa-42aa-8911-67323ce3cc2d","Type":"ContainerDied","Data":"225f6a0fba5b8ffb5c78f90b2bd0234a21dea0463626c4ad3f5568a34a7994ea"} Oct 30 17:00:21.690835 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:21.690817 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3ca339-29889660-pbd75" Oct 30 17:00:21.691369 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:21.690851 4493 scope.go:122] "RemoveContainer" containerID="7367196d8f068799c39a2bbb2a89f62a33639e402629d5931662a93b00dff52a" Oct 30 17:00:21.693519 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:21.693497 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889660-q5xx4" Oct 30 17:00:21.693632 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:21.693513 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889660-q5xx4" event={"ID":"debf5df6-8130-4b62-a23f-74a234816389","Type":"ContainerDied","Data":"e966fa75018290b2a314593f77ed27650e2ef450b933123cc62655b4c3b0d942"} Oct 30 17:00:21.694951 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:21.694928 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889660-n8scx" event={"ID":"3576003e-ed26-4e09-aef4-1f9bddc69a5d","Type":"ContainerDied","Data":"723c19e5f605fe9584308e5434faf39345a8236feab70ed5b74a23ce0af6c295"} Oct 30 17:00:21.695038 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:21.694970 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889660-n8scx" Oct 30 17:00:21.717951 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:21.717928 4493 scope.go:122] "RemoveContainer" containerID="5fad27c56efb7dc456e61ae0f61003f918513254ffb4b252436d5d8a574e07fe" Oct 30 17:00:21.730205 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:21.730177 4493 scope.go:122] "RemoveContainer" containerID="65dcd70f540cf997da6796d9e56dfe3f324278670f2a812fb2ddf1d7f7a8c463" Oct 30 17:00:21.773450 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:21.773426 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889660-n8scx"] Oct 30 17:00:21.785254 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:21.785230 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889660-n8scx"] Oct 30 17:00:21.813969 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:21.813930 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889660-q5xx4"] Oct 30 17:00:21.818087 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:21.818065 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889660-q5xx4"] Oct 30 17:00:21.850007 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:21.849980 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3ca339-29889660-pbd75"] Oct 30 17:00:21.854292 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:21.854261 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-3ca339-29889660-pbd75"] Oct 30 17:00:22.934487 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:22.934453 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3576003e-ed26-4e09-aef4-1f9bddc69a5d" path="/var/lib/kubelet/pods/3576003e-ed26-4e09-aef4-1f9bddc69a5d/volumes" Oct 30 17:00:22.935066 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:22.935047 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8757e465-e3fa-42aa-8911-67323ce3cc2d" path="/var/lib/kubelet/pods/8757e465-e3fa-42aa-8911-67323ce3cc2d/volumes" Oct 30 17:00:22.935567 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:22.935550 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94917f70-a899-4b27-86a3-7e420b6856ae" path="/var/lib/kubelet/pods/94917f70-a899-4b27-86a3-7e420b6856ae/volumes" Oct 30 17:00:22.936510 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:22.936488 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2b861e7-c17b-44a9-86c8-8102ecc2691c" path="/var/lib/kubelet/pods/d2b861e7-c17b-44a9-86c8-8102ecc2691c/volumes" Oct 30 17:00:22.937048 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:22.937031 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="debf5df6-8130-4b62-a23f-74a234816389" path="/var/lib/kubelet/pods/debf5df6-8130-4b62-a23f-74a234816389/volumes" Oct 30 17:00:25.209977 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:25.208243 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 17:00:30.359184 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:00:30.359149 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 17:00:32.809524 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:32.809497 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4674c49-tqlhw"] Oct 30 17:00:32.810040 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:32.810025 4493 cpu_manager.go:414] "containerMap: removing container" podUID="8757e465-e3fa-42aa-8911-67323ce3cc2d" containerName="cleanup" Oct 30 17:00:32.810088 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:32.810041 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8757e465-e3fa-42aa-8911-67323ce3cc2d" containerName="cleanup" Oct 30 17:00:32.810088 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:32.810056 4493 cpu_manager.go:414] "containerMap: removing container" podUID="d2b861e7-c17b-44a9-86c8-8102ecc2691c" containerName="cleanup" Oct 30 17:00:32.810088 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:32.810062 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2b861e7-c17b-44a9-86c8-8102ecc2691c" containerName="cleanup" Oct 30 17:00:32.810088 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:32.810072 4493 cpu_manager.go:414] "containerMap: removing container" podUID="32e4a7ff-2f2e-4140-a540-b59282f6a80f" containerName="cleanup" Oct 30 17:00:32.810088 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:32.810078 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32e4a7ff-2f2e-4140-a540-b59282f6a80f" containerName="cleanup" Oct 30 17:00:32.810088 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:32.810084 4493 cpu_manager.go:414] "containerMap: removing container" podUID="708fd15b-ac7c-42e1-aa73-584566905947" containerName="cleanup" Oct 30 17:00:32.810088 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:32.810088 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="708fd15b-ac7c-42e1-aa73-584566905947" containerName="cleanup" Oct 30 17:00:32.810307 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:32.810094 4493 cpu_manager.go:414] "containerMap: removing container" podUID="debf5df6-8130-4b62-a23f-74a234816389" containerName="cleanup" Oct 30 17:00:32.810307 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:32.810099 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="debf5df6-8130-4b62-a23f-74a234816389" containerName="cleanup" Oct 30 17:00:32.810307 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:32.810106 4493 cpu_manager.go:414] "containerMap: removing container" podUID="305e6ca0-4f0e-42e8-ae53-1e495e07a1a3" containerName="cleanup" Oct 30 17:00:32.810307 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:32.810110 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="305e6ca0-4f0e-42e8-ae53-1e495e07a1a3" containerName="cleanup" Oct 30 17:00:32.810307 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:32.810116 4493 cpu_manager.go:414] "containerMap: removing container" podUID="3576003e-ed26-4e09-aef4-1f9bddc69a5d" containerName="cleanup" Oct 30 17:00:32.810307 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:32.810121 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3576003e-ed26-4e09-aef4-1f9bddc69a5d" containerName="cleanup" Oct 30 17:00:32.810307 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:32.810131 4493 cpu_manager.go:414] "containerMap: removing container" podUID="94917f70-a899-4b27-86a3-7e420b6856ae" containerName="cleanup" Oct 30 17:00:32.810307 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:32.810135 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94917f70-a899-4b27-86a3-7e420b6856ae" containerName="cleanup" Oct 30 17:00:32.810307 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:32.810143 4493 cpu_manager.go:414] "containerMap: removing container" podUID="30b24448-c388-4366-a826-fddc6d987015" containerName="cleanup" Oct 30 17:00:32.810307 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:32.810147 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30b24448-c388-4366-a826-fddc6d987015" containerName="cleanup" Oct 30 17:00:32.810307 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:32.810153 4493 cpu_manager.go:414] "containerMap: removing container" podUID="32e4a7ff-2f2e-4140-a540-b59282f6a80f" containerName="cleanup" Oct 30 17:00:32.810307 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:32.810157 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32e4a7ff-2f2e-4140-a540-b59282f6a80f" containerName="cleanup" Oct 30 17:00:32.810307 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:32.810165 4493 cpu_manager.go:414] "containerMap: removing container" podUID="69e0b1a3-2a21-4412-9495-29669c26929b" containerName="cleanup" Oct 30 17:00:32.810307 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:32.810169 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69e0b1a3-2a21-4412-9495-29669c26929b" containerName="cleanup" Oct 30 17:00:32.810307 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:32.810175 4493 cpu_manager.go:414] "containerMap: removing container" podUID="708fd15b-ac7c-42e1-aa73-584566905947" containerName="cleanup" Oct 30 17:00:32.810307 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:32.810178 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="708fd15b-ac7c-42e1-aa73-584566905947" containerName="cleanup" Oct 30 17:00:32.810307 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:32.810187 4493 cpu_manager.go:414] "containerMap: removing container" podUID="30b24448-c388-4366-a826-fddc6d987015" containerName="cleanup" Oct 30 17:00:32.810307 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:32.810191 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30b24448-c388-4366-a826-fddc6d987015" containerName="cleanup" Oct 30 17:00:32.810307 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:32.810201 4493 cpu_manager.go:414] "containerMap: removing container" podUID="d2b861e7-c17b-44a9-86c8-8102ecc2691c" containerName="cleanup" Oct 30 17:00:32.810307 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:32.810205 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2b861e7-c17b-44a9-86c8-8102ecc2691c" containerName="cleanup" Oct 30 17:00:32.810307 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:32.810214 4493 cpu_manager.go:414] "containerMap: removing container" podUID="69e0b1a3-2a21-4412-9495-29669c26929b" containerName="cleanup" Oct 30 17:00:32.810307 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:32.810218 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69e0b1a3-2a21-4412-9495-29669c26929b" containerName="cleanup" Oct 30 17:00:32.810307 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:32.810225 4493 cpu_manager.go:414] "containerMap: removing container" podUID="94917f70-a899-4b27-86a3-7e420b6856ae" containerName="cleanup" Oct 30 17:00:32.810307 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:32.810229 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94917f70-a899-4b27-86a3-7e420b6856ae" containerName="cleanup" Oct 30 17:00:32.810307 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:32.810239 4493 cpu_manager.go:414] "containerMap: removing container" podUID="30b24448-c388-4366-a826-fddc6d987015" containerName="cleanup" Oct 30 17:00:32.810307 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:32.810243 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30b24448-c388-4366-a826-fddc6d987015" containerName="cleanup" Oct 30 17:00:32.810307 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:32.810251 4493 cpu_manager.go:414] "containerMap: removing container" podUID="305e6ca0-4f0e-42e8-ae53-1e495e07a1a3" containerName="cleanup" Oct 30 17:00:32.810307 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:32.810255 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="305e6ca0-4f0e-42e8-ae53-1e495e07a1a3" containerName="cleanup" Oct 30 17:00:32.810307 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:32.810260 4493 cpu_manager.go:414] "containerMap: removing container" podUID="8757e465-e3fa-42aa-8911-67323ce3cc2d" containerName="cleanup" Oct 30 17:00:32.810307 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:32.810265 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8757e465-e3fa-42aa-8911-67323ce3cc2d" containerName="cleanup" Oct 30 17:00:32.810307 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:32.810272 4493 cpu_manager.go:414] "containerMap: removing container" podUID="3576003e-ed26-4e09-aef4-1f9bddc69a5d" containerName="cleanup" Oct 30 17:00:32.810307 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:32.810276 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3576003e-ed26-4e09-aef4-1f9bddc69a5d" containerName="cleanup" Oct 30 17:00:32.810307 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:32.810299 4493 cpu_manager.go:414] "containerMap: removing container" podUID="debf5df6-8130-4b62-a23f-74a234816389" containerName="cleanup" Oct 30 17:00:32.810307 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:32.810304 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="debf5df6-8130-4b62-a23f-74a234816389" containerName="cleanup" Oct 30 17:00:32.810307 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:32.810318 4493 cpu_manager.go:414] "containerMap: removing container" podUID="69e0b1a3-2a21-4412-9495-29669c26929b" containerName="cleanup" Oct 30 17:00:32.810307 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:32.810323 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69e0b1a3-2a21-4412-9495-29669c26929b" containerName="cleanup" Oct 30 17:00:32.811771 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:32.810401 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="94917f70-a899-4b27-86a3-7e420b6856ae" containerName="cleanup" Oct 30 17:00:32.811771 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:32.810409 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="30b24448-c388-4366-a826-fddc6d987015" containerName="cleanup" Oct 30 17:00:32.811771 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:32.810414 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="32e4a7ff-2f2e-4140-a540-b59282f6a80f" containerName="cleanup" Oct 30 17:00:32.811771 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:32.810420 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="8757e465-e3fa-42aa-8911-67323ce3cc2d" containerName="cleanup" Oct 30 17:00:32.811771 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:32.810425 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="69e0b1a3-2a21-4412-9495-29669c26929b" containerName="cleanup" Oct 30 17:00:32.811771 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:32.810430 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="708fd15b-ac7c-42e1-aa73-584566905947" containerName="cleanup" Oct 30 17:00:32.811771 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:32.810436 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="30b24448-c388-4366-a826-fddc6d987015" containerName="cleanup" Oct 30 17:00:32.811771 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:32.810440 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2b861e7-c17b-44a9-86c8-8102ecc2691c" containerName="cleanup" Oct 30 17:00:32.811771 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:32.810445 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="708fd15b-ac7c-42e1-aa73-584566905947" containerName="cleanup" Oct 30 17:00:32.811771 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:32.810451 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2b861e7-c17b-44a9-86c8-8102ecc2691c" containerName="cleanup" Oct 30 17:00:32.811771 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:32.810455 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="32e4a7ff-2f2e-4140-a540-b59282f6a80f" containerName="cleanup" Oct 30 17:00:32.811771 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:32.810460 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="69e0b1a3-2a21-4412-9495-29669c26929b" containerName="cleanup" Oct 30 17:00:32.811771 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:32.810464 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="94917f70-a899-4b27-86a3-7e420b6856ae" containerName="cleanup" Oct 30 17:00:32.811771 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:32.810468 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="94917f70-a899-4b27-86a3-7e420b6856ae" containerName="cleanup" Oct 30 17:00:32.811771 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:32.810474 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="305e6ca0-4f0e-42e8-ae53-1e495e07a1a3" containerName="cleanup" Oct 30 17:00:32.811771 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:32.810479 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="3576003e-ed26-4e09-aef4-1f9bddc69a5d" containerName="cleanup" Oct 30 17:00:32.811771 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:32.810485 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="debf5df6-8130-4b62-a23f-74a234816389" containerName="cleanup" Oct 30 17:00:32.811771 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:32.810490 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="305e6ca0-4f0e-42e8-ae53-1e495e07a1a3" containerName="cleanup" Oct 30 17:00:32.811771 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:32.810495 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="3576003e-ed26-4e09-aef4-1f9bddc69a5d" containerName="cleanup" Oct 30 17:00:32.811771 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:32.810499 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="debf5df6-8130-4b62-a23f-74a234816389" containerName="cleanup" Oct 30 17:00:32.811771 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:32.810504 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="69e0b1a3-2a21-4412-9495-29669c26929b" containerName="cleanup" Oct 30 17:00:32.811771 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:32.810509 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="8757e465-e3fa-42aa-8911-67323ce3cc2d" containerName="cleanup" Oct 30 17:00:32.811771 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:32.810515 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="32e4a7ff-2f2e-4140-a540-b59282f6a80f" containerName="cleanup" Oct 30 17:00:32.811771 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:32.810520 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="debf5df6-8130-4b62-a23f-74a234816389" containerName="cleanup" Oct 30 17:00:32.811771 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:32.810524 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="305e6ca0-4f0e-42e8-ae53-1e495e07a1a3" containerName="cleanup" Oct 30 17:00:32.818866 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:32.817496 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4674c49-tqlhw" Oct 30 17:00:32.836225 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:32.836197 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4674c49-tqlhw"] Oct 30 17:00:32.843630 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:32.843609 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:00:32.843820 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:32.843616 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:00:32.867020 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:32.866994 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:00:32.867216 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:32.867144 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 17:00:32.963763 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:32.963732 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90406845-6ac6-46d0-ac71-99dfd34cc569-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4674c49-tqlhw\" (UID: \"90406845-6ac6-46d0-ac71-99dfd34cc569\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4674c49-tqlhw" Oct 30 17:00:32.963963 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:32.963798 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c245m\" (UniqueName: \"kubernetes.io/projected/90406845-6ac6-46d0-ac71-99dfd34cc569-kube-api-access-c245m\") pod \"maas-api-revoke-sub-e4674c49-tqlhw\" (UID: \"90406845-6ac6-46d0-ac71-99dfd34cc569\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4674c49-tqlhw" Oct 30 17:00:32.963963 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:32.963951 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90406845-6ac6-46d0-ac71-99dfd34cc569-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4674c49-tqlhw\" (UID: \"90406845-6ac6-46d0-ac71-99dfd34cc569\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4674c49-tqlhw" Oct 30 17:00:33.065771 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:33.065686 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90406845-6ac6-46d0-ac71-99dfd34cc569-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4674c49-tqlhw\" (UID: \"90406845-6ac6-46d0-ac71-99dfd34cc569\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4674c49-tqlhw" Oct 30 17:00:33.065771 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:33.065736 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c245m\" (UniqueName: \"kubernetes.io/projected/90406845-6ac6-46d0-ac71-99dfd34cc569-kube-api-access-c245m\") pod \"maas-api-revoke-sub-e4674c49-tqlhw\" (UID: \"90406845-6ac6-46d0-ac71-99dfd34cc569\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4674c49-tqlhw" Oct 30 17:00:33.066042 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:33.065845 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90406845-6ac6-46d0-ac71-99dfd34cc569-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4674c49-tqlhw\" (UID: \"90406845-6ac6-46d0-ac71-99dfd34cc569\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4674c49-tqlhw" Oct 30 17:00:33.066623 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:33.066597 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90406845-6ac6-46d0-ac71-99dfd34cc569-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4674c49-tqlhw\" (UID: \"90406845-6ac6-46d0-ac71-99dfd34cc569\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4674c49-tqlhw" Oct 30 17:00:33.068414 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:33.068394 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90406845-6ac6-46d0-ac71-99dfd34cc569-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4674c49-tqlhw\" (UID: \"90406845-6ac6-46d0-ac71-99dfd34cc569\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4674c49-tqlhw" Oct 30 17:00:33.089228 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:33.089205 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c245m\" (UniqueName: \"kubernetes.io/projected/90406845-6ac6-46d0-ac71-99dfd34cc569-kube-api-access-c245m\") pod \"maas-api-revoke-sub-e4674c49-tqlhw\" (UID: \"90406845-6ac6-46d0-ac71-99dfd34cc569\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4674c49-tqlhw" Oct 30 17:00:33.136547 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:33.136335 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4674c49-tqlhw" Oct 30 17:00:33.450925 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:33.450808 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4674c49-tqlhw"] Oct 30 17:00:33.453434 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:00:33.453402 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90406845_6ac6_46d0_ac71_99dfd34cc569.slice/crio-c4ace8a99ed7c8bfbd3816904341c779af852515f6ee8502b1cff3f0644fd710 WatchSource:0}: Error finding container c4ace8a99ed7c8bfbd3816904341c779af852515f6ee8502b1cff3f0644fd710: Status 404 returned error can't find the container with id c4ace8a99ed7c8bfbd3816904341c779af852515f6ee8502b1cff3f0644fd710 Oct 30 17:00:33.840161 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:33.840126 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4674c49-tqlhw" event={"ID":"90406845-6ac6-46d0-ac71-99dfd34cc569","Type":"ContainerStarted","Data":"fb14f05513ed9b5a9aa042e31de214fc0bede369020345715ecbd0448f3422ba"} Oct 30 17:00:33.840581 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:33.840192 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4674c49-tqlhw" event={"ID":"90406845-6ac6-46d0-ac71-99dfd34cc569","Type":"ContainerStarted","Data":"c4ace8a99ed7c8bfbd3816904341c779af852515f6ee8502b1cff3f0644fd710"} Oct 30 17:00:33.887971 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:33.887919 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4674c49-tqlhw" podStartSLOduration=1.887904574 podStartE2EDuration="1.887904574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:00:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:00:33.885329628 +0000 UTC m=+2261.112504730" watchObservedRunningTime="2026-10-30 17:00:33.887904574 +0000 UTC m=+2261.115079670" Oct 30 17:00:34.854818 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:34.854706 4493 generic.go:356] "Generic (PLEG): container finished" podID="90406845-6ac6-46d0-ac71-99dfd34cc569" containerID="fb14f05513ed9b5a9aa042e31de214fc0bede369020345715ecbd0448f3422ba" exitCode=0 Oct 30 17:00:34.855310 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:34.854931 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4674c49-tqlhw" event={"ID":"90406845-6ac6-46d0-ac71-99dfd34cc569","Type":"ContainerDied","Data":"fb14f05513ed9b5a9aa042e31de214fc0bede369020345715ecbd0448f3422ba"} Oct 30 17:00:36.015393 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:36.015363 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4674c49-tqlhw" Oct 30 17:00:36.043732 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:36.043700 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90406845-6ac6-46d0-ac71-99dfd34cc569-kube-api-access-c245m\" (UniqueName: \"kubernetes.io/projected/90406845-6ac6-46d0-ac71-99dfd34cc569-kube-api-access-c245m\") pod \"90406845-6ac6-46d0-ac71-99dfd34cc569\" (UID: \"90406845-6ac6-46d0-ac71-99dfd34cc569\") " Oct 30 17:00:36.044027 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:36.044009 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90406845-6ac6-46d0-ac71-99dfd34cc569-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90406845-6ac6-46d0-ac71-99dfd34cc569-maas-api-xks-ca\") pod \"90406845-6ac6-46d0-ac71-99dfd34cc569\" (UID: \"90406845-6ac6-46d0-ac71-99dfd34cc569\") " Oct 30 17:00:36.044082 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:36.044048 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90406845-6ac6-46d0-ac71-99dfd34cc569-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90406845-6ac6-46d0-ac71-99dfd34cc569-maas-api-openshift-ca\") pod \"90406845-6ac6-46d0-ac71-99dfd34cc569\" (UID: \"90406845-6ac6-46d0-ac71-99dfd34cc569\") " Oct 30 17:00:36.044477 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:36.044449 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90406845-6ac6-46d0-ac71-99dfd34cc569-maas-api-openshift-ca" pod "90406845-6ac6-46d0-ac71-99dfd34cc569" (UID: "90406845-6ac6-46d0-ac71-99dfd34cc569"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:00:36.046147 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:36.046124 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90406845-6ac6-46d0-ac71-99dfd34cc569-maas-api-xks-ca" pod "90406845-6ac6-46d0-ac71-99dfd34cc569" (UID: "90406845-6ac6-46d0-ac71-99dfd34cc569"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:00:36.046429 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:36.046407 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90406845-6ac6-46d0-ac71-99dfd34cc569-kube-api-access-c245m" pod "90406845-6ac6-46d0-ac71-99dfd34cc569" (UID: "90406845-6ac6-46d0-ac71-99dfd34cc569"). InnerVolumeSpecName "kube-api-access-c245m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:36.146049 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:36.146018 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90406845-6ac6-46d0-ac71-99dfd34cc569-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:00:36.146049 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:36.146048 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90406845-6ac6-46d0-ac71-99dfd34cc569-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:00:36.146249 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:36.146062 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c245m\" (UniqueName: \"kubernetes.io/projected/90406845-6ac6-46d0-ac71-99dfd34cc569-kube-api-access-c245m\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:00:36.870561 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:36.870528 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4674c49-tqlhw" Oct 30 17:00:36.870764 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:36.870534 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4674c49-tqlhw" event={"ID":"90406845-6ac6-46d0-ac71-99dfd34cc569","Type":"ContainerDied","Data":"c4ace8a99ed7c8bfbd3816904341c779af852515f6ee8502b1cff3f0644fd710"} Oct 30 17:00:36.870764 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:36.870704 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4ace8a99ed7c8bfbd3816904341c779af852515f6ee8502b1cff3f0644fd710" Oct 30 17:00:43.158455 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:43.158412 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4aeb7b96-6tfgm"] Oct 30 17:00:43.159231 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:43.159210 4493 cpu_manager.go:414] "containerMap: removing container" podUID="94917f70-a899-4b27-86a3-7e420b6856ae" containerName="cleanup" Oct 30 17:00:43.159231 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:43.159231 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94917f70-a899-4b27-86a3-7e420b6856ae" containerName="cleanup" Oct 30 17:00:43.159448 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:43.159259 4493 cpu_manager.go:414] "containerMap: removing container" podUID="8757e465-e3fa-42aa-8911-67323ce3cc2d" containerName="cleanup" Oct 30 17:00:43.159448 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:43.159267 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8757e465-e3fa-42aa-8911-67323ce3cc2d" containerName="cleanup" Oct 30 17:00:43.159448 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:43.159280 4493 cpu_manager.go:414] "containerMap: removing container" podUID="d2b861e7-c17b-44a9-86c8-8102ecc2691c" containerName="cleanup" Oct 30 17:00:43.159448 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:43.159285 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2b861e7-c17b-44a9-86c8-8102ecc2691c" containerName="cleanup" Oct 30 17:00:43.159448 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:43.159293 4493 cpu_manager.go:414] "containerMap: removing container" podUID="32e4a7ff-2f2e-4140-a540-b59282f6a80f" containerName="cleanup" Oct 30 17:00:43.159448 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:43.159298 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32e4a7ff-2f2e-4140-a540-b59282f6a80f" containerName="cleanup" Oct 30 17:00:43.159448 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:43.159304 4493 cpu_manager.go:414] "containerMap: removing container" podUID="90406845-6ac6-46d0-ac71-99dfd34cc569" containerName="revoke-keys" Oct 30 17:00:43.159448 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:43.159309 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90406845-6ac6-46d0-ac71-99dfd34cc569" containerName="revoke-keys" Oct 30 17:00:43.159448 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:43.159315 4493 cpu_manager.go:414] "containerMap: removing container" podUID="708fd15b-ac7c-42e1-aa73-584566905947" containerName="cleanup" Oct 30 17:00:43.159448 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:43.159320 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="708fd15b-ac7c-42e1-aa73-584566905947" containerName="cleanup" Oct 30 17:00:43.159448 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:43.159326 4493 cpu_manager.go:414] "containerMap: removing container" podUID="debf5df6-8130-4b62-a23f-74a234816389" containerName="cleanup" Oct 30 17:00:43.159448 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:43.159333 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="debf5df6-8130-4b62-a23f-74a234816389" containerName="cleanup" Oct 30 17:00:43.159448 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:43.159344 4493 cpu_manager.go:414] "containerMap: removing container" podUID="305e6ca0-4f0e-42e8-ae53-1e495e07a1a3" containerName="cleanup" Oct 30 17:00:43.159448 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:43.159349 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="305e6ca0-4f0e-42e8-ae53-1e495e07a1a3" containerName="cleanup" Oct 30 17:00:43.159448 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:43.159356 4493 cpu_manager.go:414] "containerMap: removing container" podUID="3576003e-ed26-4e09-aef4-1f9bddc69a5d" containerName="cleanup" Oct 30 17:00:43.159448 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:43.159360 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3576003e-ed26-4e09-aef4-1f9bddc69a5d" containerName="cleanup" Oct 30 17:00:43.160364 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:43.159461 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="30b24448-c388-4366-a826-fddc6d987015" containerName="cleanup" Oct 30 17:00:43.160364 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:43.159471 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="708fd15b-ac7c-42e1-aa73-584566905947" containerName="cleanup" Oct 30 17:00:43.160364 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:43.159477 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2b861e7-c17b-44a9-86c8-8102ecc2691c" containerName="cleanup" Oct 30 17:00:43.160364 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:43.159485 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="3576003e-ed26-4e09-aef4-1f9bddc69a5d" containerName="cleanup" Oct 30 17:00:43.160364 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:43.159495 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="8757e465-e3fa-42aa-8911-67323ce3cc2d" containerName="cleanup" Oct 30 17:00:43.160364 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:43.159506 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="90406845-6ac6-46d0-ac71-99dfd34cc569" containerName="revoke-keys" Oct 30 17:00:43.163344 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:43.163323 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aeb7b96-6tfgm" Oct 30 17:00:43.192808 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:43.192095 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:00:43.192808 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:43.192744 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:00:43.192808 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:43.192119 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:00:43.193080 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:43.192862 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 17:00:43.205852 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:43.204221 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4aeb7b96-6tfgm"] Oct 30 17:00:43.273251 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:43.273219 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de79a823-3142-4ce4-a646-09072a736b9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4aeb7b96-6tfgm\" (UID: \"de79a823-3142-4ce4-a646-09072a736b9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aeb7b96-6tfgm" Oct 30 17:00:43.273469 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:43.273441 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vmjt\" (UniqueName: \"kubernetes.io/projected/de79a823-3142-4ce4-a646-09072a736b9f-kube-api-access-2vmjt\") pod \"maas-api-revoke-sub-4aeb7b96-6tfgm\" (UID: \"de79a823-3142-4ce4-a646-09072a736b9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aeb7b96-6tfgm" Oct 30 17:00:43.273554 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:43.273527 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de79a823-3142-4ce4-a646-09072a736b9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4aeb7b96-6tfgm\" (UID: \"de79a823-3142-4ce4-a646-09072a736b9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aeb7b96-6tfgm" Oct 30 17:00:43.375396 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:43.375365 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vmjt\" (UniqueName: \"kubernetes.io/projected/de79a823-3142-4ce4-a646-09072a736b9f-kube-api-access-2vmjt\") pod \"maas-api-revoke-sub-4aeb7b96-6tfgm\" (UID: \"de79a823-3142-4ce4-a646-09072a736b9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aeb7b96-6tfgm" Oct 30 17:00:43.375614 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:43.375422 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de79a823-3142-4ce4-a646-09072a736b9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4aeb7b96-6tfgm\" (UID: \"de79a823-3142-4ce4-a646-09072a736b9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aeb7b96-6tfgm" Oct 30 17:00:43.375614 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:43.375466 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de79a823-3142-4ce4-a646-09072a736b9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4aeb7b96-6tfgm\" (UID: \"de79a823-3142-4ce4-a646-09072a736b9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aeb7b96-6tfgm" Oct 30 17:00:43.376115 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:43.376093 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de79a823-3142-4ce4-a646-09072a736b9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4aeb7b96-6tfgm\" (UID: \"de79a823-3142-4ce4-a646-09072a736b9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aeb7b96-6tfgm" Oct 30 17:00:43.378132 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:43.378104 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de79a823-3142-4ce4-a646-09072a736b9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4aeb7b96-6tfgm\" (UID: \"de79a823-3142-4ce4-a646-09072a736b9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aeb7b96-6tfgm" Oct 30 17:00:43.398969 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:43.398944 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vmjt\" (UniqueName: \"kubernetes.io/projected/de79a823-3142-4ce4-a646-09072a736b9f-kube-api-access-2vmjt\") pod \"maas-api-revoke-sub-4aeb7b96-6tfgm\" (UID: \"de79a823-3142-4ce4-a646-09072a736b9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aeb7b96-6tfgm" Oct 30 17:00:43.480733 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:43.480645 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aeb7b96-6tfgm" Oct 30 17:00:43.803942 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:43.803913 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4aeb7b96-6tfgm"] Oct 30 17:00:43.805229 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:00:43.805199 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde79a823_3142_4ce4_a646_09072a736b9f.slice/crio-52c892fe8ba3fcb1031705b7fb302ddd560ee12eec46704668a60c6920abbb8c WatchSource:0}: Error finding container 52c892fe8ba3fcb1031705b7fb302ddd560ee12eec46704668a60c6920abbb8c: Status 404 returned error can't find the container with id 52c892fe8ba3fcb1031705b7fb302ddd560ee12eec46704668a60c6920abbb8c Oct 30 17:00:43.935277 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:43.935236 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aeb7b96-6tfgm" event={"ID":"de79a823-3142-4ce4-a646-09072a736b9f","Type":"ContainerStarted","Data":"52c892fe8ba3fcb1031705b7fb302ddd560ee12eec46704668a60c6920abbb8c"} Oct 30 17:00:44.943869 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:44.943827 4493 generic.go:356] "Generic (PLEG): container finished" podID="de79a823-3142-4ce4-a646-09072a736b9f" containerID="57e3bad5a687545701e35ee329eef30575ad539de2677c94a64acd943979f9f3" exitCode=0 Oct 30 17:00:44.944261 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:44.943959 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aeb7b96-6tfgm" event={"ID":"de79a823-3142-4ce4-a646-09072a736b9f","Type":"ContainerDied","Data":"57e3bad5a687545701e35ee329eef30575ad539de2677c94a64acd943979f9f3"} Oct 30 17:00:46.140972 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:46.140942 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aeb7b96-6tfgm" Oct 30 17:00:46.219563 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:46.219529 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de79a823-3142-4ce4-a646-09072a736b9f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de79a823-3142-4ce4-a646-09072a736b9f-maas-api-openshift-ca\") pod \"de79a823-3142-4ce4-a646-09072a736b9f\" (UID: \"de79a823-3142-4ce4-a646-09072a736b9f\") " Oct 30 17:00:46.219563 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:46.219566 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de79a823-3142-4ce4-a646-09072a736b9f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de79a823-3142-4ce4-a646-09072a736b9f-maas-api-xks-ca\") pod \"de79a823-3142-4ce4-a646-09072a736b9f\" (UID: \"de79a823-3142-4ce4-a646-09072a736b9f\") " Oct 30 17:00:46.219869 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:46.219603 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de79a823-3142-4ce4-a646-09072a736b9f-kube-api-access-2vmjt\" (UniqueName: \"kubernetes.io/projected/de79a823-3142-4ce4-a646-09072a736b9f-kube-api-access-2vmjt\") pod \"de79a823-3142-4ce4-a646-09072a736b9f\" (UID: \"de79a823-3142-4ce4-a646-09072a736b9f\") " Oct 30 17:00:46.220068 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:46.220043 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de79a823-3142-4ce4-a646-09072a736b9f-maas-api-openshift-ca" pod "de79a823-3142-4ce4-a646-09072a736b9f" (UID: "de79a823-3142-4ce4-a646-09072a736b9f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:00:46.222129 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:46.222099 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de79a823-3142-4ce4-a646-09072a736b9f-maas-api-xks-ca" pod "de79a823-3142-4ce4-a646-09072a736b9f" (UID: "de79a823-3142-4ce4-a646-09072a736b9f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:00:46.222540 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:46.222519 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de79a823-3142-4ce4-a646-09072a736b9f-kube-api-access-2vmjt" pod "de79a823-3142-4ce4-a646-09072a736b9f" (UID: "de79a823-3142-4ce4-a646-09072a736b9f"). InnerVolumeSpecName "kube-api-access-2vmjt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:46.320796 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:46.320742 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de79a823-3142-4ce4-a646-09072a736b9f-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:00:46.320796 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:46.320773 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de79a823-3142-4ce4-a646-09072a736b9f-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:00:46.320796 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:46.320802 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vmjt\" (UniqueName: \"kubernetes.io/projected/de79a823-3142-4ce4-a646-09072a736b9f-kube-api-access-2vmjt\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:00:46.961599 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:46.961568 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aeb7b96-6tfgm" Oct 30 17:00:46.961801 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:46.961605 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4aeb7b96-6tfgm" event={"ID":"de79a823-3142-4ce4-a646-09072a736b9f","Type":"ContainerDied","Data":"52c892fe8ba3fcb1031705b7fb302ddd560ee12eec46704668a60c6920abbb8c"} Oct 30 17:00:46.961801 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:46.961659 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52c892fe8ba3fcb1031705b7fb302ddd560ee12eec46704668a60c6920abbb8c" Oct 30 17:00:47.759713 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:47.759680 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt"] Oct 30 17:00:47.760182 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:47.759945 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt" podUID="4c800b1d-54c3-4eae-911c-32979c3e23fa" containerName="istio-proxy" containerID="cri-o://9b55b8eed833264014d170f82bdfa437f47ee513770f61f670e820830be924e2" gracePeriod=30 Oct 30 17:00:47.771043 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:47.771014 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb"] Oct 30 17:00:47.771293 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:47.771270 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb" podUID="8e2c8233-f7d9-4624-99b3-e022639cbc45" containerName="istio-proxy" containerID="cri-o://872c385de502349f4b39cedb60ad37b3882de5fc255df10988cfb03051c7130c" gracePeriod=30 Oct 30 17:00:52.143147 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:52.143092 4493 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:00:52.143542 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:52.143158 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb" podUID="8e2c8233-f7d9-4624-99b3-e022639cbc45" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:00:53.011097 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:53.010929 4493 generic.go:356] "Generic (PLEG): container finished" podID="4c800b1d-54c3-4eae-911c-32979c3e23fa" containerID="9b55b8eed833264014d170f82bdfa437f47ee513770f61f670e820830be924e2" exitCode=0 Oct 30 17:00:53.011097 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:53.011024 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt" event={"ID":"4c800b1d-54c3-4eae-911c-32979c3e23fa","Type":"ContainerDied","Data":"9b55b8eed833264014d170f82bdfa437f47ee513770f61f670e820830be924e2"} Oct 30 17:00:53.013205 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:53.013175 4493 generic.go:356] "Generic (PLEG): container finished" podID="8e2c8233-f7d9-4624-99b3-e022639cbc45" containerID="872c385de502349f4b39cedb60ad37b3882de5fc255df10988cfb03051c7130c" exitCode=0 Oct 30 17:00:53.013555 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:53.013533 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb" event={"ID":"8e2c8233-f7d9-4624-99b3-e022639cbc45","Type":"ContainerDied","Data":"872c385de502349f4b39cedb60ad37b3882de5fc255df10988cfb03051c7130c"} Oct 30 17:00:53.080416 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:53.080389 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt" Oct 30 17:00:53.088089 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:53.088069 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb" Oct 30 17:00:53.202198 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:53.202161 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c800b1d-54c3-4eae-911c-32979c3e23fa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c800b1d-54c3-4eae-911c-32979c3e23fa-istio-envoy\") pod \"4c800b1d-54c3-4eae-911c-32979c3e23fa\" (UID: \"4c800b1d-54c3-4eae-911c-32979c3e23fa\") " Oct 30 17:00:53.202738 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:53.202218 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c800b1d-54c3-4eae-911c-32979c3e23fa-kube-api-access-wbj7v\" (UniqueName: \"kubernetes.io/projected/4c800b1d-54c3-4eae-911c-32979c3e23fa-kube-api-access-wbj7v\") pod \"4c800b1d-54c3-4eae-911c-32979c3e23fa\" (UID: \"4c800b1d-54c3-4eae-911c-32979c3e23fa\") " Oct 30 17:00:53.202738 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:53.202285 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e2c8233-f7d9-4624-99b3-e022639cbc45-kube-api-access-msftw\" (UniqueName: \"kubernetes.io/projected/8e2c8233-f7d9-4624-99b3-e022639cbc45-kube-api-access-msftw\") pod \"8e2c8233-f7d9-4624-99b3-e022639cbc45\" (UID: \"8e2c8233-f7d9-4624-99b3-e022639cbc45\") " Oct 30 17:00:53.202738 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:53.202320 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e2c8233-f7d9-4624-99b3-e022639cbc45-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2c8233-f7d9-4624-99b3-e022639cbc45-credential-socket\") pod \"8e2c8233-f7d9-4624-99b3-e022639cbc45\" (UID: \"8e2c8233-f7d9-4624-99b3-e022639cbc45\") " Oct 30 17:00:53.202738 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:53.202349 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e2c8233-f7d9-4624-99b3-e022639cbc45-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e2c8233-f7d9-4624-99b3-e022639cbc45-workload-certs\") pod \"8e2c8233-f7d9-4624-99b3-e022639cbc45\" (UID: \"8e2c8233-f7d9-4624-99b3-e022639cbc45\") " Oct 30 17:00:53.202738 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:53.202384 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c800b1d-54c3-4eae-911c-32979c3e23fa-istio-token\" (UniqueName: \"kubernetes.io/projected/4c800b1d-54c3-4eae-911c-32979c3e23fa-istio-token\") pod \"4c800b1d-54c3-4eae-911c-32979c3e23fa\" (UID: \"4c800b1d-54c3-4eae-911c-32979c3e23fa\") " Oct 30 17:00:53.202738 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:53.202416 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e2c8233-f7d9-4624-99b3-e022639cbc45-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2c8233-f7d9-4624-99b3-e022639cbc45-workload-socket\") pod \"8e2c8233-f7d9-4624-99b3-e022639cbc45\" (UID: \"8e2c8233-f7d9-4624-99b3-e022639cbc45\") " Oct 30 17:00:53.202738 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:53.202528 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e2c8233-f7d9-4624-99b3-e022639cbc45-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e2c8233-f7d9-4624-99b3-e022639cbc45-istiod-ca-cert\") pod \"8e2c8233-f7d9-4624-99b3-e022639cbc45\" (UID: \"8e2c8233-f7d9-4624-99b3-e022639cbc45\") " Oct 30 17:00:53.202738 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:53.202564 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e2c8233-f7d9-4624-99b3-e022639cbc45-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e2c8233-f7d9-4624-99b3-e022639cbc45-istio-envoy\") pod \"8e2c8233-f7d9-4624-99b3-e022639cbc45\" (UID: \"8e2c8233-f7d9-4624-99b3-e022639cbc45\") " Oct 30 17:00:53.202738 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:53.202606 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c800b1d-54c3-4eae-911c-32979c3e23fa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c800b1d-54c3-4eae-911c-32979c3e23fa-workload-socket\") pod \"4c800b1d-54c3-4eae-911c-32979c3e23fa\" (UID: \"4c800b1d-54c3-4eae-911c-32979c3e23fa\") " Oct 30 17:00:53.202738 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:53.202671 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c800b1d-54c3-4eae-911c-32979c3e23fa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c800b1d-54c3-4eae-911c-32979c3e23fa-workload-certs\") pod \"4c800b1d-54c3-4eae-911c-32979c3e23fa\" (UID: \"4c800b1d-54c3-4eae-911c-32979c3e23fa\") " Oct 30 17:00:53.202738 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:53.202705 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4c800b1d-54c3-4eae-911c-32979c3e23fa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c800b1d-54c3-4eae-911c-32979c3e23fa-istio-podinfo\") pod \"4c800b1d-54c3-4eae-911c-32979c3e23fa\" (UID: \"4c800b1d-54c3-4eae-911c-32979c3e23fa\") " Oct 30 17:00:53.202738 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:53.202731 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e2c8233-f7d9-4624-99b3-e022639cbc45-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e2c8233-f7d9-4624-99b3-e022639cbc45-istio-data\") pod \"8e2c8233-f7d9-4624-99b3-e022639cbc45\" (UID: \"8e2c8233-f7d9-4624-99b3-e022639cbc45\") " Oct 30 17:00:53.203493 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:53.202763 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e2c8233-f7d9-4624-99b3-e022639cbc45-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e2c8233-f7d9-4624-99b3-e022639cbc45-istio-podinfo\") pod \"8e2c8233-f7d9-4624-99b3-e022639cbc45\" (UID: \"8e2c8233-f7d9-4624-99b3-e022639cbc45\") " Oct 30 17:00:53.203493 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:53.202818 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e2c8233-f7d9-4624-99b3-e022639cbc45-workload-socket" pod "8e2c8233-f7d9-4624-99b3-e022639cbc45" (UID: "8e2c8233-f7d9-4624-99b3-e022639cbc45"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:00:53.203493 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:53.202845 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e2c8233-f7d9-4624-99b3-e022639cbc45-istio-token\" (UniqueName: \"kubernetes.io/projected/8e2c8233-f7d9-4624-99b3-e022639cbc45-istio-token\") pod \"8e2c8233-f7d9-4624-99b3-e022639cbc45\" (UID: \"8e2c8233-f7d9-4624-99b3-e022639cbc45\") " Oct 30 17:00:53.203493 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:53.202942 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c800b1d-54c3-4eae-911c-32979c3e23fa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c800b1d-54c3-4eae-911c-32979c3e23fa-istiod-ca-cert\") pod \"4c800b1d-54c3-4eae-911c-32979c3e23fa\" (UID: \"4c800b1d-54c3-4eae-911c-32979c3e23fa\") " Oct 30 17:00:53.203493 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:53.202981 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c800b1d-54c3-4eae-911c-32979c3e23fa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c800b1d-54c3-4eae-911c-32979c3e23fa-istio-data\") pod \"4c800b1d-54c3-4eae-911c-32979c3e23fa\" (UID: \"4c800b1d-54c3-4eae-911c-32979c3e23fa\") " Oct 30 17:00:53.203493 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:53.203014 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c800b1d-54c3-4eae-911c-32979c3e23fa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c800b1d-54c3-4eae-911c-32979c3e23fa-credential-socket\") pod \"4c800b1d-54c3-4eae-911c-32979c3e23fa\" (UID: \"4c800b1d-54c3-4eae-911c-32979c3e23fa\") " Oct 30 17:00:53.203493 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:53.203165 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e2c8233-f7d9-4624-99b3-e022639cbc45-workload-certs" pod "8e2c8233-f7d9-4624-99b3-e022639cbc45" (UID: "8e2c8233-f7d9-4624-99b3-e022639cbc45"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:00:53.203493 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:53.203282 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c800b1d-54c3-4eae-911c-32979c3e23fa-workload-socket" pod "4c800b1d-54c3-4eae-911c-32979c3e23fa" (UID: "4c800b1d-54c3-4eae-911c-32979c3e23fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:00:53.203974 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:53.203607 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e2c8233-f7d9-4624-99b3-e022639cbc45-workload-certs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:00:53.203974 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:53.203629 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2c8233-f7d9-4624-99b3-e022639cbc45-workload-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:00:53.203974 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:53.203644 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c800b1d-54c3-4eae-911c-32979c3e23fa-workload-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:00:53.204136 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:53.203997 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e2c8233-f7d9-4624-99b3-e022639cbc45-istiod-ca-cert" pod "8e2c8233-f7d9-4624-99b3-e022639cbc45" (UID: "8e2c8233-f7d9-4624-99b3-e022639cbc45"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:00:53.204986 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:53.204924 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c800b1d-54c3-4eae-911c-32979c3e23fa-istio-data" pod "4c800b1d-54c3-4eae-911c-32979c3e23fa" (UID: "4c800b1d-54c3-4eae-911c-32979c3e23fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:00:53.205158 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:53.205011 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c800b1d-54c3-4eae-911c-32979c3e23fa-istio-envoy" pod "4c800b1d-54c3-4eae-911c-32979c3e23fa" (UID: "4c800b1d-54c3-4eae-911c-32979c3e23fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:00:53.205239 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:53.205180 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c800b1d-54c3-4eae-911c-32979c3e23fa-workload-certs" pod "4c800b1d-54c3-4eae-911c-32979c3e23fa" (UID: "4c800b1d-54c3-4eae-911c-32979c3e23fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:00:53.205485 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:53.205357 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c800b1d-54c3-4eae-911c-32979c3e23fa-credential-socket" pod "4c800b1d-54c3-4eae-911c-32979c3e23fa" (UID: "4c800b1d-54c3-4eae-911c-32979c3e23fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:00:53.205485 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:53.205454 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c800b1d-54c3-4eae-911c-32979c3e23fa-istiod-ca-cert" pod "4c800b1d-54c3-4eae-911c-32979c3e23fa" (UID: "4c800b1d-54c3-4eae-911c-32979c3e23fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:00:53.205665 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:53.205563 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e2c8233-f7d9-4624-99b3-e022639cbc45-istio-data" pod "8e2c8233-f7d9-4624-99b3-e022639cbc45" (UID: "8e2c8233-f7d9-4624-99b3-e022639cbc45"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:00:53.205774 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:53.205752 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e2c8233-f7d9-4624-99b3-e022639cbc45-credential-socket" pod "8e2c8233-f7d9-4624-99b3-e022639cbc45" (UID: "8e2c8233-f7d9-4624-99b3-e022639cbc45"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:00:53.208259 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:53.208221 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e2c8233-f7d9-4624-99b3-e022639cbc45-istio-token" pod "8e2c8233-f7d9-4624-99b3-e022639cbc45" (UID: "8e2c8233-f7d9-4624-99b3-e022639cbc45"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:53.208974 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:53.208921 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e2c8233-f7d9-4624-99b3-e022639cbc45-kube-api-access-msftw" pod "8e2c8233-f7d9-4624-99b3-e022639cbc45" (UID: "8e2c8233-f7d9-4624-99b3-e022639cbc45"). InnerVolumeSpecName "kube-api-access-msftw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:53.215522 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:53.215494 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e2c8233-f7d9-4624-99b3-e022639cbc45-istio-podinfo" pod "8e2c8233-f7d9-4624-99b3-e022639cbc45" (UID: "8e2c8233-f7d9-4624-99b3-e022639cbc45"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:00:53.215630 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:53.215522 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e2c8233-f7d9-4624-99b3-e022639cbc45-istio-envoy" pod "8e2c8233-f7d9-4624-99b3-e022639cbc45" (UID: "8e2c8233-f7d9-4624-99b3-e022639cbc45"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:00:53.215630 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:53.215555 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c800b1d-54c3-4eae-911c-32979c3e23fa-istio-token" pod "4c800b1d-54c3-4eae-911c-32979c3e23fa" (UID: "4c800b1d-54c3-4eae-911c-32979c3e23fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:53.215630 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:53.215548 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c800b1d-54c3-4eae-911c-32979c3e23fa-kube-api-access-wbj7v" pod "4c800b1d-54c3-4eae-911c-32979c3e23fa" (UID: "4c800b1d-54c3-4eae-911c-32979c3e23fa"). InnerVolumeSpecName "kube-api-access-wbj7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:53.217252 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:53.217223 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4c800b1d-54c3-4eae-911c-32979c3e23fa-istio-podinfo" pod "4c800b1d-54c3-4eae-911c-32979c3e23fa" (UID: "4c800b1d-54c3-4eae-911c-32979c3e23fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:00:53.305862 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:53.305565 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c800b1d-54c3-4eae-911c-32979c3e23fa-workload-certs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:00:53.305862 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:53.305603 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c800b1d-54c3-4eae-911c-32979c3e23fa-istio-podinfo\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:00:53.305862 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:53.305617 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e2c8233-f7d9-4624-99b3-e022639cbc45-istio-data\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:00:53.305862 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:53.305631 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e2c8233-f7d9-4624-99b3-e022639cbc45-istio-podinfo\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:00:53.305862 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:53.305645 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e2c8233-f7d9-4624-99b3-e022639cbc45-istio-token\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:00:53.305862 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:53.305658 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c800b1d-54c3-4eae-911c-32979c3e23fa-istiod-ca-cert\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:00:53.305862 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:53.305669 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c800b1d-54c3-4eae-911c-32979c3e23fa-istio-data\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:00:53.305862 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:53.305681 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c800b1d-54c3-4eae-911c-32979c3e23fa-credential-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:00:53.305862 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:53.305692 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c800b1d-54c3-4eae-911c-32979c3e23fa-istio-envoy\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:00:53.305862 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:53.305706 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbj7v\" (UniqueName: \"kubernetes.io/projected/4c800b1d-54c3-4eae-911c-32979c3e23fa-kube-api-access-wbj7v\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:00:53.305862 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:53.305719 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msftw\" (UniqueName: \"kubernetes.io/projected/8e2c8233-f7d9-4624-99b3-e022639cbc45-kube-api-access-msftw\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:00:53.305862 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:53.305732 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2c8233-f7d9-4624-99b3-e022639cbc45-credential-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:00:53.305862 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:53.305744 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c800b1d-54c3-4eae-911c-32979c3e23fa-istio-token\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:00:53.305862 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:53.305758 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e2c8233-f7d9-4624-99b3-e022639cbc45-istiod-ca-cert\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:00:53.305862 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:53.305768 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e2c8233-f7d9-4624-99b3-e022639cbc45-istio-envoy\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:00:54.021702 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:54.021660 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt" event={"ID":"4c800b1d-54c3-4eae-911c-32979c3e23fa","Type":"ContainerDied","Data":"f57c4e75aad0073ff3e341f3246d935b5e9d84cf191a15157bb35f8c6e93b587"} Oct 30 17:00:54.021949 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:54.021737 4493 scope.go:122] "RemoveContainer" containerID="9b55b8eed833264014d170f82bdfa437f47ee513770f61f670e820830be924e2" Oct 30 17:00:54.021949 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:54.021675 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt" Oct 30 17:00:54.023306 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:54.023287 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb" Oct 30 17:00:54.023306 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:54.023294 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb" event={"ID":"8e2c8233-f7d9-4624-99b3-e022639cbc45","Type":"ContainerDied","Data":"d4568bba043d6effa567f04f6b5b025b6a2cdb32bbe4cd899b0b9086c2b96063"} Oct 30 17:00:54.034771 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:54.034753 4493 scope.go:122] "RemoveContainer" containerID="872c385de502349f4b39cedb60ad37b3882de5fc255df10988cfb03051c7130c" Oct 30 17:00:54.075138 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:54.075110 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt"] Oct 30 17:00:54.086825 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:54.086798 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vzhvt"] Oct 30 17:00:54.110392 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:54.110363 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb"] Oct 30 17:00:54.113608 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:54.113578 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-dtkxb"] Oct 30 17:00:54.932980 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:54.932947 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c800b1d-54c3-4eae-911c-32979c3e23fa" path="/var/lib/kubelet/pods/4c800b1d-54c3-4eae-911c-32979c3e23fa/volumes" Oct 30 17:00:54.933470 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:54.933385 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e2c8233-f7d9-4624-99b3-e022639cbc45" path="/var/lib/kubelet/pods/8e2c8233-f7d9-4624-99b3-e022639cbc45/volumes" Oct 30 17:00:58.435911 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:58.435876 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-8ddjc"] Oct 30 17:00:58.436599 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:58.436577 4493 cpu_manager.go:414] "containerMap: removing container" podUID="8e2c8233-f7d9-4624-99b3-e022639cbc45" containerName="istio-proxy" Oct 30 17:00:58.436599 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:58.436597 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e2c8233-f7d9-4624-99b3-e022639cbc45" containerName="istio-proxy" Oct 30 17:00:58.436882 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:58.436614 4493 cpu_manager.go:414] "containerMap: removing container" podUID="de79a823-3142-4ce4-a646-09072a736b9f" containerName="revoke-keys" Oct 30 17:00:58.436882 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:58.436621 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de79a823-3142-4ce4-a646-09072a736b9f" containerName="revoke-keys" Oct 30 17:00:58.436882 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:58.436654 4493 cpu_manager.go:414] "containerMap: removing container" podUID="4c800b1d-54c3-4eae-911c-32979c3e23fa" containerName="istio-proxy" Oct 30 17:00:58.436882 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:58.436661 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c800b1d-54c3-4eae-911c-32979c3e23fa" containerName="istio-proxy" Oct 30 17:00:58.436882 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:58.436829 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e2c8233-f7d9-4624-99b3-e022639cbc45" containerName="istio-proxy" Oct 30 17:00:58.436882 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:58.436842 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="de79a823-3142-4ce4-a646-09072a736b9f" containerName="revoke-keys" Oct 30 17:00:58.436882 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:58.436880 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c800b1d-54c3-4eae-911c-32979c3e23fa" containerName="istio-proxy" Oct 30 17:00:58.443345 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:58.443325 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-8ddjc" Oct 30 17:00:58.462454 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:58.460829 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-81733d-openshift-default-dockercfg-6hh6m\"" Oct 30 17:00:58.466576 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:58.466306 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16b79973-7449-41cf-8709-e8109aa31cd2-istio-podinfo\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-8ddjc\" (UID: \"16b79973-7449-41cf-8709-e8109aa31cd2\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-8ddjc" Oct 30 17:00:58.466576 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:58.466387 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16b79973-7449-41cf-8709-e8109aa31cd2-istio-envoy\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-8ddjc\" (UID: \"16b79973-7449-41cf-8709-e8109aa31cd2\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-8ddjc" Oct 30 17:00:58.466576 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:58.466420 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp2l4\" (UniqueName: \"kubernetes.io/projected/16b79973-7449-41cf-8709-e8109aa31cd2-kube-api-access-pp2l4\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-8ddjc\" (UID: \"16b79973-7449-41cf-8709-e8109aa31cd2\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-8ddjc" Oct 30 17:00:58.466576 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:58.466487 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b79973-7449-41cf-8709-e8109aa31cd2-workload-socket\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-8ddjc\" (UID: \"16b79973-7449-41cf-8709-e8109aa31cd2\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-8ddjc" Oct 30 17:00:58.466576 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:58.466511 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16b79973-7449-41cf-8709-e8109aa31cd2-istio-token\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-8ddjc\" (UID: \"16b79973-7449-41cf-8709-e8109aa31cd2\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-8ddjc" Oct 30 17:00:58.466576 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:58.466553 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16b79973-7449-41cf-8709-e8109aa31cd2-istio-data\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-8ddjc\" (UID: \"16b79973-7449-41cf-8709-e8109aa31cd2\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-8ddjc" Oct 30 17:00:58.467388 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:58.466642 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16b79973-7449-41cf-8709-e8109aa31cd2-istiod-ca-cert\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-8ddjc\" (UID: \"16b79973-7449-41cf-8709-e8109aa31cd2\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-8ddjc" Oct 30 17:00:58.467388 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:58.466689 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b79973-7449-41cf-8709-e8109aa31cd2-credential-socket\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-8ddjc\" (UID: \"16b79973-7449-41cf-8709-e8109aa31cd2\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-8ddjc" Oct 30 17:00:58.467388 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:58.466712 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16b79973-7449-41cf-8709-e8109aa31cd2-workload-certs\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-8ddjc\" (UID: \"16b79973-7449-41cf-8709-e8109aa31cd2\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-8ddjc" Oct 30 17:00:58.476202 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:58.476174 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-8ddjc"] Oct 30 17:00:58.568004 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:58.567963 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16b79973-7449-41cf-8709-e8109aa31cd2-istio-podinfo\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-8ddjc\" (UID: \"16b79973-7449-41cf-8709-e8109aa31cd2\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-8ddjc" Oct 30 17:00:58.568182 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:58.568055 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16b79973-7449-41cf-8709-e8109aa31cd2-istio-envoy\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-8ddjc\" (UID: \"16b79973-7449-41cf-8709-e8109aa31cd2\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-8ddjc" Oct 30 17:00:58.568182 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:58.568081 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp2l4\" (UniqueName: \"kubernetes.io/projected/16b79973-7449-41cf-8709-e8109aa31cd2-kube-api-access-pp2l4\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-8ddjc\" (UID: \"16b79973-7449-41cf-8709-e8109aa31cd2\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-8ddjc" Oct 30 17:00:58.568182 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:58.568144 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b79973-7449-41cf-8709-e8109aa31cd2-workload-socket\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-8ddjc\" (UID: \"16b79973-7449-41cf-8709-e8109aa31cd2\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-8ddjc" Oct 30 17:00:58.568182 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:58.568169 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16b79973-7449-41cf-8709-e8109aa31cd2-istio-token\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-8ddjc\" (UID: \"16b79973-7449-41cf-8709-e8109aa31cd2\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-8ddjc" Oct 30 17:00:58.568435 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:58.568207 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16b79973-7449-41cf-8709-e8109aa31cd2-istio-data\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-8ddjc\" (UID: \"16b79973-7449-41cf-8709-e8109aa31cd2\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-8ddjc" Oct 30 17:00:58.568435 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:58.568303 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16b79973-7449-41cf-8709-e8109aa31cd2-istiod-ca-cert\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-8ddjc\" (UID: \"16b79973-7449-41cf-8709-e8109aa31cd2\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-8ddjc" Oct 30 17:00:58.568435 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:58.568353 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b79973-7449-41cf-8709-e8109aa31cd2-credential-socket\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-8ddjc\" (UID: \"16b79973-7449-41cf-8709-e8109aa31cd2\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-8ddjc" Oct 30 17:00:58.568435 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:58.568378 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16b79973-7449-41cf-8709-e8109aa31cd2-workload-certs\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-8ddjc\" (UID: \"16b79973-7449-41cf-8709-e8109aa31cd2\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-8ddjc" Oct 30 17:00:58.569127 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:58.569091 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16b79973-7449-41cf-8709-e8109aa31cd2-istio-data\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-8ddjc\" (UID: \"16b79973-7449-41cf-8709-e8109aa31cd2\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-8ddjc" Oct 30 17:00:58.569398 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:58.569336 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b79973-7449-41cf-8709-e8109aa31cd2-credential-socket\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-8ddjc\" (UID: \"16b79973-7449-41cf-8709-e8109aa31cd2\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-8ddjc" Oct 30 17:00:58.569653 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:58.569451 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b79973-7449-41cf-8709-e8109aa31cd2-workload-socket\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-8ddjc\" (UID: \"16b79973-7449-41cf-8709-e8109aa31cd2\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-8ddjc" Oct 30 17:00:58.569742 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:58.569717 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16b79973-7449-41cf-8709-e8109aa31cd2-workload-certs\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-8ddjc\" (UID: \"16b79973-7449-41cf-8709-e8109aa31cd2\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-8ddjc" Oct 30 17:00:58.570413 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:58.570383 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16b79973-7449-41cf-8709-e8109aa31cd2-istiod-ca-cert\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-8ddjc\" (UID: \"16b79973-7449-41cf-8709-e8109aa31cd2\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-8ddjc" Oct 30 17:00:58.571374 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:58.571244 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16b79973-7449-41cf-8709-e8109aa31cd2-istio-envoy\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-8ddjc\" (UID: \"16b79973-7449-41cf-8709-e8109aa31cd2\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-8ddjc" Oct 30 17:00:58.571374 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:58.571280 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16b79973-7449-41cf-8709-e8109aa31cd2-istio-podinfo\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-8ddjc\" (UID: \"16b79973-7449-41cf-8709-e8109aa31cd2\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-8ddjc" Oct 30 17:00:58.591384 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:58.591352 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16b79973-7449-41cf-8709-e8109aa31cd2-istio-token\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-8ddjc\" (UID: \"16b79973-7449-41cf-8709-e8109aa31cd2\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-8ddjc" Oct 30 17:00:58.591973 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:58.591948 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp2l4\" (UniqueName: \"kubernetes.io/projected/16b79973-7449-41cf-8709-e8109aa31cd2-kube-api-access-pp2l4\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-8ddjc\" (UID: \"16b79973-7449-41cf-8709-e8109aa31cd2\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-8ddjc" Oct 30 17:00:58.779536 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:58.779399 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-8ddjc" Oct 30 17:00:59.118026 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:59.117993 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-8ddjc"] Oct 30 17:00:59.124536 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:59.124386 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:00:59.124675 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:59.124537 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:00:59.124675 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:00:59.124589 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:01:00.073420 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:00.073383 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-8ddjc" event={"ID":"16b79973-7449-41cf-8709-e8109aa31cd2","Type":"ContainerStarted","Data":"43de03ac859c91924838516a621c6252fbcbe8e77f117f77f0eb5888cab5cb4a"} Oct 30 17:01:00.073825 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:00.073445 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-8ddjc" event={"ID":"16b79973-7449-41cf-8709-e8109aa31cd2","Type":"ContainerStarted","Data":"62aa58d48188ed62dadcde085277ab110ec732d1bcd501c6c529e4a630f6a6b6"} Oct 30 17:01:00.139831 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:00.139742 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-8ddjc" podStartSLOduration=2.139723399 podStartE2EDuration="2.139723399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:00:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:01:00.136637204 +0000 UTC m=+2287.363812303" watchObservedRunningTime="2026-10-30 17:01:00.139723399 +0000 UTC m=+2287.366898501" Oct 30 17:01:00.352523 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:01:00.352417 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 17:01:00.455996 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:00.455966 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn"] Oct 30 17:01:00.460640 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:00.460619 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn" Oct 30 17:01:00.483805 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:00.480803 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-abda73-openshift-default-dockercfg-wwgz9\"" Oct 30 17:01:00.495946 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:00.495913 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn"] Oct 30 17:01:00.498992 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:00.498966 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43ebbaf3-58f1-4f88-99d2-171e4ca824c6-credential-socket\") pod \"e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn\" (UID: \"43ebbaf3-58f1-4f88-99d2-171e4ca824c6\") " pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn" Oct 30 17:01:00.499109 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:00.498998 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43ebbaf3-58f1-4f88-99d2-171e4ca824c6-istio-token\") pod \"e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn\" (UID: \"43ebbaf3-58f1-4f88-99d2-171e4ca824c6\") " pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn" Oct 30 17:01:00.499109 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:00.499038 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43ebbaf3-58f1-4f88-99d2-171e4ca824c6-istio-envoy\") pod \"e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn\" (UID: \"43ebbaf3-58f1-4f88-99d2-171e4ca824c6\") " pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn" Oct 30 17:01:00.499233 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:00.499144 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43ebbaf3-58f1-4f88-99d2-171e4ca824c6-workload-certs\") pod \"e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn\" (UID: \"43ebbaf3-58f1-4f88-99d2-171e4ca824c6\") " pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn" Oct 30 17:01:00.499233 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:00.499190 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43ebbaf3-58f1-4f88-99d2-171e4ca824c6-istiod-ca-cert\") pod \"e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn\" (UID: \"43ebbaf3-58f1-4f88-99d2-171e4ca824c6\") " pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn" Oct 30 17:01:00.499342 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:00.499256 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43ebbaf3-58f1-4f88-99d2-171e4ca824c6-istio-podinfo\") pod \"e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn\" (UID: \"43ebbaf3-58f1-4f88-99d2-171e4ca824c6\") " pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn" Oct 30 17:01:00.499342 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:00.499307 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43ebbaf3-58f1-4f88-99d2-171e4ca824c6-workload-socket\") pod \"e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn\" (UID: \"43ebbaf3-58f1-4f88-99d2-171e4ca824c6\") " pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn" Oct 30 17:01:00.499342 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:00.499325 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgffk\" (UniqueName: \"kubernetes.io/projected/43ebbaf3-58f1-4f88-99d2-171e4ca824c6-kube-api-access-dgffk\") pod \"e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn\" (UID: \"43ebbaf3-58f1-4f88-99d2-171e4ca824c6\") " pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn" Oct 30 17:01:00.499499 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:00.499413 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43ebbaf3-58f1-4f88-99d2-171e4ca824c6-istio-data\") pod \"e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn\" (UID: \"43ebbaf3-58f1-4f88-99d2-171e4ca824c6\") " pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn" Oct 30 17:01:00.600695 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:00.600660 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43ebbaf3-58f1-4f88-99d2-171e4ca824c6-istio-podinfo\") pod \"e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn\" (UID: \"43ebbaf3-58f1-4f88-99d2-171e4ca824c6\") " pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn" Oct 30 17:01:00.600901 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:00.600755 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43ebbaf3-58f1-4f88-99d2-171e4ca824c6-workload-socket\") pod \"e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn\" (UID: \"43ebbaf3-58f1-4f88-99d2-171e4ca824c6\") " pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn" Oct 30 17:01:00.600901 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:00.600823 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgffk\" (UniqueName: \"kubernetes.io/projected/43ebbaf3-58f1-4f88-99d2-171e4ca824c6-kube-api-access-dgffk\") pod \"e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn\" (UID: \"43ebbaf3-58f1-4f88-99d2-171e4ca824c6\") " pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn" Oct 30 17:01:00.601021 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:00.600915 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43ebbaf3-58f1-4f88-99d2-171e4ca824c6-istio-data\") pod \"e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn\" (UID: \"43ebbaf3-58f1-4f88-99d2-171e4ca824c6\") " pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn" Oct 30 17:01:00.601063 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:00.601018 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43ebbaf3-58f1-4f88-99d2-171e4ca824c6-credential-socket\") pod \"e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn\" (UID: \"43ebbaf3-58f1-4f88-99d2-171e4ca824c6\") " pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn" Oct 30 17:01:00.601063 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:00.601043 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43ebbaf3-58f1-4f88-99d2-171e4ca824c6-istio-token\") pod \"e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn\" (UID: \"43ebbaf3-58f1-4f88-99d2-171e4ca824c6\") " pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn" Oct 30 17:01:00.601220 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:00.601188 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43ebbaf3-58f1-4f88-99d2-171e4ca824c6-istio-envoy\") pod \"e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn\" (UID: \"43ebbaf3-58f1-4f88-99d2-171e4ca824c6\") " pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn" Oct 30 17:01:00.601285 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:00.601255 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43ebbaf3-58f1-4f88-99d2-171e4ca824c6-workload-certs\") pod \"e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn\" (UID: \"43ebbaf3-58f1-4f88-99d2-171e4ca824c6\") " pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn" Oct 30 17:01:00.601285 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:00.601264 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43ebbaf3-58f1-4f88-99d2-171e4ca824c6-workload-socket\") pod \"e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn\" (UID: \"43ebbaf3-58f1-4f88-99d2-171e4ca824c6\") " pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn" Oct 30 17:01:00.601396 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:00.601280 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43ebbaf3-58f1-4f88-99d2-171e4ca824c6-istio-data\") pod \"e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn\" (UID: \"43ebbaf3-58f1-4f88-99d2-171e4ca824c6\") " pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn" Oct 30 17:01:00.601396 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:00.601302 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43ebbaf3-58f1-4f88-99d2-171e4ca824c6-istiod-ca-cert\") pod \"e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn\" (UID: \"43ebbaf3-58f1-4f88-99d2-171e4ca824c6\") " pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn" Oct 30 17:01:00.601396 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:00.601340 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43ebbaf3-58f1-4f88-99d2-171e4ca824c6-credential-socket\") pod \"e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn\" (UID: \"43ebbaf3-58f1-4f88-99d2-171e4ca824c6\") " pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn" Oct 30 17:01:00.602238 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:00.602206 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43ebbaf3-58f1-4f88-99d2-171e4ca824c6-workload-certs\") pod \"e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn\" (UID: \"43ebbaf3-58f1-4f88-99d2-171e4ca824c6\") " pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn" Oct 30 17:01:00.602996 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:00.602942 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43ebbaf3-58f1-4f88-99d2-171e4ca824c6-istiod-ca-cert\") pod \"e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn\" (UID: \"43ebbaf3-58f1-4f88-99d2-171e4ca824c6\") " pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn" Oct 30 17:01:00.619562 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:00.619533 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43ebbaf3-58f1-4f88-99d2-171e4ca824c6-istio-podinfo\") pod \"e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn\" (UID: \"43ebbaf3-58f1-4f88-99d2-171e4ca824c6\") " pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn" Oct 30 17:01:00.621141 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:00.621118 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43ebbaf3-58f1-4f88-99d2-171e4ca824c6-istio-envoy\") pod \"e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn\" (UID: \"43ebbaf3-58f1-4f88-99d2-171e4ca824c6\") " pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn" Oct 30 17:01:00.626042 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:00.626021 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43ebbaf3-58f1-4f88-99d2-171e4ca824c6-istio-token\") pod \"e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn\" (UID: \"43ebbaf3-58f1-4f88-99d2-171e4ca824c6\") " pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn" Oct 30 17:01:00.628075 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:00.628013 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgffk\" (UniqueName: \"kubernetes.io/projected/43ebbaf3-58f1-4f88-99d2-171e4ca824c6-kube-api-access-dgffk\") pod \"e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn\" (UID: \"43ebbaf3-58f1-4f88-99d2-171e4ca824c6\") " pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn" Oct 30 17:01:00.781182 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:00.781149 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-8ddjc" Oct 30 17:01:00.781746 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:00.781723 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn" Oct 30 17:01:00.811767 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:00.811739 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-8ddjc" Oct 30 17:01:01.080538 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:01.080507 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-8ddjc" Oct 30 17:01:01.081549 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:01.081530 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-8ddjc" Oct 30 17:01:01.168047 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:01:01.167999 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43ebbaf3_58f1_4f88_99d2_171e4ca824c6.slice/crio-742910dc22cadde3d6ae915d69499747946cef283825f44c8edb4cff5af71d88 WatchSource:0}: Error finding container 742910dc22cadde3d6ae915d69499747946cef283825f44c8edb4cff5af71d88: Status 404 returned error can't find the container with id 742910dc22cadde3d6ae915d69499747946cef283825f44c8edb4cff5af71d88 Oct 30 17:01:01.175740 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:01.175707 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn"] Oct 30 17:01:01.182381 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:01.182326 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:01:01.182542 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:01.182438 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:01:01.182542 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:01.182483 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:01:02.089563 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:02.089519 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn" event={"ID":"43ebbaf3-58f1-4f88-99d2-171e4ca824c6","Type":"ContainerStarted","Data":"fdb775980abbbda51861f2242ffb1f4e8423cd2cc5e89b2aceeadc035555db07"} Oct 30 17:01:02.090032 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:02.089588 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn" event={"ID":"43ebbaf3-58f1-4f88-99d2-171e4ca824c6","Type":"ContainerStarted","Data":"742910dc22cadde3d6ae915d69499747946cef283825f44c8edb4cff5af71d88"} Oct 30 17:01:02.138008 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:02.137947 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn" podStartSLOduration=2.137928326 podStartE2EDuration="2.137928326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:01:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:01:02.135891962 +0000 UTC m=+2289.363067061" watchObservedRunningTime="2026-10-30 17:01:02.137928326 +0000 UTC m=+2289.365103424" Oct 30 17:01:02.782985 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:02.782956 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn" Oct 30 17:01:02.788672 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:02.788648 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn" Oct 30 17:01:03.096771 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:03.096687 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn" Oct 30 17:01:03.097603 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:03.097575 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn" Oct 30 17:01:07.720309 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:07.720229 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 17:01:08.036147 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:08.036051 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2d14cbb-44s9f"] Oct 30 17:01:08.047959 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:08.047924 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2d14cbb-44s9f"] Oct 30 17:01:08.933305 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:08.933274 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d340ae1-f1e3-4b62-a1b2-8555489dc185" path="/var/lib/kubelet/pods/3d340ae1-f1e3-4b62-a1b2-8555489dc185/volumes" Oct 30 17:01:16.258763 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:16.258727 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2"] Oct 30 17:01:16.267831 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:16.264476 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2" Oct 30 17:01:16.282834 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:16.282809 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-13c495-openshift-default-dockercfg-pnbrr\"" Oct 30 17:01:16.343796 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:16.343747 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2"] Oct 30 17:01:16.437770 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:16.437726 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98af0974-af46-432d-a622-4279717ff916-istio-data\") pod \"e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2\" (UID: \"98af0974-af46-432d-a622-4279717ff916\") " pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2" Oct 30 17:01:16.441167 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:16.441120 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98af0974-af46-432d-a622-4279717ff916-credential-socket\") pod \"e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2\" (UID: \"98af0974-af46-432d-a622-4279717ff916\") " pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2" Oct 30 17:01:16.441431 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:16.441410 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98af0974-af46-432d-a622-4279717ff916-workload-certs\") pod \"e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2\" (UID: \"98af0974-af46-432d-a622-4279717ff916\") " pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2" Oct 30 17:01:16.441553 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:16.441538 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98af0974-af46-432d-a622-4279717ff916-istiod-ca-cert\") pod \"e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2\" (UID: \"98af0974-af46-432d-a622-4279717ff916\") " pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2" Oct 30 17:01:16.441704 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:16.441689 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98af0974-af46-432d-a622-4279717ff916-istio-token\") pod \"e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2\" (UID: \"98af0974-af46-432d-a622-4279717ff916\") " pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2" Oct 30 17:01:16.442400 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:16.441925 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98af0974-af46-432d-a622-4279717ff916-istio-envoy\") pod \"e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2\" (UID: \"98af0974-af46-432d-a622-4279717ff916\") " pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2" Oct 30 17:01:16.442710 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:16.442690 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98af0974-af46-432d-a622-4279717ff916-istio-podinfo\") pod \"e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2\" (UID: \"98af0974-af46-432d-a622-4279717ff916\") " pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2" Oct 30 17:01:16.453968 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:16.453932 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98af0974-af46-432d-a622-4279717ff916-workload-socket\") pod \"e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2\" (UID: \"98af0974-af46-432d-a622-4279717ff916\") " pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2" Oct 30 17:01:16.454221 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:16.454204 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4ds7\" (UniqueName: \"kubernetes.io/projected/98af0974-af46-432d-a622-4279717ff916-kube-api-access-w4ds7\") pod \"e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2\" (UID: \"98af0974-af46-432d-a622-4279717ff916\") " pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2" Oct 30 17:01:16.558881 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:16.558528 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98af0974-af46-432d-a622-4279717ff916-istio-data\") pod \"e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2\" (UID: \"98af0974-af46-432d-a622-4279717ff916\") " pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2" Oct 30 17:01:16.558881 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:16.558583 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98af0974-af46-432d-a622-4279717ff916-credential-socket\") pod \"e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2\" (UID: \"98af0974-af46-432d-a622-4279717ff916\") " pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2" Oct 30 17:01:16.558881 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:16.558643 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98af0974-af46-432d-a622-4279717ff916-workload-certs\") pod \"e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2\" (UID: \"98af0974-af46-432d-a622-4279717ff916\") " pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2" Oct 30 17:01:16.558881 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:16.558669 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98af0974-af46-432d-a622-4279717ff916-istiod-ca-cert\") pod \"e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2\" (UID: \"98af0974-af46-432d-a622-4279717ff916\") " pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2" Oct 30 17:01:16.558881 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:16.558748 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98af0974-af46-432d-a622-4279717ff916-istio-token\") pod \"e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2\" (UID: \"98af0974-af46-432d-a622-4279717ff916\") " pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2" Oct 30 17:01:16.564835 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:16.559482 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98af0974-af46-432d-a622-4279717ff916-istio-data\") pod \"e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2\" (UID: \"98af0974-af46-432d-a622-4279717ff916\") " pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2" Oct 30 17:01:16.564835 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:16.559623 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98af0974-af46-432d-a622-4279717ff916-istio-envoy\") pod \"e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2\" (UID: \"98af0974-af46-432d-a622-4279717ff916\") " pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2" Oct 30 17:01:16.564835 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:16.559653 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98af0974-af46-432d-a622-4279717ff916-istio-podinfo\") pod \"e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2\" (UID: \"98af0974-af46-432d-a622-4279717ff916\") " pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2" Oct 30 17:01:16.564835 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:16.559687 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98af0974-af46-432d-a622-4279717ff916-workload-socket\") pod \"e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2\" (UID: \"98af0974-af46-432d-a622-4279717ff916\") " pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2" Oct 30 17:01:16.564835 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:16.559721 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4ds7\" (UniqueName: \"kubernetes.io/projected/98af0974-af46-432d-a622-4279717ff916-kube-api-access-w4ds7\") pod \"e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2\" (UID: \"98af0974-af46-432d-a622-4279717ff916\") " pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2" Oct 30 17:01:16.564835 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:16.560349 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98af0974-af46-432d-a622-4279717ff916-credential-socket\") pod \"e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2\" (UID: \"98af0974-af46-432d-a622-4279717ff916\") " pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2" Oct 30 17:01:16.564835 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:16.560562 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98af0974-af46-432d-a622-4279717ff916-workload-certs\") pod \"e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2\" (UID: \"98af0974-af46-432d-a622-4279717ff916\") " pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2" Oct 30 17:01:16.564835 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:16.561162 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98af0974-af46-432d-a622-4279717ff916-istiod-ca-cert\") pod \"e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2\" (UID: \"98af0974-af46-432d-a622-4279717ff916\") " pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2" Oct 30 17:01:16.566075 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:16.566051 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98af0974-af46-432d-a622-4279717ff916-workload-socket\") pod \"e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2\" (UID: \"98af0974-af46-432d-a622-4279717ff916\") " pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2" Oct 30 17:01:16.575891 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:16.571529 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98af0974-af46-432d-a622-4279717ff916-istio-podinfo\") pod \"e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2\" (UID: \"98af0974-af46-432d-a622-4279717ff916\") " pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2" Oct 30 17:01:16.587637 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:16.586324 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98af0974-af46-432d-a622-4279717ff916-istio-envoy\") pod \"e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2\" (UID: \"98af0974-af46-432d-a622-4279717ff916\") " pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2" Oct 30 17:01:16.591473 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:16.591417 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98af0974-af46-432d-a622-4279717ff916-istio-token\") pod \"e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2\" (UID: \"98af0974-af46-432d-a622-4279717ff916\") " pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2" Oct 30 17:01:16.592247 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:16.592213 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4ds7\" (UniqueName: \"kubernetes.io/projected/98af0974-af46-432d-a622-4279717ff916-kube-api-access-w4ds7\") pod \"e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2\" (UID: \"98af0974-af46-432d-a622-4279717ff916\") " pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2" Oct 30 17:01:16.889365 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:16.889281 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2" Oct 30 17:01:17.257242 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:17.257212 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2"] Oct 30 17:01:17.257583 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:01:17.257559 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98af0974_af46_432d_a622_4279717ff916.slice/crio-e4109deafca5d40be6679454f707462d4b290aae0d7604a89315b43b6a8a0c4c WatchSource:0}: Error finding container e4109deafca5d40be6679454f707462d4b290aae0d7604a89315b43b6a8a0c4c: Status 404 returned error can't find the container with id e4109deafca5d40be6679454f707462d4b290aae0d7604a89315b43b6a8a0c4c Oct 30 17:01:17.260726 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:17.260686 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:01:17.261135 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:17.260775 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:01:17.261135 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:17.260863 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:01:18.254999 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:18.254963 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2" event={"ID":"98af0974-af46-432d-a622-4279717ff916","Type":"ContainerStarted","Data":"8ef450d4c2c76eaad33b7c06f4f13933980135fe562eeff48190e8642a734f56"} Oct 30 17:01:18.255190 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:18.255017 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2" event={"ID":"98af0974-af46-432d-a622-4279717ff916","Type":"ContainerStarted","Data":"e4109deafca5d40be6679454f707462d4b290aae0d7604a89315b43b6a8a0c4c"} Oct 30 17:01:18.312361 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:18.312301 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2" podStartSLOduration=2.312287136 podStartE2EDuration="2.312287136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:01:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:01:18.311083015 +0000 UTC m=+2305.538258114" watchObservedRunningTime="2026-10-30 17:01:18.312287136 +0000 UTC m=+2305.539462235" Oct 30 17:01:18.889609 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:18.889576 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2" Oct 30 17:01:18.895722 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:18.895697 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2" Oct 30 17:01:19.265476 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:19.265446 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2" Oct 30 17:01:19.266506 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:19.266481 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2" Oct 30 17:01:30.342235 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:01:30.342192 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 17:01:44.507626 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:44.507586 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 17:01:45.269434 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:45.269399 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w"] Oct 30 17:01:45.279018 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:45.278992 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w" Oct 30 17:01:45.295258 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:45.294805 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-5c6b3c-openshift-default-dockercfg-zhh7l\"" Oct 30 17:01:45.315247 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:45.315213 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w"] Oct 30 17:01:45.339088 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:45.339059 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e21f846-6c01-496f-998e-ccfe2a18d5cd-istio-envoy\") pod \"e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w\" (UID: \"8e21f846-6c01-496f-998e-ccfe2a18d5cd\") " pod="openshift-ingress/e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w" Oct 30 17:01:45.339280 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:45.339101 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e21f846-6c01-496f-998e-ccfe2a18d5cd-istio-token\") pod \"e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w\" (UID: \"8e21f846-6c01-496f-998e-ccfe2a18d5cd\") " pod="openshift-ingress/e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w" Oct 30 17:01:45.339280 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:45.339129 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e21f846-6c01-496f-998e-ccfe2a18d5cd-istio-data\") pod \"e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w\" (UID: \"8e21f846-6c01-496f-998e-ccfe2a18d5cd\") " pod="openshift-ingress/e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w" Oct 30 17:01:45.339280 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:45.339234 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e21f846-6c01-496f-998e-ccfe2a18d5cd-credential-socket\") pod \"e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w\" (UID: \"8e21f846-6c01-496f-998e-ccfe2a18d5cd\") " pod="openshift-ingress/e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w" Oct 30 17:01:45.339280 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:45.339272 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e21f846-6c01-496f-998e-ccfe2a18d5cd-istiod-ca-cert\") pod \"e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w\" (UID: \"8e21f846-6c01-496f-998e-ccfe2a18d5cd\") " pod="openshift-ingress/e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w" Oct 30 17:01:45.339496 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:45.339340 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e21f846-6c01-496f-998e-ccfe2a18d5cd-workload-socket\") pod \"e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w\" (UID: \"8e21f846-6c01-496f-998e-ccfe2a18d5cd\") " pod="openshift-ingress/e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w" Oct 30 17:01:45.339496 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:45.339375 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e21f846-6c01-496f-998e-ccfe2a18d5cd-istio-podinfo\") pod \"e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w\" (UID: \"8e21f846-6c01-496f-998e-ccfe2a18d5cd\") " pod="openshift-ingress/e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w" Oct 30 17:01:45.339496 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:45.339404 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz7cb\" (UniqueName: \"kubernetes.io/projected/8e21f846-6c01-496f-998e-ccfe2a18d5cd-kube-api-access-qz7cb\") pod \"e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w\" (UID: \"8e21f846-6c01-496f-998e-ccfe2a18d5cd\") " pod="openshift-ingress/e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w" Oct 30 17:01:45.339496 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:45.339447 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e21f846-6c01-496f-998e-ccfe2a18d5cd-workload-certs\") pod \"e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w\" (UID: \"8e21f846-6c01-496f-998e-ccfe2a18d5cd\") " pod="openshift-ingress/e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w" Oct 30 17:01:45.445591 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:45.445546 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e21f846-6c01-496f-998e-ccfe2a18d5cd-istio-envoy\") pod \"e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w\" (UID: \"8e21f846-6c01-496f-998e-ccfe2a18d5cd\") " pod="openshift-ingress/e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w" Oct 30 17:01:45.445843 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:45.445618 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e21f846-6c01-496f-998e-ccfe2a18d5cd-istio-token\") pod \"e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w\" (UID: \"8e21f846-6c01-496f-998e-ccfe2a18d5cd\") " pod="openshift-ingress/e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w" Oct 30 17:01:45.445843 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:45.445648 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e21f846-6c01-496f-998e-ccfe2a18d5cd-istio-data\") pod \"e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w\" (UID: \"8e21f846-6c01-496f-998e-ccfe2a18d5cd\") " pod="openshift-ingress/e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w" Oct 30 17:01:45.445843 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:45.445693 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e21f846-6c01-496f-998e-ccfe2a18d5cd-credential-socket\") pod \"e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w\" (UID: \"8e21f846-6c01-496f-998e-ccfe2a18d5cd\") " pod="openshift-ingress/e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w" Oct 30 17:01:45.445843 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:45.445736 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e21f846-6c01-496f-998e-ccfe2a18d5cd-istiod-ca-cert\") pod \"e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w\" (UID: \"8e21f846-6c01-496f-998e-ccfe2a18d5cd\") " pod="openshift-ingress/e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w" Oct 30 17:01:45.445843 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:45.445803 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e21f846-6c01-496f-998e-ccfe2a18d5cd-workload-socket\") pod \"e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w\" (UID: \"8e21f846-6c01-496f-998e-ccfe2a18d5cd\") " pod="openshift-ingress/e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w" Oct 30 17:01:45.446134 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:45.445853 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e21f846-6c01-496f-998e-ccfe2a18d5cd-istio-podinfo\") pod \"e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w\" (UID: \"8e21f846-6c01-496f-998e-ccfe2a18d5cd\") " pod="openshift-ingress/e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w" Oct 30 17:01:45.446134 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:45.445898 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz7cb\" (UniqueName: \"kubernetes.io/projected/8e21f846-6c01-496f-998e-ccfe2a18d5cd-kube-api-access-qz7cb\") pod \"e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w\" (UID: \"8e21f846-6c01-496f-998e-ccfe2a18d5cd\") " pod="openshift-ingress/e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w" Oct 30 17:01:45.446134 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:45.445966 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e21f846-6c01-496f-998e-ccfe2a18d5cd-workload-certs\") pod \"e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w\" (UID: \"8e21f846-6c01-496f-998e-ccfe2a18d5cd\") " pod="openshift-ingress/e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w" Oct 30 17:01:45.446454 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:45.446432 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e21f846-6c01-496f-998e-ccfe2a18d5cd-workload-certs\") pod \"e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w\" (UID: \"8e21f846-6c01-496f-998e-ccfe2a18d5cd\") " pod="openshift-ingress/e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w" Oct 30 17:01:45.447402 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:45.447376 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e21f846-6c01-496f-998e-ccfe2a18d5cd-istiod-ca-cert\") pod \"e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w\" (UID: \"8e21f846-6c01-496f-998e-ccfe2a18d5cd\") " pod="openshift-ingress/e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w" Oct 30 17:01:45.448080 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:45.448059 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e21f846-6c01-496f-998e-ccfe2a18d5cd-istio-data\") pod \"e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w\" (UID: \"8e21f846-6c01-496f-998e-ccfe2a18d5cd\") " pod="openshift-ingress/e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w" Oct 30 17:01:45.448436 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:45.448420 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e21f846-6c01-496f-998e-ccfe2a18d5cd-credential-socket\") pod \"e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w\" (UID: \"8e21f846-6c01-496f-998e-ccfe2a18d5cd\") " pod="openshift-ingress/e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w" Oct 30 17:01:45.448752 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:45.448736 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e21f846-6c01-496f-998e-ccfe2a18d5cd-workload-socket\") pod \"e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w\" (UID: \"8e21f846-6c01-496f-998e-ccfe2a18d5cd\") " pod="openshift-ingress/e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w" Oct 30 17:01:45.502938 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:45.474587 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e21f846-6c01-496f-998e-ccfe2a18d5cd-istio-envoy\") pod \"e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w\" (UID: \"8e21f846-6c01-496f-998e-ccfe2a18d5cd\") " pod="openshift-ingress/e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w" Oct 30 17:01:45.502938 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:45.494281 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e21f846-6c01-496f-998e-ccfe2a18d5cd-istio-token\") pod \"e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w\" (UID: \"8e21f846-6c01-496f-998e-ccfe2a18d5cd\") " pod="openshift-ingress/e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w" Oct 30 17:01:45.502938 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:45.496145 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz7cb\" (UniqueName: \"kubernetes.io/projected/8e21f846-6c01-496f-998e-ccfe2a18d5cd-kube-api-access-qz7cb\") pod \"e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w\" (UID: \"8e21f846-6c01-496f-998e-ccfe2a18d5cd\") " pod="openshift-ingress/e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w" Oct 30 17:01:45.512869 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:45.510392 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e21f846-6c01-496f-998e-ccfe2a18d5cd-istio-podinfo\") pod \"e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w\" (UID: \"8e21f846-6c01-496f-998e-ccfe2a18d5cd\") " pod="openshift-ingress/e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w" Oct 30 17:01:45.604704 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:45.604609 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w" Oct 30 17:01:45.934690 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:45.934666 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w"] Oct 30 17:01:45.936457 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:01:45.936431 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e21f846_6c01_496f_998e_ccfe2a18d5cd.slice/crio-3498d6c1599541a6286792da99b9fdd6b7b48e52c2b1ecf42abbd6f038a27025 WatchSource:0}: Error finding container 3498d6c1599541a6286792da99b9fdd6b7b48e52c2b1ecf42abbd6f038a27025: Status 404 returned error can't find the container with id 3498d6c1599541a6286792da99b9fdd6b7b48e52c2b1ecf42abbd6f038a27025 Oct 30 17:01:45.942571 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:45.939693 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:01:45.942571 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:45.939805 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:01:45.942571 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:45.939846 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:01:46.536631 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:46.536592 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w" event={"ID":"8e21f846-6c01-496f-998e-ccfe2a18d5cd","Type":"ContainerStarted","Data":"5df355cf8894c3cd55e3c567e4a600995b29412c5e2b8a1ce4dc7165513c8b38"} Oct 30 17:01:46.537194 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:46.536670 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w" event={"ID":"8e21f846-6c01-496f-998e-ccfe2a18d5cd","Type":"ContainerStarted","Data":"3498d6c1599541a6286792da99b9fdd6b7b48e52c2b1ecf42abbd6f038a27025"} Oct 30 17:01:46.605002 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:46.604950 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w" podStartSLOduration=1.604935432 podStartE2EDuration="1.604935432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:01:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:01:46.592498859 +0000 UTC m=+2333.819673960" watchObservedRunningTime="2026-10-30 17:01:46.604935432 +0000 UTC m=+2333.832110614" Oct 30 17:01:46.605266 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:46.605247 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w" Oct 30 17:01:47.611603 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:47.611569 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w" Oct 30 17:01:48.557298 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:48.557270 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w" Oct 30 17:01:48.560000 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:48.559032 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w" Oct 30 17:01:55.892338 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:55.892180 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz"] Oct 30 17:01:55.897023 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:55.897000 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz" Oct 30 17:01:55.925285 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:55.919199 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz"] Oct 30 17:01:55.925285 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:55.924049 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-5758b7-openshift-default-dockercfg-95dtl\"" Oct 30 17:01:56.016833 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:56.012958 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/defb21aa-d0d6-48b8-ba50-4fcb900bc547-workload-socket\") pod \"e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz\" (UID: \"defb21aa-d0d6-48b8-ba50-4fcb900bc547\") " pod="openshift-ingress/e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz" Oct 30 17:01:56.016833 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:56.013014 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/defb21aa-d0d6-48b8-ba50-4fcb900bc547-istio-data\") pod \"e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz\" (UID: \"defb21aa-d0d6-48b8-ba50-4fcb900bc547\") " pod="openshift-ingress/e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz" Oct 30 17:01:56.016833 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:56.013126 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/defb21aa-d0d6-48b8-ba50-4fcb900bc547-istio-podinfo\") pod \"e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz\" (UID: \"defb21aa-d0d6-48b8-ba50-4fcb900bc547\") " pod="openshift-ingress/e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz" Oct 30 17:01:56.016833 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:56.013155 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvbmc\" (UniqueName: \"kubernetes.io/projected/defb21aa-d0d6-48b8-ba50-4fcb900bc547-kube-api-access-qvbmc\") pod \"e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz\" (UID: \"defb21aa-d0d6-48b8-ba50-4fcb900bc547\") " pod="openshift-ingress/e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz" Oct 30 17:01:56.016833 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:56.013214 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/defb21aa-d0d6-48b8-ba50-4fcb900bc547-istio-envoy\") pod \"e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz\" (UID: \"defb21aa-d0d6-48b8-ba50-4fcb900bc547\") " pod="openshift-ingress/e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz" Oct 30 17:01:56.016833 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:56.013267 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/defb21aa-d0d6-48b8-ba50-4fcb900bc547-istiod-ca-cert\") pod \"e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz\" (UID: \"defb21aa-d0d6-48b8-ba50-4fcb900bc547\") " pod="openshift-ingress/e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz" Oct 30 17:01:56.016833 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:56.013300 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/defb21aa-d0d6-48b8-ba50-4fcb900bc547-credential-socket\") pod \"e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz\" (UID: \"defb21aa-d0d6-48b8-ba50-4fcb900bc547\") " pod="openshift-ingress/e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz" Oct 30 17:01:56.016833 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:56.013354 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/defb21aa-d0d6-48b8-ba50-4fcb900bc547-istio-token\") pod \"e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz\" (UID: \"defb21aa-d0d6-48b8-ba50-4fcb900bc547\") " pod="openshift-ingress/e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz" Oct 30 17:01:56.016833 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:56.013420 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/defb21aa-d0d6-48b8-ba50-4fcb900bc547-workload-certs\") pod \"e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz\" (UID: \"defb21aa-d0d6-48b8-ba50-4fcb900bc547\") " pod="openshift-ingress/e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz" Oct 30 17:01:56.102831 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:56.098876 4493 scope.go:122] "RemoveContainer" containerID="2cc76d05b0df466327eb91ffc64e474a793e39c41b328c66a3787a73f41d4559" Oct 30 17:01:56.123580 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:56.123540 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/defb21aa-d0d6-48b8-ba50-4fcb900bc547-workload-socket\") pod \"e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz\" (UID: \"defb21aa-d0d6-48b8-ba50-4fcb900bc547\") " pod="openshift-ingress/e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz" Oct 30 17:01:56.123818 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:56.123775 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/defb21aa-d0d6-48b8-ba50-4fcb900bc547-istio-data\") pod \"e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz\" (UID: \"defb21aa-d0d6-48b8-ba50-4fcb900bc547\") " pod="openshift-ingress/e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz" Oct 30 17:01:56.124104 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:56.124070 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/defb21aa-d0d6-48b8-ba50-4fcb900bc547-istio-podinfo\") pod \"e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz\" (UID: \"defb21aa-d0d6-48b8-ba50-4fcb900bc547\") " pod="openshift-ingress/e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz" Oct 30 17:01:56.124250 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:56.124233 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvbmc\" (UniqueName: \"kubernetes.io/projected/defb21aa-d0d6-48b8-ba50-4fcb900bc547-kube-api-access-qvbmc\") pod \"e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz\" (UID: \"defb21aa-d0d6-48b8-ba50-4fcb900bc547\") " pod="openshift-ingress/e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz" Oct 30 17:01:56.124423 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:56.124395 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/defb21aa-d0d6-48b8-ba50-4fcb900bc547-istio-envoy\") pod \"e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz\" (UID: \"defb21aa-d0d6-48b8-ba50-4fcb900bc547\") " pod="openshift-ingress/e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz" Oct 30 17:01:56.124595 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:56.124568 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/defb21aa-d0d6-48b8-ba50-4fcb900bc547-istiod-ca-cert\") pod \"e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz\" (UID: \"defb21aa-d0d6-48b8-ba50-4fcb900bc547\") " pod="openshift-ingress/e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz" Oct 30 17:01:56.124737 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:56.124723 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/defb21aa-d0d6-48b8-ba50-4fcb900bc547-credential-socket\") pod \"e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz\" (UID: \"defb21aa-d0d6-48b8-ba50-4fcb900bc547\") " pod="openshift-ingress/e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz" Oct 30 17:01:56.124928 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:56.124898 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/defb21aa-d0d6-48b8-ba50-4fcb900bc547-istio-token\") pod \"e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz\" (UID: \"defb21aa-d0d6-48b8-ba50-4fcb900bc547\") " pod="openshift-ingress/e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz" Oct 30 17:01:56.125123 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:56.125107 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/defb21aa-d0d6-48b8-ba50-4fcb900bc547-workload-certs\") pod \"e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz\" (UID: \"defb21aa-d0d6-48b8-ba50-4fcb900bc547\") " pod="openshift-ingress/e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz" Oct 30 17:01:56.146866 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:56.127506 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/defb21aa-d0d6-48b8-ba50-4fcb900bc547-istiod-ca-cert\") pod \"e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz\" (UID: \"defb21aa-d0d6-48b8-ba50-4fcb900bc547\") " pod="openshift-ingress/e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz" Oct 30 17:01:56.146866 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:56.128509 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/defb21aa-d0d6-48b8-ba50-4fcb900bc547-credential-socket\") pod \"e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz\" (UID: \"defb21aa-d0d6-48b8-ba50-4fcb900bc547\") " pod="openshift-ingress/e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz" Oct 30 17:01:56.146866 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:56.129065 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/defb21aa-d0d6-48b8-ba50-4fcb900bc547-workload-socket\") pod \"e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz\" (UID: \"defb21aa-d0d6-48b8-ba50-4fcb900bc547\") " pod="openshift-ingress/e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz" Oct 30 17:01:56.146866 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:56.129559 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/defb21aa-d0d6-48b8-ba50-4fcb900bc547-istio-data\") pod \"e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz\" (UID: \"defb21aa-d0d6-48b8-ba50-4fcb900bc547\") " pod="openshift-ingress/e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz" Oct 30 17:01:56.153822 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:56.125779 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/defb21aa-d0d6-48b8-ba50-4fcb900bc547-workload-certs\") pod \"e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz\" (UID: \"defb21aa-d0d6-48b8-ba50-4fcb900bc547\") " pod="openshift-ingress/e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz" Oct 30 17:01:56.167809 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:56.167385 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/defb21aa-d0d6-48b8-ba50-4fcb900bc547-istio-podinfo\") pod \"e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz\" (UID: \"defb21aa-d0d6-48b8-ba50-4fcb900bc547\") " pod="openshift-ingress/e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz" Oct 30 17:01:56.186818 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:56.175648 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/defb21aa-d0d6-48b8-ba50-4fcb900bc547-istio-token\") pod \"e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz\" (UID: \"defb21aa-d0d6-48b8-ba50-4fcb900bc547\") " pod="openshift-ingress/e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz" Oct 30 17:01:56.212832 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:56.211052 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/defb21aa-d0d6-48b8-ba50-4fcb900bc547-istio-envoy\") pod \"e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz\" (UID: \"defb21aa-d0d6-48b8-ba50-4fcb900bc547\") " pod="openshift-ingress/e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz" Oct 30 17:01:56.212832 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:56.212101 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvbmc\" (UniqueName: \"kubernetes.io/projected/defb21aa-d0d6-48b8-ba50-4fcb900bc547-kube-api-access-qvbmc\") pod \"e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz\" (UID: \"defb21aa-d0d6-48b8-ba50-4fcb900bc547\") " pod="openshift-ingress/e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz" Oct 30 17:01:56.232810 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:56.232748 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz" Oct 30 17:01:56.590140 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:56.590115 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz"] Oct 30 17:01:56.591404 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:01:56.591376 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddefb21aa_d0d6_48b8_ba50_4fcb900bc547.slice/crio-eb3eb4494aaf96fe565eead77279e47a13663f22c79358099a6668c3b1654470 WatchSource:0}: Error finding container eb3eb4494aaf96fe565eead77279e47a13663f22c79358099a6668c3b1654470: Status 404 returned error can't find the container with id eb3eb4494aaf96fe565eead77279e47a13663f22c79358099a6668c3b1654470 Oct 30 17:01:56.594202 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:56.594168 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:01:56.594287 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:56.594234 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:01:56.594287 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:56.594262 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:01:56.627554 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:56.627513 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz" event={"ID":"defb21aa-d0d6-48b8-ba50-4fcb900bc547","Type":"ContainerStarted","Data":"eb3eb4494aaf96fe565eead77279e47a13663f22c79358099a6668c3b1654470"} Oct 30 17:01:57.636212 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:57.636170 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz" event={"ID":"defb21aa-d0d6-48b8-ba50-4fcb900bc547","Type":"ContainerStarted","Data":"0f67a2519bbe7709e504d80e7a989283a61519544a02432f59742bce309ac05b"} Oct 30 17:01:57.686377 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:57.686320 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz" podStartSLOduration=2.686301892 podStartE2EDuration="2.686301892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:01:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:01:57.682233721 +0000 UTC m=+2344.909408815" watchObservedRunningTime="2026-10-30 17:01:57.686301892 +0000 UTC m=+2344.913476994" Oct 30 17:01:58.234686 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:58.234654 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz" Oct 30 17:01:58.241340 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:58.241314 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz" Oct 30 17:01:58.644319 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:58.644284 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz" Oct 30 17:01:58.645373 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:01:58.645354 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz" Oct 30 17:02:00.342055 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:02:00.342012 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 17:02:13.710089 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:13.710040 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-jj9m9"] Oct 30 17:02:13.734167 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:13.734135 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-jj9m9" Oct 30 17:02:13.756729 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:13.756699 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-jj9m9"] Oct 30 17:02:13.809325 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:13.807815 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35bcc211-d8a1-482e-a44d-82870293177f-istio-podinfo\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-jj9m9\" (UID: \"35bcc211-d8a1-482e-a44d-82870293177f\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-jj9m9" Oct 30 17:02:13.809325 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:13.807923 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clrnm\" (UniqueName: \"kubernetes.io/projected/35bcc211-d8a1-482e-a44d-82870293177f-kube-api-access-clrnm\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-jj9m9\" (UID: \"35bcc211-d8a1-482e-a44d-82870293177f\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-jj9m9" Oct 30 17:02:13.809325 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:13.808029 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35bcc211-d8a1-482e-a44d-82870293177f-istio-data\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-jj9m9\" (UID: \"35bcc211-d8a1-482e-a44d-82870293177f\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-jj9m9" Oct 30 17:02:13.809325 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:13.808102 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35bcc211-d8a1-482e-a44d-82870293177f-workload-certs\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-jj9m9\" (UID: \"35bcc211-d8a1-482e-a44d-82870293177f\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-jj9m9" Oct 30 17:02:13.809325 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:13.808161 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35bcc211-d8a1-482e-a44d-82870293177f-workload-socket\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-jj9m9\" (UID: \"35bcc211-d8a1-482e-a44d-82870293177f\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-jj9m9" Oct 30 17:02:13.809325 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:13.808266 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35bcc211-d8a1-482e-a44d-82870293177f-credential-socket\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-jj9m9\" (UID: \"35bcc211-d8a1-482e-a44d-82870293177f\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-jj9m9" Oct 30 17:02:13.809325 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:13.808296 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35bcc211-d8a1-482e-a44d-82870293177f-istio-envoy\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-jj9m9\" (UID: \"35bcc211-d8a1-482e-a44d-82870293177f\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-jj9m9" Oct 30 17:02:13.809325 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:13.808414 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35bcc211-d8a1-482e-a44d-82870293177f-istio-token\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-jj9m9\" (UID: \"35bcc211-d8a1-482e-a44d-82870293177f\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-jj9m9" Oct 30 17:02:13.809325 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:13.808542 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35bcc211-d8a1-482e-a44d-82870293177f-istiod-ca-cert\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-jj9m9\" (UID: \"35bcc211-d8a1-482e-a44d-82870293177f\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-jj9m9" Oct 30 17:02:14.039517 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:13.909817 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35bcc211-d8a1-482e-a44d-82870293177f-istiod-ca-cert\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-jj9m9\" (UID: \"35bcc211-d8a1-482e-a44d-82870293177f\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-jj9m9" Oct 30 17:02:14.039517 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:13.909890 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35bcc211-d8a1-482e-a44d-82870293177f-istio-podinfo\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-jj9m9\" (UID: \"35bcc211-d8a1-482e-a44d-82870293177f\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-jj9m9" Oct 30 17:02:14.039517 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:13.909938 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clrnm\" (UniqueName: \"kubernetes.io/projected/35bcc211-d8a1-482e-a44d-82870293177f-kube-api-access-clrnm\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-jj9m9\" (UID: \"35bcc211-d8a1-482e-a44d-82870293177f\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-jj9m9" Oct 30 17:02:14.039517 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:13.909991 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35bcc211-d8a1-482e-a44d-82870293177f-istio-data\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-jj9m9\" (UID: \"35bcc211-d8a1-482e-a44d-82870293177f\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-jj9m9" Oct 30 17:02:14.039517 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:13.910056 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35bcc211-d8a1-482e-a44d-82870293177f-workload-certs\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-jj9m9\" (UID: \"35bcc211-d8a1-482e-a44d-82870293177f\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-jj9m9" Oct 30 17:02:14.039517 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:13.910109 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35bcc211-d8a1-482e-a44d-82870293177f-workload-socket\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-jj9m9\" (UID: \"35bcc211-d8a1-482e-a44d-82870293177f\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-jj9m9" Oct 30 17:02:14.039517 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:13.910149 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35bcc211-d8a1-482e-a44d-82870293177f-credential-socket\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-jj9m9\" (UID: \"35bcc211-d8a1-482e-a44d-82870293177f\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-jj9m9" Oct 30 17:02:14.039517 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:13.910173 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35bcc211-d8a1-482e-a44d-82870293177f-istio-envoy\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-jj9m9\" (UID: \"35bcc211-d8a1-482e-a44d-82870293177f\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-jj9m9" Oct 30 17:02:14.039517 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:13.910237 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35bcc211-d8a1-482e-a44d-82870293177f-istio-token\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-jj9m9\" (UID: \"35bcc211-d8a1-482e-a44d-82870293177f\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-jj9m9" Oct 30 17:02:14.039517 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:13.910640 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35bcc211-d8a1-482e-a44d-82870293177f-istiod-ca-cert\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-jj9m9\" (UID: \"35bcc211-d8a1-482e-a44d-82870293177f\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-jj9m9" Oct 30 17:02:14.039517 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:13.910693 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35bcc211-d8a1-482e-a44d-82870293177f-workload-certs\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-jj9m9\" (UID: \"35bcc211-d8a1-482e-a44d-82870293177f\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-jj9m9" Oct 30 17:02:14.039517 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:13.910850 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35bcc211-d8a1-482e-a44d-82870293177f-workload-socket\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-jj9m9\" (UID: \"35bcc211-d8a1-482e-a44d-82870293177f\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-jj9m9" Oct 30 17:02:14.039517 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:13.911025 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35bcc211-d8a1-482e-a44d-82870293177f-istio-data\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-jj9m9\" (UID: \"35bcc211-d8a1-482e-a44d-82870293177f\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-jj9m9" Oct 30 17:02:14.039517 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:13.911237 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35bcc211-d8a1-482e-a44d-82870293177f-credential-socket\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-jj9m9\" (UID: \"35bcc211-d8a1-482e-a44d-82870293177f\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-jj9m9" Oct 30 17:02:14.039517 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:13.913509 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35bcc211-d8a1-482e-a44d-82870293177f-istio-podinfo\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-jj9m9\" (UID: \"35bcc211-d8a1-482e-a44d-82870293177f\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-jj9m9" Oct 30 17:02:14.039517 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:13.913709 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35bcc211-d8a1-482e-a44d-82870293177f-istio-envoy\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-jj9m9\" (UID: \"35bcc211-d8a1-482e-a44d-82870293177f\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-jj9m9" Oct 30 17:02:14.039517 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:13.933934 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35bcc211-d8a1-482e-a44d-82870293177f-istio-token\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-jj9m9\" (UID: \"35bcc211-d8a1-482e-a44d-82870293177f\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-jj9m9" Oct 30 17:02:14.039517 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:13.934619 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clrnm\" (UniqueName: \"kubernetes.io/projected/35bcc211-d8a1-482e-a44d-82870293177f-kube-api-access-clrnm\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-jj9m9\" (UID: \"35bcc211-d8a1-482e-a44d-82870293177f\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-jj9m9" Oct 30 17:02:14.080436 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:14.080364 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-jj9m9" Oct 30 17:02:14.475246 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:14.436172 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-jj9m9"] Oct 30 17:02:14.475246 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:02:14.437351 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35bcc211_d8a1_482e_a44d_82870293177f.slice/crio-a6393aea6f1eb838f4888a71f79f9b6b13359d5da1e908ae913b71c5e4bc6d12 WatchSource:0}: Error finding container a6393aea6f1eb838f4888a71f79f9b6b13359d5da1e908ae913b71c5e4bc6d12: Status 404 returned error can't find the container with id a6393aea6f1eb838f4888a71f79f9b6b13359d5da1e908ae913b71c5e4bc6d12 Oct 30 17:02:14.475246 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:14.440859 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:02:14.475246 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:14.440925 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:02:14.475246 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:14.440952 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:02:14.844978 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:14.844941 4493 generic.go:356] "Generic (PLEG): container finished" podID="43ebbaf3-58f1-4f88-99d2-171e4ca824c6" containerID="fdb775980abbbda51861f2242ffb1f4e8423cd2cc5e89b2aceeadc035555db07" exitCode=0 Oct 30 17:02:14.845463 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:14.845023 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn" event={"ID":"43ebbaf3-58f1-4f88-99d2-171e4ca824c6","Type":"ContainerDied","Data":"fdb775980abbbda51861f2242ffb1f4e8423cd2cc5e89b2aceeadc035555db07"} Oct 30 17:02:14.845585 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:14.845524 4493 scope.go:122] "RemoveContainer" containerID="fdb775980abbbda51861f2242ffb1f4e8423cd2cc5e89b2aceeadc035555db07" Oct 30 17:02:14.847120 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:14.847095 4493 generic.go:356] "Generic (PLEG): container finished" podID="16b79973-7449-41cf-8709-e8109aa31cd2" containerID="43de03ac859c91924838516a621c6252fbcbe8e77f117f77f0eb5888cab5cb4a" exitCode=0 Oct 30 17:02:14.847260 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:14.847168 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-8ddjc" event={"ID":"16b79973-7449-41cf-8709-e8109aa31cd2","Type":"ContainerDied","Data":"43de03ac859c91924838516a621c6252fbcbe8e77f117f77f0eb5888cab5cb4a"} Oct 30 17:02:14.847565 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:14.847538 4493 scope.go:122] "RemoveContainer" containerID="43de03ac859c91924838516a621c6252fbcbe8e77f117f77f0eb5888cab5cb4a" Oct 30 17:02:14.847674 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:14.847623 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:02:14.847733 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:14.847709 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:02:14.847801 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:14.847754 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:02:14.849595 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:14.849560 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:02:14.849676 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:14.849650 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:02:14.849718 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:14.849699 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:02:14.853545 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:14.853521 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-jj9m9" event={"ID":"35bcc211-d8a1-482e-a44d-82870293177f","Type":"ContainerStarted","Data":"7ac4160356b09b9789fedbbebf161b2650803404479f97d644d8dbf0141cc397"} Oct 30 17:02:14.853652 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:14.853559 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-jj9m9" event={"ID":"35bcc211-d8a1-482e-a44d-82870293177f","Type":"ContainerStarted","Data":"a6393aea6f1eb838f4888a71f79f9b6b13359d5da1e908ae913b71c5e4bc6d12"} Oct 30 17:02:15.065497 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:15.065435 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-jj9m9" podStartSLOduration=2.065416332 podStartE2EDuration="2.065416332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:02:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:02:15.023901282 +0000 UTC m=+2362.251076388" watchObservedRunningTime="2026-10-30 17:02:15.065416332 +0000 UTC m=+2362.292591433" Oct 30 17:02:15.086729 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:15.086696 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-jj9m9" Oct 30 17:02:15.087127 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:15.087105 4493 patch_prober.go:29] interesting pod/e2e-autores-81733d-openshift-default-688ffd6987-jj9m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body= Oct 30 17:02:15.087323 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:15.087298 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-jj9m9" podUID="35bcc211-d8a1-482e-a44d-82870293177f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" Oct 30 17:02:15.870829 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:15.870691 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn" event={"ID":"43ebbaf3-58f1-4f88-99d2-171e4ca824c6","Type":"ContainerStarted","Data":"7e75f8583e4bd652ef609889cf82d0f2bbe06e4767db2b1226ad15bdedecee62"} Oct 30 17:02:15.874597 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:15.874428 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-8ddjc" event={"ID":"16b79973-7449-41cf-8709-e8109aa31cd2","Type":"ContainerStarted","Data":"54f2529a1477ea931be8854ab40ac7433fe8631fe8244c969ed07d8710e9c876"} Oct 30 17:02:16.081810 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:16.080886 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-8ddjc" Oct 30 17:02:16.081810 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:16.081378 4493 patch_prober.go:29] interesting pod/e2e-autores-81733d-openshift-default-688ffd6987-jj9m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body= Oct 30 17:02:16.081810 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:16.081426 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-jj9m9" podUID="35bcc211-d8a1-482e-a44d-82870293177f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" Oct 30 17:02:16.781161 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:16.781131 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-8ddjc" Oct 30 17:02:16.785091 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:16.785064 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn" Oct 30 17:02:16.795877 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:16.795047 4493 patch_prober.go:29] interesting pod/e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body= Oct 30 17:02:16.795877 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:16.795116 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn" podUID="43ebbaf3-58f1-4f88-99d2-171e4ca824c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" Oct 30 17:02:16.795877 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:16.795457 4493 patch_prober.go:29] interesting pod/e2e-autores-81733d-openshift-default-688ffd6987-8ddjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Oct 30 17:02:16.795877 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:16.795492 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-8ddjc" podUID="16b79973-7449-41cf-8709-e8109aa31cd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Oct 30 17:02:17.084863 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:17.084240 4493 patch_prober.go:29] interesting pod/e2e-autores-81733d-openshift-default-688ffd6987-jj9m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body= Oct 30 17:02:17.084863 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:17.084313 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-jj9m9" podUID="35bcc211-d8a1-482e-a44d-82870293177f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" Oct 30 17:02:17.790264 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:17.790227 4493 patch_prober.go:29] interesting pod/e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body= Oct 30 17:02:17.790595 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:17.790566 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn" podUID="43ebbaf3-58f1-4f88-99d2-171e4ca824c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" Oct 30 17:02:17.791376 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:17.791356 4493 patch_prober.go:29] interesting pod/e2e-autores-81733d-openshift-default-688ffd6987-8ddjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Oct 30 17:02:17.791557 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:17.791537 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-8ddjc" podUID="16b79973-7449-41cf-8709-e8109aa31cd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Oct 30 17:02:18.086256 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:18.086165 4493 patch_prober.go:29] interesting pod/e2e-autores-81733d-openshift-default-688ffd6987-jj9m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body= Oct 30 17:02:18.087221 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:18.086829 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-jj9m9" podUID="35bcc211-d8a1-482e-a44d-82870293177f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" Oct 30 17:02:18.097429 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:18.096523 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn" Oct 30 17:02:18.782215 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:18.782174 4493 patch_prober.go:29] interesting pod/e2e-autores-81733d-openshift-default-688ffd6987-8ddjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Oct 30 17:02:18.782764 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:18.782735 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-8ddjc" podUID="16b79973-7449-41cf-8709-e8109aa31cd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Oct 30 17:02:18.783303 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:18.782664 4493 patch_prober.go:29] interesting pod/e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body= Oct 30 17:02:18.783473 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:18.783452 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn" podUID="43ebbaf3-58f1-4f88-99d2-171e4ca824c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" Oct 30 17:02:19.081305 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:19.081208 4493 patch_prober.go:29] interesting pod/e2e-autores-81733d-openshift-default-688ffd6987-jj9m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body= Oct 30 17:02:19.081590 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:19.081561 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-jj9m9" podUID="35bcc211-d8a1-482e-a44d-82870293177f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" Oct 30 17:02:19.792810 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:19.792187 4493 patch_prober.go:29] interesting pod/e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body= Oct 30 17:02:19.792810 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:19.792265 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn" podUID="43ebbaf3-58f1-4f88-99d2-171e4ca824c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" Oct 30 17:02:19.794916 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:19.794876 4493 patch_prober.go:29] interesting pod/e2e-autores-81733d-openshift-default-688ffd6987-8ddjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Oct 30 17:02:19.795083 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:19.794918 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-8ddjc" podUID="16b79973-7449-41cf-8709-e8109aa31cd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Oct 30 17:02:20.086271 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:20.086183 4493 patch_prober.go:29] interesting pod/e2e-autores-81733d-openshift-default-688ffd6987-jj9m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body= Oct 30 17:02:20.086531 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:20.086499 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-jj9m9" podUID="35bcc211-d8a1-482e-a44d-82870293177f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" Oct 30 17:02:20.792806 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:20.787208 4493 patch_prober.go:29] interesting pod/e2e-autores-81733d-openshift-default-688ffd6987-8ddjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Oct 30 17:02:20.792806 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:20.787291 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-8ddjc" podUID="16b79973-7449-41cf-8709-e8109aa31cd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Oct 30 17:02:20.792806 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:20.788458 4493 patch_prober.go:29] interesting pod/e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body= Oct 30 17:02:20.792806 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:20.788514 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn" podUID="43ebbaf3-58f1-4f88-99d2-171e4ca824c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" Oct 30 17:02:21.111867 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:21.110726 4493 patch_prober.go:29] interesting pod/e2e-autores-81733d-openshift-default-688ffd6987-jj9m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body= Oct 30 17:02:21.111867 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:21.110803 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-jj9m9" podUID="35bcc211-d8a1-482e-a44d-82870293177f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" Oct 30 17:02:21.219808 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:21.208229 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 17:02:21.783957 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:21.783917 4493 patch_prober.go:29] interesting pod/e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body= Oct 30 17:02:21.784212 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:21.784184 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn" podUID="43ebbaf3-58f1-4f88-99d2-171e4ca824c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" Oct 30 17:02:21.785501 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:21.785477 4493 patch_prober.go:29] interesting pod/e2e-autores-81733d-openshift-default-688ffd6987-8ddjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Oct 30 17:02:21.785684 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:21.785664 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-8ddjc" podUID="16b79973-7449-41cf-8709-e8109aa31cd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Oct 30 17:02:22.081976 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:22.081864 4493 patch_prober.go:29] interesting pod/e2e-autores-81733d-openshift-default-688ffd6987-jj9m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body= Oct 30 17:02:22.082566 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:22.082533 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-jj9m9" podUID="35bcc211-d8a1-482e-a44d-82870293177f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" Oct 30 17:02:22.882810 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:22.882536 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-8ddjc" Oct 30 17:02:22.885307 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:22.885066 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-8ddjc" Oct 30 17:02:22.943954 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:22.943923 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn" Oct 30 17:02:22.946517 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:22.946494 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn" Oct 30 17:02:23.132657 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:23.129753 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-jj9m9" Oct 30 17:02:23.132657 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:23.131213 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-jj9m9" Oct 30 17:02:23.134685 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:23.134630 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-jj9m9" Oct 30 17:02:30.366750 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:02:30.366706 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 17:02:43.125107 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:43.124884 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" podUID="b5abe703-fee0-4185-aa22-88d1db06392b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:02:43.125851 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:43.125816 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" podUID="14805045-a727-41ba-8bd0-100d41f9f27d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:02:43.136815 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:43.127714 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" podUID="e76172b1-659d-45bd-91c6-1a0ce4062335" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:02:43.170350 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:43.170305 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" podUID="e90f43d3-f5eb-45ab-af1e-340d0ce18966" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Oct 30 17:02:43.255035 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:43.254987 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" podUID="18b72648-f523-405e-b5cd-1266ff13f8f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:02:49.598939 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:49.598904 4493 patch_prober.go:29] interesting pod/e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:02:49.599517 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:49.598981 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w" podUID="8e21f846-6c01-496f-998e-ccfe2a18d5cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:02:50.306956 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:50.306909 4493 patch_prober.go:29] interesting pod/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:02:50.307174 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:50.307016 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2" podUID="98af0974-af46-432d-a622-4279717ff916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:02:53.227524 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:53.227468 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-81733d-rhbkd"] Oct 30 17:02:53.255213 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:53.255139 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-81733d-rhbkd" Oct 30 17:02:53.256831 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:53.256317 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-81733d-rhbkd"] Oct 30 17:02:53.270827 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:53.270763 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:02:53.284575 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:53.283142 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:02:53.284575 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:53.283232 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 17:02:53.284575 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:53.284544 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:02:53.402274 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:53.402241 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtl4t\" (UniqueName: \"kubernetes.io/projected/225982e6-1c95-400d-9ad0-ad6984dfd8d5-kube-api-access-vtl4t\") pod \"maas-api-revoke-keys-e2e-autores-81733d-rhbkd\" (UID: \"225982e6-1c95-400d-9ad0-ad6984dfd8d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-81733d-rhbkd" Oct 30 17:02:53.402468 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:53.402363 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/225982e6-1c95-400d-9ad0-ad6984dfd8d5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-81733d-rhbkd\" (UID: \"225982e6-1c95-400d-9ad0-ad6984dfd8d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-81733d-rhbkd" Oct 30 17:02:53.402540 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:53.402479 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/225982e6-1c95-400d-9ad0-ad6984dfd8d5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-81733d-rhbkd\" (UID: \"225982e6-1c95-400d-9ad0-ad6984dfd8d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-81733d-rhbkd" Oct 30 17:02:53.503456 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:53.503367 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/225982e6-1c95-400d-9ad0-ad6984dfd8d5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-81733d-rhbkd\" (UID: \"225982e6-1c95-400d-9ad0-ad6984dfd8d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-81733d-rhbkd" Oct 30 17:02:53.503639 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:53.503475 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/225982e6-1c95-400d-9ad0-ad6984dfd8d5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-81733d-rhbkd\" (UID: \"225982e6-1c95-400d-9ad0-ad6984dfd8d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-81733d-rhbkd" Oct 30 17:02:53.503639 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:53.503574 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtl4t\" (UniqueName: \"kubernetes.io/projected/225982e6-1c95-400d-9ad0-ad6984dfd8d5-kube-api-access-vtl4t\") pod \"maas-api-revoke-keys-e2e-autores-81733d-rhbkd\" (UID: \"225982e6-1c95-400d-9ad0-ad6984dfd8d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-81733d-rhbkd" Oct 30 17:02:53.504178 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:53.504134 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/225982e6-1c95-400d-9ad0-ad6984dfd8d5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-81733d-rhbkd\" (UID: \"225982e6-1c95-400d-9ad0-ad6984dfd8d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-81733d-rhbkd" Oct 30 17:02:53.506695 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:53.506662 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/225982e6-1c95-400d-9ad0-ad6984dfd8d5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-81733d-rhbkd\" (UID: \"225982e6-1c95-400d-9ad0-ad6984dfd8d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-81733d-rhbkd" Oct 30 17:02:53.528350 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:53.528315 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtl4t\" (UniqueName: \"kubernetes.io/projected/225982e6-1c95-400d-9ad0-ad6984dfd8d5-kube-api-access-vtl4t\") pod \"maas-api-revoke-keys-e2e-autores-81733d-rhbkd\" (UID: \"225982e6-1c95-400d-9ad0-ad6984dfd8d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-81733d-rhbkd" Oct 30 17:02:53.594876 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:53.594837 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-81733d-rhbkd" Oct 30 17:02:53.930870 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:53.930836 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-81733d-rhbkd"] Oct 30 17:02:53.931075 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:02:53.930896 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod225982e6_1c95_400d_9ad0_ad6984dfd8d5.slice/crio-84aed0b282add5a17ade317dd6673eaeeb5bfdb3736fcb2627ba304cc3625861 WatchSource:0}: Error finding container 84aed0b282add5a17ade317dd6673eaeeb5bfdb3736fcb2627ba304cc3625861: Status 404 returned error can't find the container with id 84aed0b282add5a17ade317dd6673eaeeb5bfdb3736fcb2627ba304cc3625861 Oct 30 17:02:54.899280 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:54.573775 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-81733d-rhbkd" event={"ID":"225982e6-1c95-400d-9ad0-ad6984dfd8d5","Type":"ContainerStarted","Data":"a9ed7bb3ceaf6984c5e8f08d5098cac6a78c4d7f7b4a0ed77b6dac62df2309f2"} Oct 30 17:02:54.899280 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:54.573839 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-81733d-rhbkd" event={"ID":"225982e6-1c95-400d-9ad0-ad6984dfd8d5","Type":"ContainerStarted","Data":"84aed0b282add5a17ade317dd6673eaeeb5bfdb3736fcb2627ba304cc3625861"} Oct 30 17:02:54.899280 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:54.624912 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-81733d-rhbkd" podStartSLOduration=1.624894208 podStartE2EDuration="1.624894208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:02:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:02:54.61963964 +0000 UTC m=+2401.846814758" watchObservedRunningTime="2026-10-30 17:02:54.624894208 +0000 UTC m=+2401.852069310" Oct 30 17:02:54.899280 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:54.662403 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-56v9s_d3a1c417-2698-4462-b7ff-1d2e66d7a10f/ovn-acl-logging/0.log" Oct 30 17:02:54.899280 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:54.669182 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-56v9s_d3a1c417-2698-4462-b7ff-1d2e66d7a10f/ovn-acl-logging/0.log" Oct 30 17:02:55.582650 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:55.582615 4493 generic.go:356] "Generic (PLEG): container finished" podID="225982e6-1c95-400d-9ad0-ad6984dfd8d5" containerID="a9ed7bb3ceaf6984c5e8f08d5098cac6a78c4d7f7b4a0ed77b6dac62df2309f2" exitCode=0 Oct 30 17:02:55.582944 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:55.582657 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-81733d-rhbkd" event={"ID":"225982e6-1c95-400d-9ad0-ad6984dfd8d5","Type":"ContainerDied","Data":"a9ed7bb3ceaf6984c5e8f08d5098cac6a78c4d7f7b4a0ed77b6dac62df2309f2"} Oct 30 17:02:55.584431 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:55.584410 4493 generic.go:356] "Generic (PLEG): container finished" podID="98af0974-af46-432d-a622-4279717ff916" containerID="8ef450d4c2c76eaad33b7c06f4f13933980135fe562eeff48190e8642a734f56" exitCode=0 Oct 30 17:02:55.584548 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:55.584488 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2" event={"ID":"98af0974-af46-432d-a622-4279717ff916","Type":"ContainerDied","Data":"8ef450d4c2c76eaad33b7c06f4f13933980135fe562eeff48190e8642a734f56"} Oct 30 17:02:55.584931 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:55.584911 4493 scope.go:122] "RemoveContainer" containerID="8ef450d4c2c76eaad33b7c06f4f13933980135fe562eeff48190e8642a734f56" Oct 30 17:02:55.586411 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:55.586388 4493 generic.go:356] "Generic (PLEG): container finished" podID="8e21f846-6c01-496f-998e-ccfe2a18d5cd" containerID="5df355cf8894c3cd55e3c567e4a600995b29412c5e2b8a1ce4dc7165513c8b38" exitCode=0 Oct 30 17:02:55.586502 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:55.586456 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w" event={"ID":"8e21f846-6c01-496f-998e-ccfe2a18d5cd","Type":"ContainerDied","Data":"5df355cf8894c3cd55e3c567e4a600995b29412c5e2b8a1ce4dc7165513c8b38"} Oct 30 17:02:55.586677 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:55.586651 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:02:55.586734 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:55.586721 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:02:55.586773 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:55.586758 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:02:55.586832 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:55.586819 4493 scope.go:122] "RemoveContainer" containerID="5df355cf8894c3cd55e3c567e4a600995b29412c5e2b8a1ce4dc7165513c8b38" Oct 30 17:02:55.588293 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:55.588266 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:02:55.588380 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:55.588326 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:02:55.588380 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:55.588358 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:02:56.278969 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:56.277765 4493 scope.go:122] "RemoveContainer" containerID="c07523f6483e7498f42a9e9a1088e7ab2b12c24eef4b19750fbd332a4bab7145" Oct 30 17:02:56.596855 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:56.596741 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2" event={"ID":"98af0974-af46-432d-a622-4279717ff916","Type":"ContainerStarted","Data":"09d5075eaf353d561bd7b8b3a7a0a550b6027cd31ea24c0d73cf4ca45c7d5e03"} Oct 30 17:02:56.599202 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:56.599163 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w" event={"ID":"8e21f846-6c01-496f-998e-ccfe2a18d5cd","Type":"ContainerStarted","Data":"b00a36923850817ca34a07fff2dc81de8bbd8b84e0459f629104b19a2bce9295"} Oct 30 17:02:56.616167 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:56.616141 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w" Oct 30 17:02:56.617496 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:56.617469 4493 patch_prober.go:29] interesting pod/e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body= Oct 30 17:02:56.617725 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:56.617700 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w" podUID="8e21f846-6c01-496f-998e-ccfe2a18d5cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" Oct 30 17:02:56.890142 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:56.890052 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2" Oct 30 17:02:56.890326 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:56.890205 4493 patch_prober.go:29] interesting pod/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Oct 30 17:02:56.890326 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:56.890248 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2" podUID="98af0974-af46-432d-a622-4279717ff916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Oct 30 17:02:56.994674 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:56.994647 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-81733d-rhbkd" Oct 30 17:02:57.070841 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:57.070526 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/225982e6-1c95-400d-9ad0-ad6984dfd8d5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/225982e6-1c95-400d-9ad0-ad6984dfd8d5-maas-api-xks-ca\") pod \"225982e6-1c95-400d-9ad0-ad6984dfd8d5\" (UID: \"225982e6-1c95-400d-9ad0-ad6984dfd8d5\") " Oct 30 17:02:57.071048 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:57.070849 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/225982e6-1c95-400d-9ad0-ad6984dfd8d5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/225982e6-1c95-400d-9ad0-ad6984dfd8d5-maas-api-openshift-ca\") pod \"225982e6-1c95-400d-9ad0-ad6984dfd8d5\" (UID: \"225982e6-1c95-400d-9ad0-ad6984dfd8d5\") " Oct 30 17:02:57.071048 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:57.070926 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/225982e6-1c95-400d-9ad0-ad6984dfd8d5-kube-api-access-vtl4t\" (UniqueName: \"kubernetes.io/projected/225982e6-1c95-400d-9ad0-ad6984dfd8d5-kube-api-access-vtl4t\") pod \"225982e6-1c95-400d-9ad0-ad6984dfd8d5\" (UID: \"225982e6-1c95-400d-9ad0-ad6984dfd8d5\") " Oct 30 17:02:57.072704 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:57.071470 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/225982e6-1c95-400d-9ad0-ad6984dfd8d5-maas-api-openshift-ca" pod "225982e6-1c95-400d-9ad0-ad6984dfd8d5" (UID: "225982e6-1c95-400d-9ad0-ad6984dfd8d5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:02:57.075804 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:57.075494 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/225982e6-1c95-400d-9ad0-ad6984dfd8d5-kube-api-access-vtl4t" pod "225982e6-1c95-400d-9ad0-ad6984dfd8d5" (UID: "225982e6-1c95-400d-9ad0-ad6984dfd8d5"). InnerVolumeSpecName "kube-api-access-vtl4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:02:57.075804 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:57.075605 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/225982e6-1c95-400d-9ad0-ad6984dfd8d5-maas-api-xks-ca" pod "225982e6-1c95-400d-9ad0-ad6984dfd8d5" (UID: "225982e6-1c95-400d-9ad0-ad6984dfd8d5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:02:57.198939 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:57.198150 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/225982e6-1c95-400d-9ad0-ad6984dfd8d5-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:02:57.198939 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:57.198184 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtl4t\" (UniqueName: \"kubernetes.io/projected/225982e6-1c95-400d-9ad0-ad6984dfd8d5-kube-api-access-vtl4t\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:02:57.198939 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:57.198199 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/225982e6-1c95-400d-9ad0-ad6984dfd8d5-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:02:57.627867 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:57.608087 4493 patch_prober.go:29] interesting pod/e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body= Oct 30 17:02:57.627867 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:57.608142 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w" podUID="8e21f846-6c01-496f-998e-ccfe2a18d5cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" Oct 30 17:02:57.633807 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:57.630829 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-81733d-rhbkd" event={"ID":"225982e6-1c95-400d-9ad0-ad6984dfd8d5","Type":"ContainerDied","Data":"84aed0b282add5a17ade317dd6673eaeeb5bfdb3736fcb2627ba304cc3625861"} Oct 30 17:02:57.633807 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:57.630896 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="84aed0b282add5a17ade317dd6673eaeeb5bfdb3736fcb2627ba304cc3625861" Oct 30 17:02:57.637806 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:57.634283 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-81733d-rhbkd" Oct 30 17:02:57.890256 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:57.890168 4493 patch_prober.go:29] interesting pod/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Oct 30 17:02:57.890256 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:57.890226 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2" podUID="98af0974-af46-432d-a622-4279717ff916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Oct 30 17:02:58.611811 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:58.608198 4493 patch_prober.go:29] interesting pod/e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body= Oct 30 17:02:58.611811 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:58.608259 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w" podUID="8e21f846-6c01-496f-998e-ccfe2a18d5cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" Oct 30 17:02:58.890099 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:58.890011 4493 patch_prober.go:29] interesting pod/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Oct 30 17:02:58.890099 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:58.890085 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2" podUID="98af0974-af46-432d-a622-4279717ff916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Oct 30 17:02:59.605581 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:59.605137 4493 patch_prober.go:29] interesting pod/e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body= Oct 30 17:02:59.605581 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:59.605208 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w" podUID="8e21f846-6c01-496f-998e-ccfe2a18d5cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" Oct 30 17:02:59.890610 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:59.890579 4493 patch_prober.go:29] interesting pod/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Oct 30 17:02:59.891134 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:02:59.890639 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2" podUID="98af0974-af46-432d-a622-4279717ff916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Oct 30 17:03:00.385929 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:03:00.385839 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 17:03:00.608285 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:00.608251 4493 patch_prober.go:29] interesting pod/e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body= Oct 30 17:03:00.608587 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:00.608555 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w" podUID="8e21f846-6c01-496f-998e-ccfe2a18d5cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" Oct 30 17:03:00.890503 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:00.890467 4493 patch_prober.go:29] interesting pod/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Oct 30 17:03:00.890677 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:00.890525 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2" podUID="98af0974-af46-432d-a622-4279717ff916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Oct 30 17:03:01.608209 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:01.608171 4493 patch_prober.go:29] interesting pod/e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body= Oct 30 17:03:01.608514 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:01.608483 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w" podUID="8e21f846-6c01-496f-998e-ccfe2a18d5cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" Oct 30 17:03:01.890122 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:01.889998 4493 patch_prober.go:29] interesting pod/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Oct 30 17:03:01.890313 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:01.890117 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2" podUID="98af0974-af46-432d-a622-4279717ff916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Oct 30 17:03:02.610700 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:02.610668 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w" Oct 30 17:03:02.611228 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:02.611183 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w" Oct 30 17:03:02.611946 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:02.611893 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w" Oct 30 17:03:02.895654 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:02.895625 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2" Oct 30 17:03:02.896058 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:02.896039 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2" Oct 30 17:03:02.896842 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:02.896815 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2" Oct 30 17:03:13.950351 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:13.949989 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 17:03:26.064870 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:26.064836 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e09cf108-z22kj"] Oct 30 17:03:26.140329 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:26.140291 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e09cf108-z22kj"] Oct 30 17:03:26.942255 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:26.942217 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fbc97f7-5a3b-440a-af9f-628fba76f84c" path="/var/lib/kubelet/pods/0fbc97f7-5a3b-440a-af9f-628fba76f84c/volumes" Oct 30 17:03:30.603414 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:03:30.603343 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 17:03:36.046407 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:36.046373 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4196ea9-8vvfn"] Oct 30 17:03:36.065409 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:36.055949 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4196ea9-8vvfn"] Oct 30 17:03:36.935590 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:36.935556 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfbf22a8-600e-4cbd-a986-53846ab58d02" path="/var/lib/kubelet/pods/dfbf22a8-600e-4cbd-a986-53846ab58d02/volumes" Oct 30 17:03:37.135221 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:37.135191 4493 generic.go:356] "Generic (PLEG): container finished" podID="defb21aa-d0d6-48b8-ba50-4fcb900bc547" containerID="0f67a2519bbe7709e504d80e7a989283a61519544a02432f59742bce309ac05b" exitCode=0 Oct 30 17:03:37.135647 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:37.135228 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz" event={"ID":"defb21aa-d0d6-48b8-ba50-4fcb900bc547","Type":"ContainerDied","Data":"0f67a2519bbe7709e504d80e7a989283a61519544a02432f59742bce309ac05b"} Oct 30 17:03:37.135729 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:37.135684 4493 scope.go:122] "RemoveContainer" containerID="0f67a2519bbe7709e504d80e7a989283a61519544a02432f59742bce309ac05b" Oct 30 17:03:37.137577 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:37.137546 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:03:37.137637 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:37.137620 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:03:37.137674 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:37.137651 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:03:38.150693 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:38.150656 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz" event={"ID":"defb21aa-d0d6-48b8-ba50-4fcb900bc547","Type":"ContainerStarted","Data":"cd0d3f94c0db2d1fe7238d36caed62993ca7f437542ca6581af4ba858a26dcf8"} Oct 30 17:03:38.235171 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:38.235140 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz" Oct 30 17:03:38.235549 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:38.235460 4493 patch_prober.go:29] interesting pod/e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body= Oct 30 17:03:38.235735 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:38.235712 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz" podUID="defb21aa-d0d6-48b8-ba50-4fcb900bc547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" Oct 30 17:03:39.258611 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:39.258572 4493 patch_prober.go:29] interesting pod/e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body= Oct 30 17:03:39.259364 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:39.259333 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz" podUID="defb21aa-d0d6-48b8-ba50-4fcb900bc547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" Oct 30 17:03:40.235670 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:40.235238 4493 patch_prober.go:29] interesting pod/e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body= Oct 30 17:03:40.235670 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:40.235297 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz" podUID="defb21aa-d0d6-48b8-ba50-4fcb900bc547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" Oct 30 17:03:41.236807 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:41.236164 4493 patch_prober.go:29] interesting pod/e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body= Oct 30 17:03:41.236807 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:41.236239 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz" podUID="defb21aa-d0d6-48b8-ba50-4fcb900bc547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" Oct 30 17:03:42.236820 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:42.235171 4493 patch_prober.go:29] interesting pod/e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body= Oct 30 17:03:42.236820 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:42.235249 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz" podUID="defb21aa-d0d6-48b8-ba50-4fcb900bc547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" Oct 30 17:03:43.235201 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:43.235164 4493 patch_prober.go:29] interesting pod/e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body= Oct 30 17:03:43.235529 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:43.235499 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz" podUID="defb21aa-d0d6-48b8-ba50-4fcb900bc547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" Oct 30 17:03:43.384665 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:43.384634 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-81733d-rhbkd"] Oct 30 17:03:43.389564 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:43.389533 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-81733d-rhbkd"] Oct 30 17:03:43.645168 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:43.645134 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz" Oct 30 17:03:44.240853 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:44.240825 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz" Oct 30 17:03:44.242043 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:44.242020 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz" Oct 30 17:03:44.934506 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:44.934472 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="225982e6-1c95-400d-9ad0-ad6984dfd8d5" path="/var/lib/kubelet/pods/225982e6-1c95-400d-9ad0-ad6984dfd8d5/volumes" Oct 30 17:03:46.044838 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:46.044723 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b90ec4e-w76hj"] Oct 30 17:03:46.052111 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:46.049825 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b90ec4e-w76hj"] Oct 30 17:03:46.954640 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:46.954605 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="993cb549-cec9-4159-a0b2-a023e2cd26c8" path="/var/lib/kubelet/pods/993cb549-cec9-4159-a0b2-a023e2cd26c8/volumes" Oct 30 17:03:47.320255 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:47.320156 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw"] Oct 30 17:03:47.320745 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:47.320335 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw" podUID="92c1d749-8598-4a7b-b840-042533ac0bea" containerName="istio-proxy" containerID="cri-o://a3f3b65b8f0a1f539fbe0e280ce6cf5432a16a5f7e4d05b092031dc5a9e00201" gracePeriod=30 Oct 30 17:03:50.824314 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:50.824260 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-jj9m9"] Oct 30 17:03:50.825236 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:50.825198 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-jj9m9" podUID="35bcc211-d8a1-482e-a44d-82870293177f" containerName="istio-proxy" containerID="cri-o://7ac4160356b09b9789fedbbebf161b2650803404479f97d644d8dbf0141cc397" gracePeriod=30 Oct 30 17:03:50.839285 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:50.839255 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-8ddjc"] Oct 30 17:03:50.839853 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:50.839820 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-8ddjc" podUID="16b79973-7449-41cf-8709-e8109aa31cd2" containerName="istio-proxy" containerID="cri-o://54f2529a1477ea931be8854ab40ac7433fe8631fe8244c969ed07d8710e9c876" gracePeriod=30 Oct 30 17:03:52.606309 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:52.606239 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw" Oct 30 17:03:52.735207 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:52.735173 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/92c1d749-8598-4a7b-b840-042533ac0bea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92c1d749-8598-4a7b-b840-042533ac0bea-istio-podinfo\") pod \"92c1d749-8598-4a7b-b840-042533ac0bea\" (UID: \"92c1d749-8598-4a7b-b840-042533ac0bea\") " Oct 30 17:03:52.735207 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:52.735214 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92c1d749-8598-4a7b-b840-042533ac0bea-istio-token\" (UniqueName: \"kubernetes.io/projected/92c1d749-8598-4a7b-b840-042533ac0bea-istio-token\") pod \"92c1d749-8598-4a7b-b840-042533ac0bea\" (UID: \"92c1d749-8598-4a7b-b840-042533ac0bea\") " Oct 30 17:03:52.735442 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:52.735231 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92c1d749-8598-4a7b-b840-042533ac0bea-kube-api-access-6zdqz\" (UniqueName: \"kubernetes.io/projected/92c1d749-8598-4a7b-b840-042533ac0bea-kube-api-access-6zdqz\") pod \"92c1d749-8598-4a7b-b840-042533ac0bea\" (UID: \"92c1d749-8598-4a7b-b840-042533ac0bea\") " Oct 30 17:03:52.735442 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:52.735309 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92c1d749-8598-4a7b-b840-042533ac0bea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92c1d749-8598-4a7b-b840-042533ac0bea-workload-socket\") pod \"92c1d749-8598-4a7b-b840-042533ac0bea\" (UID: \"92c1d749-8598-4a7b-b840-042533ac0bea\") " Oct 30 17:03:52.735442 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:52.735345 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92c1d749-8598-4a7b-b840-042533ac0bea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92c1d749-8598-4a7b-b840-042533ac0bea-istio-data\") pod \"92c1d749-8598-4a7b-b840-042533ac0bea\" (UID: \"92c1d749-8598-4a7b-b840-042533ac0bea\") " Oct 30 17:03:52.735442 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:52.735439 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92c1d749-8598-4a7b-b840-042533ac0bea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92c1d749-8598-4a7b-b840-042533ac0bea-istiod-ca-cert\") pod \"92c1d749-8598-4a7b-b840-042533ac0bea\" (UID: \"92c1d749-8598-4a7b-b840-042533ac0bea\") " Oct 30 17:03:52.735680 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:52.735497 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92c1d749-8598-4a7b-b840-042533ac0bea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92c1d749-8598-4a7b-b840-042533ac0bea-istio-envoy\") pod \"92c1d749-8598-4a7b-b840-042533ac0bea\" (UID: \"92c1d749-8598-4a7b-b840-042533ac0bea\") " Oct 30 17:03:52.735680 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:52.735539 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92c1d749-8598-4a7b-b840-042533ac0bea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92c1d749-8598-4a7b-b840-042533ac0bea-workload-certs\") pod \"92c1d749-8598-4a7b-b840-042533ac0bea\" (UID: \"92c1d749-8598-4a7b-b840-042533ac0bea\") " Oct 30 17:03:52.735680 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:52.735580 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92c1d749-8598-4a7b-b840-042533ac0bea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92c1d749-8598-4a7b-b840-042533ac0bea-credential-socket\") pod \"92c1d749-8598-4a7b-b840-042533ac0bea\" (UID: \"92c1d749-8598-4a7b-b840-042533ac0bea\") " Oct 30 17:03:52.735680 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:52.735612 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92c1d749-8598-4a7b-b840-042533ac0bea-workload-socket" pod "92c1d749-8598-4a7b-b840-042533ac0bea" (UID: "92c1d749-8598-4a7b-b840-042533ac0bea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:03:52.735968 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:52.735676 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92c1d749-8598-4a7b-b840-042533ac0bea-istio-data" pod "92c1d749-8598-4a7b-b840-042533ac0bea" (UID: "92c1d749-8598-4a7b-b840-042533ac0bea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:03:52.735968 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:52.735853 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92c1d749-8598-4a7b-b840-042533ac0bea-workload-certs" pod "92c1d749-8598-4a7b-b840-042533ac0bea" (UID: "92c1d749-8598-4a7b-b840-042533ac0bea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:03:52.736075 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:52.735961 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92c1d749-8598-4a7b-b840-042533ac0bea-credential-socket" pod "92c1d749-8598-4a7b-b840-042533ac0bea" (UID: "92c1d749-8598-4a7b-b840-042533ac0bea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:03:52.736075 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:52.735911 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92c1d749-8598-4a7b-b840-042533ac0bea-istiod-ca-cert" pod "92c1d749-8598-4a7b-b840-042533ac0bea" (UID: "92c1d749-8598-4a7b-b840-042533ac0bea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:03:52.736247 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:52.736226 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92c1d749-8598-4a7b-b840-042533ac0bea-workload-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:03:52.736322 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:52.736254 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92c1d749-8598-4a7b-b840-042533ac0bea-istio-data\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:03:52.736322 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:52.736269 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92c1d749-8598-4a7b-b840-042533ac0bea-istiod-ca-cert\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:03:52.736322 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:52.736282 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92c1d749-8598-4a7b-b840-042533ac0bea-workload-certs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:03:52.736322 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:52.736295 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92c1d749-8598-4a7b-b840-042533ac0bea-credential-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:03:52.738174 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:52.738149 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92c1d749-8598-4a7b-b840-042533ac0bea-kube-api-access-6zdqz" pod "92c1d749-8598-4a7b-b840-042533ac0bea" (UID: "92c1d749-8598-4a7b-b840-042533ac0bea"). InnerVolumeSpecName "kube-api-access-6zdqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:03:52.738270 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:52.738172 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/92c1d749-8598-4a7b-b840-042533ac0bea-istio-podinfo" pod "92c1d749-8598-4a7b-b840-042533ac0bea" (UID: "92c1d749-8598-4a7b-b840-042533ac0bea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:03:52.738400 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:52.738371 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92c1d749-8598-4a7b-b840-042533ac0bea-istio-envoy" pod "92c1d749-8598-4a7b-b840-042533ac0bea" (UID: "92c1d749-8598-4a7b-b840-042533ac0bea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:03:52.738400 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:52.738375 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92c1d749-8598-4a7b-b840-042533ac0bea-istio-token" pod "92c1d749-8598-4a7b-b840-042533ac0bea" (UID: "92c1d749-8598-4a7b-b840-042533ac0bea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:03:52.837395 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:52.837354 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92c1d749-8598-4a7b-b840-042533ac0bea-istio-podinfo\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:03:52.837395 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:52.837384 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92c1d749-8598-4a7b-b840-042533ac0bea-istio-token\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:03:52.837395 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:52.837397 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zdqz\" (UniqueName: \"kubernetes.io/projected/92c1d749-8598-4a7b-b840-042533ac0bea-kube-api-access-6zdqz\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:03:52.837701 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:52.837410 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92c1d749-8598-4a7b-b840-042533ac0bea-istio-envoy\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:03:52.886176 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:52.886100 4493 patch_prober.go:29] interesting pod/e2e-autores-81733d-openshift-default-688ffd6987-8ddjc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:03:52.886322 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:52.886167 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-8ddjc" podUID="16b79973-7449-41cf-8709-e8109aa31cd2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:03:53.133413 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:53.133380 4493 patch_prober.go:29] interesting pod/e2e-autores-81733d-openshift-default-688ffd6987-jj9m9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:03:53.133578 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:53.133443 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-jj9m9" podUID="35bcc211-d8a1-482e-a44d-82870293177f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:03:53.283030 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:53.282994 4493 generic.go:356] "Generic (PLEG): container finished" podID="92c1d749-8598-4a7b-b840-042533ac0bea" containerID="a3f3b65b8f0a1f539fbe0e280ce6cf5432a16a5f7e4d05b092031dc5a9e00201" exitCode=0 Oct 30 17:03:53.283239 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:53.283068 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw" Oct 30 17:03:53.283239 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:53.283069 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw" event={"ID":"92c1d749-8598-4a7b-b840-042533ac0bea","Type":"ContainerDied","Data":"a3f3b65b8f0a1f539fbe0e280ce6cf5432a16a5f7e4d05b092031dc5a9e00201"} Oct 30 17:03:53.283239 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:53.283135 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw" event={"ID":"92c1d749-8598-4a7b-b840-042533ac0bea","Type":"ContainerDied","Data":"73fd323d337921eb5709b87c19290b7539250f089bea62b493f48d37d677f12d"} Oct 30 17:03:53.283239 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:53.283164 4493 scope.go:122] "RemoveContainer" containerID="a3f3b65b8f0a1f539fbe0e280ce6cf5432a16a5f7e4d05b092031dc5a9e00201" Oct 30 17:03:53.296014 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:53.295933 4493 scope.go:122] "RemoveContainer" containerID="a3f3b65b8f0a1f539fbe0e280ce6cf5432a16a5f7e4d05b092031dc5a9e00201" Oct 30 17:03:53.296288 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:03:53.296265 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3f3b65b8f0a1f539fbe0e280ce6cf5432a16a5f7e4d05b092031dc5a9e00201\": container with ID starting with a3f3b65b8f0a1f539fbe0e280ce6cf5432a16a5f7e4d05b092031dc5a9e00201 not found: ID does not exist" containerID="a3f3b65b8f0a1f539fbe0e280ce6cf5432a16a5f7e4d05b092031dc5a9e00201" Oct 30 17:03:53.296339 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:53.296302 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3f3b65b8f0a1f539fbe0e280ce6cf5432a16a5f7e4d05b092031dc5a9e00201"} err="failed to get container status \"a3f3b65b8f0a1f539fbe0e280ce6cf5432a16a5f7e4d05b092031dc5a9e00201\": rpc error: code = NotFound desc = could not find container \"a3f3b65b8f0a1f539fbe0e280ce6cf5432a16a5f7e4d05b092031dc5a9e00201\": container with ID starting with a3f3b65b8f0a1f539fbe0e280ce6cf5432a16a5f7e4d05b092031dc5a9e00201 not found: ID does not exist" Oct 30 17:03:53.360165 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:53.360116 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw"] Oct 30 17:03:53.360415 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:53.360190 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-mkchw"] Oct 30 17:03:53.397812 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:53.397749 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw"] Oct 30 17:03:53.399000 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:53.398975 4493 cpu_manager.go:414] "containerMap: removing container" podUID="92c1d749-8598-4a7b-b840-042533ac0bea" containerName="istio-proxy" Oct 30 17:03:53.399175 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:53.399159 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92c1d749-8598-4a7b-b840-042533ac0bea" containerName="istio-proxy" Oct 30 17:03:53.399326 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:53.399312 4493 cpu_manager.go:414] "containerMap: removing container" podUID="225982e6-1c95-400d-9ad0-ad6984dfd8d5" containerName="revoke-keys" Oct 30 17:03:53.399414 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:53.399403 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="225982e6-1c95-400d-9ad0-ad6984dfd8d5" containerName="revoke-keys" Oct 30 17:03:53.399682 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:53.399666 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="225982e6-1c95-400d-9ad0-ad6984dfd8d5" containerName="revoke-keys" Oct 30 17:03:53.399834 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:53.399820 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="92c1d749-8598-4a7b-b840-042533ac0bea" containerName="istio-proxy" Oct 30 17:03:53.407834 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:53.404335 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw" Oct 30 17:03:53.431554 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:53.431520 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-9bb5b1-openshift-default-dockercfg-k4wrt\"" Oct 30 17:03:53.438994 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:53.438948 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw"] Oct 30 17:03:53.557771 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:53.557662 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7475582-02dd-456b-a5d5-d894755aba74-istiod-ca-cert\") pod \"e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw\" (UID: \"e7475582-02dd-456b-a5d5-d894755aba74\") " pod="openshift-ingress/e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw" Oct 30 17:03:53.558177 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:53.558148 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7475582-02dd-456b-a5d5-d894755aba74-istio-envoy\") pod \"e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw\" (UID: \"e7475582-02dd-456b-a5d5-d894755aba74\") " pod="openshift-ingress/e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw" Oct 30 17:03:53.558374 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:53.558355 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7475582-02dd-456b-a5d5-d894755aba74-credential-socket\") pod \"e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw\" (UID: \"e7475582-02dd-456b-a5d5-d894755aba74\") " pod="openshift-ingress/e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw" Oct 30 17:03:53.558584 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:53.558567 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7475582-02dd-456b-a5d5-d894755aba74-workload-socket\") pod \"e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw\" (UID: \"e7475582-02dd-456b-a5d5-d894755aba74\") " pod="openshift-ingress/e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw" Oct 30 17:03:53.558713 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:53.558696 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7475582-02dd-456b-a5d5-d894755aba74-istio-podinfo\") pod \"e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw\" (UID: \"e7475582-02dd-456b-a5d5-d894755aba74\") " pod="openshift-ingress/e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw" Oct 30 17:03:53.558911 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:53.558877 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7475582-02dd-456b-a5d5-d894755aba74-istio-token\") pod \"e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw\" (UID: \"e7475582-02dd-456b-a5d5-d894755aba74\") " pod="openshift-ingress/e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw" Oct 30 17:03:53.559224 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:53.559203 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7475582-02dd-456b-a5d5-d894755aba74-workload-certs\") pod \"e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw\" (UID: \"e7475582-02dd-456b-a5d5-d894755aba74\") " pod="openshift-ingress/e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw" Oct 30 17:03:53.559412 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:53.559393 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6v76\" (UniqueName: \"kubernetes.io/projected/e7475582-02dd-456b-a5d5-d894755aba74-kube-api-access-x6v76\") pod \"e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw\" (UID: \"e7475582-02dd-456b-a5d5-d894755aba74\") " pod="openshift-ingress/e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw" Oct 30 17:03:53.559624 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:53.559607 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7475582-02dd-456b-a5d5-d894755aba74-istio-data\") pod \"e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw\" (UID: \"e7475582-02dd-456b-a5d5-d894755aba74\") " pod="openshift-ingress/e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw" Oct 30 17:03:53.670814 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:53.669760 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7475582-02dd-456b-a5d5-d894755aba74-istio-data\") pod \"e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw\" (UID: \"e7475582-02dd-456b-a5d5-d894755aba74\") " pod="openshift-ingress/e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw" Oct 30 17:03:53.670814 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:53.669926 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7475582-02dd-456b-a5d5-d894755aba74-istiod-ca-cert\") pod \"e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw\" (UID: \"e7475582-02dd-456b-a5d5-d894755aba74\") " pod="openshift-ingress/e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw" Oct 30 17:03:53.670814 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:53.669988 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7475582-02dd-456b-a5d5-d894755aba74-istio-envoy\") pod \"e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw\" (UID: \"e7475582-02dd-456b-a5d5-d894755aba74\") " pod="openshift-ingress/e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw" Oct 30 17:03:53.670814 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:53.670044 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7475582-02dd-456b-a5d5-d894755aba74-credential-socket\") pod \"e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw\" (UID: \"e7475582-02dd-456b-a5d5-d894755aba74\") " pod="openshift-ingress/e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw" Oct 30 17:03:53.670814 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:53.670151 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7475582-02dd-456b-a5d5-d894755aba74-workload-socket\") pod \"e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw\" (UID: \"e7475582-02dd-456b-a5d5-d894755aba74\") " pod="openshift-ingress/e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw" Oct 30 17:03:53.670814 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:53.670192 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7475582-02dd-456b-a5d5-d894755aba74-istio-podinfo\") pod \"e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw\" (UID: \"e7475582-02dd-456b-a5d5-d894755aba74\") " pod="openshift-ingress/e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw" Oct 30 17:03:53.670814 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:53.670217 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7475582-02dd-456b-a5d5-d894755aba74-istio-token\") pod \"e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw\" (UID: \"e7475582-02dd-456b-a5d5-d894755aba74\") " pod="openshift-ingress/e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw" Oct 30 17:03:53.670814 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:53.670279 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7475582-02dd-456b-a5d5-d894755aba74-workload-certs\") pod \"e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw\" (UID: \"e7475582-02dd-456b-a5d5-d894755aba74\") " pod="openshift-ingress/e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw" Oct 30 17:03:53.670814 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:53.670321 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6v76\" (UniqueName: \"kubernetes.io/projected/e7475582-02dd-456b-a5d5-d894755aba74-kube-api-access-x6v76\") pod \"e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw\" (UID: \"e7475582-02dd-456b-a5d5-d894755aba74\") " pod="openshift-ingress/e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw" Oct 30 17:03:53.671827 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:53.671652 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7475582-02dd-456b-a5d5-d894755aba74-istio-data\") pod \"e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw\" (UID: \"e7475582-02dd-456b-a5d5-d894755aba74\") " pod="openshift-ingress/e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw" Oct 30 17:03:53.671827 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:53.671724 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7475582-02dd-456b-a5d5-d894755aba74-workload-socket\") pod \"e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw\" (UID: \"e7475582-02dd-456b-a5d5-d894755aba74\") " pod="openshift-ingress/e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw" Oct 30 17:03:53.672331 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:53.672278 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7475582-02dd-456b-a5d5-d894755aba74-workload-certs\") pod \"e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw\" (UID: \"e7475582-02dd-456b-a5d5-d894755aba74\") " pod="openshift-ingress/e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw" Oct 30 17:03:53.672681 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:53.672659 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7475582-02dd-456b-a5d5-d894755aba74-credential-socket\") pod \"e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw\" (UID: \"e7475582-02dd-456b-a5d5-d894755aba74\") " pod="openshift-ingress/e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw" Oct 30 17:03:53.673622 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:53.673601 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7475582-02dd-456b-a5d5-d894755aba74-istiod-ca-cert\") pod \"e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw\" (UID: \"e7475582-02dd-456b-a5d5-d894755aba74\") " pod="openshift-ingress/e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw" Oct 30 17:03:53.676129 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:53.676108 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7475582-02dd-456b-a5d5-d894755aba74-istio-podinfo\") pod \"e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw\" (UID: \"e7475582-02dd-456b-a5d5-d894755aba74\") " pod="openshift-ingress/e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw" Oct 30 17:03:53.677168 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:53.677139 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7475582-02dd-456b-a5d5-d894755aba74-istio-envoy\") pod \"e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw\" (UID: \"e7475582-02dd-456b-a5d5-d894755aba74\") " pod="openshift-ingress/e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw" Oct 30 17:03:53.723691 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:53.723619 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7475582-02dd-456b-a5d5-d894755aba74-istio-token\") pod \"e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw\" (UID: \"e7475582-02dd-456b-a5d5-d894755aba74\") " pod="openshift-ingress/e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw" Oct 30 17:03:53.723691 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:53.723677 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6v76\" (UniqueName: \"kubernetes.io/projected/e7475582-02dd-456b-a5d5-d894755aba74-kube-api-access-x6v76\") pod \"e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw\" (UID: \"e7475582-02dd-456b-a5d5-d894755aba74\") " pod="openshift-ingress/e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw" Oct 30 17:03:53.732017 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:53.731750 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw" Oct 30 17:03:54.068476 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:03:54.068443 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7475582_02dd_456b_a5d5_d894755aba74.slice/crio-c95fb708f621b3523ffbd44a28e29ea7adc82422fb32249f0e12f9789d8dd624 WatchSource:0}: Error finding container c95fb708f621b3523ffbd44a28e29ea7adc82422fb32249f0e12f9789d8dd624: Status 404 returned error can't find the container with id c95fb708f621b3523ffbd44a28e29ea7adc82422fb32249f0e12f9789d8dd624 Oct 30 17:03:54.068643 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:54.068619 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw"] Oct 30 17:03:54.071599 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:54.071564 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:03:54.071716 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:54.071629 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:03:54.071716 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:54.071659 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:03:54.295151 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:54.295111 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw" event={"ID":"e7475582-02dd-456b-a5d5-d894755aba74","Type":"ContainerStarted","Data":"c723df0cc3d721fbe627362f65e2b31c4f3b447d38484c601f0b610ff0fc7e58"} Oct 30 17:03:54.295358 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:54.295177 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw" event={"ID":"e7475582-02dd-456b-a5d5-d894755aba74","Type":"ContainerStarted","Data":"c95fb708f621b3523ffbd44a28e29ea7adc82422fb32249f0e12f9789d8dd624"} Oct 30 17:03:54.345861 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:54.345045 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw" podStartSLOduration=1.345025221 podStartE2EDuration="1.345025221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:03:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:03:54.344324477 +0000 UTC m=+2461.571499592" watchObservedRunningTime="2026-10-30 17:03:54.345025221 +0000 UTC m=+2461.572200322" Oct 30 17:03:54.733210 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:54.733171 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw" Oct 30 17:03:54.937231 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:54.937196 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92c1d749-8598-4a7b-b840-042533ac0bea" path="/var/lib/kubelet/pods/92c1d749-8598-4a7b-b840-042533ac0bea/volumes" Oct 30 17:03:55.738361 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:55.738333 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw" Oct 30 17:03:56.214627 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.214593 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-jj9m9" Oct 30 17:03:56.227005 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.226977 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-8ddjc" Oct 30 17:03:56.305853 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.305742 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35bcc211-d8a1-482e-a44d-82870293177f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35bcc211-d8a1-482e-a44d-82870293177f-istiod-ca-cert\") pod \"35bcc211-d8a1-482e-a44d-82870293177f\" (UID: \"35bcc211-d8a1-482e-a44d-82870293177f\") " Oct 30 17:03:56.306776 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.306755 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35bcc211-d8a1-482e-a44d-82870293177f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35bcc211-d8a1-482e-a44d-82870293177f-workload-socket\") pod \"35bcc211-d8a1-482e-a44d-82870293177f\" (UID: \"35bcc211-d8a1-482e-a44d-82870293177f\") " Oct 30 17:03:56.306971 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.306824 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35bcc211-d8a1-482e-a44d-82870293177f-istio-token\" (UniqueName: \"kubernetes.io/projected/35bcc211-d8a1-482e-a44d-82870293177f-istio-token\") pod \"35bcc211-d8a1-482e-a44d-82870293177f\" (UID: \"35bcc211-d8a1-482e-a44d-82870293177f\") " Oct 30 17:03:56.306971 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.306891 4493 scope.go:122] "RemoveContainer" containerID="6fd139d3315fd40a540805bd12e40bd5f8d6eb403d08a6b15e8afabb34efa364" Oct 30 17:03:56.306971 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.306920 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35bcc211-d8a1-482e-a44d-82870293177f-kube-api-access-clrnm\" (UniqueName: \"kubernetes.io/projected/35bcc211-d8a1-482e-a44d-82870293177f-kube-api-access-clrnm\") pod \"35bcc211-d8a1-482e-a44d-82870293177f\" (UID: \"35bcc211-d8a1-482e-a44d-82870293177f\") " Oct 30 17:03:56.306971 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.306949 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35bcc211-d8a1-482e-a44d-82870293177f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35bcc211-d8a1-482e-a44d-82870293177f-istio-data\") pod \"35bcc211-d8a1-482e-a44d-82870293177f\" (UID: \"35bcc211-d8a1-482e-a44d-82870293177f\") " Oct 30 17:03:56.307712 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.307022 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35bcc211-d8a1-482e-a44d-82870293177f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35bcc211-d8a1-482e-a44d-82870293177f-istio-envoy\") pod \"35bcc211-d8a1-482e-a44d-82870293177f\" (UID: \"35bcc211-d8a1-482e-a44d-82870293177f\") " Oct 30 17:03:56.307712 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.307069 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/35bcc211-d8a1-482e-a44d-82870293177f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35bcc211-d8a1-482e-a44d-82870293177f-istio-podinfo\") pod \"35bcc211-d8a1-482e-a44d-82870293177f\" (UID: \"35bcc211-d8a1-482e-a44d-82870293177f\") " Oct 30 17:03:56.307712 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.307265 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35bcc211-d8a1-482e-a44d-82870293177f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35bcc211-d8a1-482e-a44d-82870293177f-workload-certs\") pod \"35bcc211-d8a1-482e-a44d-82870293177f\" (UID: \"35bcc211-d8a1-482e-a44d-82870293177f\") " Oct 30 17:03:56.307712 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.307296 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35bcc211-d8a1-482e-a44d-82870293177f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35bcc211-d8a1-482e-a44d-82870293177f-credential-socket\") pod \"35bcc211-d8a1-482e-a44d-82870293177f\" (UID: \"35bcc211-d8a1-482e-a44d-82870293177f\") " Oct 30 17:03:56.307712 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.306274 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35bcc211-d8a1-482e-a44d-82870293177f-istiod-ca-cert" pod "35bcc211-d8a1-482e-a44d-82870293177f" (UID: "35bcc211-d8a1-482e-a44d-82870293177f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:03:56.307712 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.307321 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35bcc211-d8a1-482e-a44d-82870293177f-workload-socket" pod "35bcc211-d8a1-482e-a44d-82870293177f" (UID: "35bcc211-d8a1-482e-a44d-82870293177f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:03:56.308204 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.307822 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35bcc211-d8a1-482e-a44d-82870293177f-istiod-ca-cert\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:03:56.308204 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.307868 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35bcc211-d8a1-482e-a44d-82870293177f-workload-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:03:56.308204 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.308074 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35bcc211-d8a1-482e-a44d-82870293177f-credential-socket" pod "35bcc211-d8a1-482e-a44d-82870293177f" (UID: "35bcc211-d8a1-482e-a44d-82870293177f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:03:56.313449 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.310874 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35bcc211-d8a1-482e-a44d-82870293177f-istio-data" pod "35bcc211-d8a1-482e-a44d-82870293177f" (UID: "35bcc211-d8a1-482e-a44d-82870293177f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:03:56.313449 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.311099 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35bcc211-d8a1-482e-a44d-82870293177f-workload-certs" pod "35bcc211-d8a1-482e-a44d-82870293177f" (UID: "35bcc211-d8a1-482e-a44d-82870293177f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:03:56.319323 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.319281 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35bcc211-d8a1-482e-a44d-82870293177f-istio-envoy" pod "35bcc211-d8a1-482e-a44d-82870293177f" (UID: "35bcc211-d8a1-482e-a44d-82870293177f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:03:56.319601 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.319575 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35bcc211-d8a1-482e-a44d-82870293177f-istio-podinfo" pod "35bcc211-d8a1-482e-a44d-82870293177f" (UID: "35bcc211-d8a1-482e-a44d-82870293177f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:03:56.320847 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.320820 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35bcc211-d8a1-482e-a44d-82870293177f-kube-api-access-clrnm" pod "35bcc211-d8a1-482e-a44d-82870293177f" (UID: "35bcc211-d8a1-482e-a44d-82870293177f"). InnerVolumeSpecName "kube-api-access-clrnm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:03:56.324587 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.323536 4493 generic.go:356] "Generic (PLEG): container finished" podID="16b79973-7449-41cf-8709-e8109aa31cd2" containerID="54f2529a1477ea931be8854ab40ac7433fe8631fe8244c969ed07d8710e9c876" exitCode=0 Oct 30 17:03:56.324587 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.323622 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-8ddjc" event={"ID":"16b79973-7449-41cf-8709-e8109aa31cd2","Type":"ContainerDied","Data":"54f2529a1477ea931be8854ab40ac7433fe8631fe8244c969ed07d8710e9c876"} Oct 30 17:03:56.324587 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.323701 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-8ddjc" event={"ID":"16b79973-7449-41cf-8709-e8109aa31cd2","Type":"ContainerDied","Data":"62aa58d48188ed62dadcde085277ab110ec732d1bcd501c6c529e4a630f6a6b6"} Oct 30 17:03:56.324587 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.324150 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-8ddjc" Oct 30 17:03:56.324587 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.324479 4493 scope.go:122] "RemoveContainer" containerID="54f2529a1477ea931be8854ab40ac7433fe8631fe8244c969ed07d8710e9c876" Oct 30 17:03:56.326746 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.326726 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35bcc211-d8a1-482e-a44d-82870293177f-istio-token" pod "35bcc211-d8a1-482e-a44d-82870293177f" (UID: "35bcc211-d8a1-482e-a44d-82870293177f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:03:56.328178 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.328153 4493 generic.go:356] "Generic (PLEG): container finished" podID="35bcc211-d8a1-482e-a44d-82870293177f" containerID="7ac4160356b09b9789fedbbebf161b2650803404479f97d644d8dbf0141cc397" exitCode=0 Oct 30 17:03:56.329333 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.329317 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-jj9m9" Oct 30 17:03:56.329454 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.329431 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-jj9m9" event={"ID":"35bcc211-d8a1-482e-a44d-82870293177f","Type":"ContainerDied","Data":"7ac4160356b09b9789fedbbebf161b2650803404479f97d644d8dbf0141cc397"} Oct 30 17:03:56.330028 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.330008 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw" Oct 30 17:03:56.330119 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.330055 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-jj9m9" event={"ID":"35bcc211-d8a1-482e-a44d-82870293177f","Type":"ContainerDied","Data":"a6393aea6f1eb838f4888a71f79f9b6b13359d5da1e908ae913b71c5e4bc6d12"} Oct 30 17:03:56.331062 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.331039 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw" Oct 30 17:03:56.342315 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.342292 4493 scope.go:122] "RemoveContainer" containerID="43de03ac859c91924838516a621c6252fbcbe8e77f117f77f0eb5888cab5cb4a" Oct 30 17:03:56.357245 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.357173 4493 scope.go:122] "RemoveContainer" containerID="54f2529a1477ea931be8854ab40ac7433fe8631fe8244c969ed07d8710e9c876" Oct 30 17:03:56.357473 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:03:56.357445 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54f2529a1477ea931be8854ab40ac7433fe8631fe8244c969ed07d8710e9c876\": container with ID starting with 54f2529a1477ea931be8854ab40ac7433fe8631fe8244c969ed07d8710e9c876 not found: ID does not exist" containerID="54f2529a1477ea931be8854ab40ac7433fe8631fe8244c969ed07d8710e9c876" Oct 30 17:03:56.357516 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.357468 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54f2529a1477ea931be8854ab40ac7433fe8631fe8244c969ed07d8710e9c876"} err="failed to get container status \"54f2529a1477ea931be8854ab40ac7433fe8631fe8244c969ed07d8710e9c876\": rpc error: code = NotFound desc = could not find container \"54f2529a1477ea931be8854ab40ac7433fe8631fe8244c969ed07d8710e9c876\": container with ID starting with 54f2529a1477ea931be8854ab40ac7433fe8631fe8244c969ed07d8710e9c876 not found: ID does not exist" Oct 30 17:03:56.357516 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.357483 4493 scope.go:122] "RemoveContainer" containerID="43de03ac859c91924838516a621c6252fbcbe8e77f117f77f0eb5888cab5cb4a" Oct 30 17:03:56.357730 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:03:56.357702 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43de03ac859c91924838516a621c6252fbcbe8e77f117f77f0eb5888cab5cb4a\": container with ID starting with 43de03ac859c91924838516a621c6252fbcbe8e77f117f77f0eb5888cab5cb4a not found: ID does not exist" containerID="43de03ac859c91924838516a621c6252fbcbe8e77f117f77f0eb5888cab5cb4a" Oct 30 17:03:56.357774 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.357734 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43de03ac859c91924838516a621c6252fbcbe8e77f117f77f0eb5888cab5cb4a"} err="failed to get container status \"43de03ac859c91924838516a621c6252fbcbe8e77f117f77f0eb5888cab5cb4a\": rpc error: code = NotFound desc = could not find container \"43de03ac859c91924838516a621c6252fbcbe8e77f117f77f0eb5888cab5cb4a\": container with ID starting with 43de03ac859c91924838516a621c6252fbcbe8e77f117f77f0eb5888cab5cb4a not found: ID does not exist" Oct 30 17:03:56.357774 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.357745 4493 scope.go:122] "RemoveContainer" containerID="7ac4160356b09b9789fedbbebf161b2650803404479f97d644d8dbf0141cc397" Oct 30 17:03:56.368113 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.368081 4493 scope.go:122] "RemoveContainer" containerID="7ac4160356b09b9789fedbbebf161b2650803404479f97d644d8dbf0141cc397" Oct 30 17:03:56.368381 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:03:56.368361 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ac4160356b09b9789fedbbebf161b2650803404479f97d644d8dbf0141cc397\": container with ID starting with 7ac4160356b09b9789fedbbebf161b2650803404479f97d644d8dbf0141cc397 not found: ID does not exist" containerID="7ac4160356b09b9789fedbbebf161b2650803404479f97d644d8dbf0141cc397" Oct 30 17:03:56.368474 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.368387 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ac4160356b09b9789fedbbebf161b2650803404479f97d644d8dbf0141cc397"} err="failed to get container status \"7ac4160356b09b9789fedbbebf161b2650803404479f97d644d8dbf0141cc397\": rpc error: code = NotFound desc = could not find container \"7ac4160356b09b9789fedbbebf161b2650803404479f97d644d8dbf0141cc397\": container with ID starting with 7ac4160356b09b9789fedbbebf161b2650803404479f97d644d8dbf0141cc397 not found: ID does not exist" Oct 30 17:03:56.408778 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.408743 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16b79973-7449-41cf-8709-e8109aa31cd2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16b79973-7449-41cf-8709-e8109aa31cd2-istio-data\") pod \"16b79973-7449-41cf-8709-e8109aa31cd2\" (UID: \"16b79973-7449-41cf-8709-e8109aa31cd2\") " Oct 30 17:03:56.409944 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.409916 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16b79973-7449-41cf-8709-e8109aa31cd2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16b79973-7449-41cf-8709-e8109aa31cd2-workload-certs\") pod \"16b79973-7449-41cf-8709-e8109aa31cd2\" (UID: \"16b79973-7449-41cf-8709-e8109aa31cd2\") " Oct 30 17:03:56.410167 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.410151 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16b79973-7449-41cf-8709-e8109aa31cd2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16b79973-7449-41cf-8709-e8109aa31cd2-istio-envoy\") pod \"16b79973-7449-41cf-8709-e8109aa31cd2\" (UID: \"16b79973-7449-41cf-8709-e8109aa31cd2\") " Oct 30 17:03:56.410341 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.410312 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16b79973-7449-41cf-8709-e8109aa31cd2-kube-api-access-pp2l4\" (UniqueName: \"kubernetes.io/projected/16b79973-7449-41cf-8709-e8109aa31cd2-kube-api-access-pp2l4\") pod \"16b79973-7449-41cf-8709-e8109aa31cd2\" (UID: \"16b79973-7449-41cf-8709-e8109aa31cd2\") " Oct 30 17:03:56.410564 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.410550 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16b79973-7449-41cf-8709-e8109aa31cd2-istio-token\" (UniqueName: \"kubernetes.io/projected/16b79973-7449-41cf-8709-e8109aa31cd2-istio-token\") pod \"16b79973-7449-41cf-8709-e8109aa31cd2\" (UID: \"16b79973-7449-41cf-8709-e8109aa31cd2\") " Oct 30 17:03:56.411065 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.411044 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16b79973-7449-41cf-8709-e8109aa31cd2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16b79973-7449-41cf-8709-e8109aa31cd2-istiod-ca-cert\") pod \"16b79973-7449-41cf-8709-e8109aa31cd2\" (UID: \"16b79973-7449-41cf-8709-e8109aa31cd2\") " Oct 30 17:03:56.411849 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.411829 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16b79973-7449-41cf-8709-e8109aa31cd2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b79973-7449-41cf-8709-e8109aa31cd2-workload-socket\") pod \"16b79973-7449-41cf-8709-e8109aa31cd2\" (UID: \"16b79973-7449-41cf-8709-e8109aa31cd2\") " Oct 30 17:03:56.412269 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.412249 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/16b79973-7449-41cf-8709-e8109aa31cd2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16b79973-7449-41cf-8709-e8109aa31cd2-istio-podinfo\") pod \"16b79973-7449-41cf-8709-e8109aa31cd2\" (UID: \"16b79973-7449-41cf-8709-e8109aa31cd2\") " Oct 30 17:03:56.412420 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.412405 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16b79973-7449-41cf-8709-e8109aa31cd2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b79973-7449-41cf-8709-e8109aa31cd2-credential-socket\") pod \"16b79973-7449-41cf-8709-e8109aa31cd2\" (UID: \"16b79973-7449-41cf-8709-e8109aa31cd2\") " Oct 30 17:03:56.412641 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.409162 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16b79973-7449-41cf-8709-e8109aa31cd2-istio-data" pod "16b79973-7449-41cf-8709-e8109aa31cd2" (UID: "16b79973-7449-41cf-8709-e8109aa31cd2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:03:56.412711 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.410123 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16b79973-7449-41cf-8709-e8109aa31cd2-workload-certs" pod "16b79973-7449-41cf-8709-e8109aa31cd2" (UID: "16b79973-7449-41cf-8709-e8109aa31cd2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:03:56.412711 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.411698 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16b79973-7449-41cf-8709-e8109aa31cd2-istiod-ca-cert" pod "16b79973-7449-41cf-8709-e8109aa31cd2" (UID: "16b79973-7449-41cf-8709-e8109aa31cd2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:03:56.412711 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.412164 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16b79973-7449-41cf-8709-e8109aa31cd2-workload-socket" pod "16b79973-7449-41cf-8709-e8109aa31cd2" (UID: "16b79973-7449-41cf-8709-e8109aa31cd2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:03:56.412918 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.412866 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16b79973-7449-41cf-8709-e8109aa31cd2-istio-envoy" pod "16b79973-7449-41cf-8709-e8109aa31cd2" (UID: "16b79973-7449-41cf-8709-e8109aa31cd2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:03:56.412987 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.412948 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16b79973-7449-41cf-8709-e8109aa31cd2-credential-socket" pod "16b79973-7449-41cf-8709-e8109aa31cd2" (UID: "16b79973-7449-41cf-8709-e8109aa31cd2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:03:56.413683 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.413658 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16b79973-7449-41cf-8709-e8109aa31cd2-istio-data\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:03:56.413834 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.413687 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16b79973-7449-41cf-8709-e8109aa31cd2-workload-certs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:03:56.413834 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.413700 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16b79973-7449-41cf-8709-e8109aa31cd2-istio-envoy\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:03:56.413834 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.413713 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35bcc211-d8a1-482e-a44d-82870293177f-workload-certs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:03:56.413834 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.413725 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35bcc211-d8a1-482e-a44d-82870293177f-credential-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:03:56.413834 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.413737 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35bcc211-d8a1-482e-a44d-82870293177f-istio-token\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:03:56.413834 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.413749 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16b79973-7449-41cf-8709-e8109aa31cd2-istiod-ca-cert\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:03:56.413834 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.413762 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clrnm\" (UniqueName: \"kubernetes.io/projected/35bcc211-d8a1-482e-a44d-82870293177f-kube-api-access-clrnm\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:03:56.413834 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.413777 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35bcc211-d8a1-482e-a44d-82870293177f-istio-data\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:03:56.413834 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.413809 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b79973-7449-41cf-8709-e8109aa31cd2-workload-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:03:56.413834 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.413821 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16b79973-7449-41cf-8709-e8109aa31cd2-credential-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:03:56.413834 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.413835 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35bcc211-d8a1-482e-a44d-82870293177f-istio-envoy\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:03:56.414428 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.413849 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35bcc211-d8a1-482e-a44d-82870293177f-istio-podinfo\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:03:56.415559 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.415535 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/16b79973-7449-41cf-8709-e8109aa31cd2-istio-podinfo" pod "16b79973-7449-41cf-8709-e8109aa31cd2" (UID: "16b79973-7449-41cf-8709-e8109aa31cd2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:03:56.423358 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.417140 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16b79973-7449-41cf-8709-e8109aa31cd2-kube-api-access-pp2l4" pod "16b79973-7449-41cf-8709-e8109aa31cd2" (UID: "16b79973-7449-41cf-8709-e8109aa31cd2"). InnerVolumeSpecName "kube-api-access-pp2l4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:03:56.423358 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.419543 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16b79973-7449-41cf-8709-e8109aa31cd2-istio-token" pod "16b79973-7449-41cf-8709-e8109aa31cd2" (UID: "16b79973-7449-41cf-8709-e8109aa31cd2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:03:56.441151 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.441122 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-jj9m9"] Oct 30 17:03:56.445422 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.445390 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-jj9m9"] Oct 30 17:03:56.515234 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.515202 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16b79973-7449-41cf-8709-e8109aa31cd2-istio-podinfo\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:03:56.515234 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.515234 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp2l4\" (UniqueName: \"kubernetes.io/projected/16b79973-7449-41cf-8709-e8109aa31cd2-kube-api-access-pp2l4\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:03:56.515529 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.515252 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16b79973-7449-41cf-8709-e8109aa31cd2-istio-token\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:03:56.670993 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.670960 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-8ddjc"] Oct 30 17:03:56.674555 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.674528 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-8ddjc"] Oct 30 17:03:56.935035 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.934999 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16b79973-7449-41cf-8709-e8109aa31cd2" path="/var/lib/kubelet/pods/16b79973-7449-41cf-8709-e8109aa31cd2/volumes" Oct 30 17:03:56.935485 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:56.935470 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35bcc211-d8a1-482e-a44d-82870293177f" path="/var/lib/kubelet/pods/35bcc211-d8a1-482e-a44d-82870293177f/volumes" Oct 30 17:03:57.032589 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:57.032557 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bddff506-spjzp"] Oct 30 17:03:57.035592 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:57.035570 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bddff506-spjzp"] Oct 30 17:03:58.996663 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:03:58.996606 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fbae41c-770a-4146-b445-417b90cde0dd" path="/var/lib/kubelet/pods/0fbae41c-770a-4146-b445-417b90cde0dd/volumes" Oct 30 17:04:00.343921 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:04:00.343881 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 17:04:07.992009 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:04:07.991934 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 17:04:14.126057 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:04:14.126026 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bc58j42p"] Oct 30 17:04:14.126729 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:04:14.126708 4493 cpu_manager.go:414] "containerMap: removing container" podUID="16b79973-7449-41cf-8709-e8109aa31cd2" containerName="istio-proxy" Oct 30 17:04:14.126887 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:04:14.126729 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16b79973-7449-41cf-8709-e8109aa31cd2" containerName="istio-proxy" Oct 30 17:04:14.126887 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:04:14.126758 4493 cpu_manager.go:414] "containerMap: removing container" podUID="35bcc211-d8a1-482e-a44d-82870293177f" containerName="istio-proxy" Oct 30 17:04:14.126887 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:04:14.126766 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35bcc211-d8a1-482e-a44d-82870293177f" containerName="istio-proxy" Oct 30 17:04:14.127049 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:04:14.126968 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="16b79973-7449-41cf-8709-e8109aa31cd2" containerName="istio-proxy" Oct 30 17:04:14.127049 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:04:14.126985 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="35bcc211-d8a1-482e-a44d-82870293177f" containerName="istio-proxy" Oct 30 17:04:14.130477 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:04:14.130457 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bc58j42p" Oct 30 17:04:14.145008 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:04:14.144981 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-13c495\"/\"agc-cred-a445c4f60b33\"" Oct 30 17:04:14.145295 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:04:14.145269 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-13c495\"/\"openshift-service-ca.crt\"" Oct 30 17:04:14.145435 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:04:14.145412 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-13c495\"/\"agc-cred-4bf9743d880e\"" Oct 30 17:04:14.145435 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:04:14.145350 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-13c495\"/\"routing-overlay\"" Oct 30 17:04:14.145587 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:04:14.145356 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-13c495\"/\"payload-processing-external-model-plugins-e2e-extmod-b-13c495\"" Oct 30 17:04:14.145587 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:04:14.145356 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-13c495\"/\"payload-processing-external-model-e2e--be0deb1b-dockercfg-vxb7r\"" Oct 30 17:04:14.145587 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:04:14.145328 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-13c495\"/\"kube-root-ca.crt\"" Oct 30 17:04:14.152911 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:04:14.152892 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bc58j42p"] Oct 30 17:04:14.184929 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:04:14.184890 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/82f0f9de-fb45-48b1-9a13-9d9149c893a3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-13c495-6bc58j42p\" (UID: \"82f0f9de-fb45-48b1-9a13-9d9149c893a3\") " pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bc58j42p" Oct 30 17:04:14.185147 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:04:14.184956 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/82f0f9de-fb45-48b1-9a13-9d9149c893a3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-13c495-6bc58j42p\" (UID: \"82f0f9de-fb45-48b1-9a13-9d9149c893a3\") " pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bc58j42p" Oct 30 17:04:14.185147 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:04:14.185080 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82f0f9de-fb45-48b1-9a13-9d9149c893a3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-13c495-6bc58j42p\" (UID: \"82f0f9de-fb45-48b1-9a13-9d9149c893a3\") " pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bc58j42p" Oct 30 17:04:14.185288 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:04:14.185265 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljllb\" (UniqueName: \"kubernetes.io/projected/82f0f9de-fb45-48b1-9a13-9d9149c893a3-kube-api-access-ljllb\") pod \"payload-processing-external-model-e2e-extmod-b-13c495-6bc58j42p\" (UID: \"82f0f9de-fb45-48b1-9a13-9d9149c893a3\") " pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bc58j42p" Oct 30 17:04:14.288590 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:04:14.287879 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/82f0f9de-fb45-48b1-9a13-9d9149c893a3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-13c495-6bc58j42p\" (UID: \"82f0f9de-fb45-48b1-9a13-9d9149c893a3\") " pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bc58j42p" Oct 30 17:04:14.288590 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:04:14.287970 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/82f0f9de-fb45-48b1-9a13-9d9149c893a3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-13c495-6bc58j42p\" (UID: \"82f0f9de-fb45-48b1-9a13-9d9149c893a3\") " pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bc58j42p" Oct 30 17:04:14.288590 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:04:14.288039 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82f0f9de-fb45-48b1-9a13-9d9149c893a3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-13c495-6bc58j42p\" (UID: \"82f0f9de-fb45-48b1-9a13-9d9149c893a3\") " pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bc58j42p" Oct 30 17:04:14.288590 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:04:14.288243 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljllb\" (UniqueName: \"kubernetes.io/projected/82f0f9de-fb45-48b1-9a13-9d9149c893a3-kube-api-access-ljllb\") pod \"payload-processing-external-model-e2e-extmod-b-13c495-6bc58j42p\" (UID: \"82f0f9de-fb45-48b1-9a13-9d9149c893a3\") " pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bc58j42p" Oct 30 17:04:14.289884 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:04:14.289850 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/82f0f9de-fb45-48b1-9a13-9d9149c893a3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-13c495-6bc58j42p\" (UID: \"82f0f9de-fb45-48b1-9a13-9d9149c893a3\") " pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bc58j42p" Oct 30 17:04:14.290167 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:04:14.290144 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82f0f9de-fb45-48b1-9a13-9d9149c893a3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-13c495-6bc58j42p\" (UID: \"82f0f9de-fb45-48b1-9a13-9d9149c893a3\") " pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bc58j42p" Oct 30 17:04:14.292670 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:04:14.292650 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/82f0f9de-fb45-48b1-9a13-9d9149c893a3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-13c495-6bc58j42p\" (UID: \"82f0f9de-fb45-48b1-9a13-9d9149c893a3\") " pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bc58j42p" Oct 30 17:04:14.311480 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:04:14.311457 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljllb\" (UniqueName: \"kubernetes.io/projected/82f0f9de-fb45-48b1-9a13-9d9149c893a3-kube-api-access-ljllb\") pod \"payload-processing-external-model-e2e-extmod-b-13c495-6bc58j42p\" (UID: \"82f0f9de-fb45-48b1-9a13-9d9149c893a3\") " pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bc58j42p" Oct 30 17:04:14.447149 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:04:14.447113 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bc58j42p" Oct 30 17:04:14.753006 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:04:14.752966 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bc58j42p"] Oct 30 17:04:14.754543 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:04:14.754514 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82f0f9de_fb45_48b1_9a13_9d9149c893a3.slice/crio-0726f68122bddca26e20da2a9c436de89d930ccef22cbf04fa4dbe31d2261e2d WatchSource:0}: Error finding container 0726f68122bddca26e20da2a9c436de89d930ccef22cbf04fa4dbe31d2261e2d: Status 404 returned error can't find the container with id 0726f68122bddca26e20da2a9c436de89d930ccef22cbf04fa4dbe31d2261e2d Oct 30 17:04:15.486416 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:04:15.486380 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bc58j42p" event={"ID":"82f0f9de-fb45-48b1-9a13-9d9149c893a3","Type":"ContainerStarted","Data":"39cb612607a51558dc58245e07720e8f26880794068e32614311d68b8342859b"} Oct 30 17:04:15.487167 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:04:15.486430 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bc58j42p" event={"ID":"82f0f9de-fb45-48b1-9a13-9d9149c893a3","Type":"ContainerStarted","Data":"0726f68122bddca26e20da2a9c436de89d930ccef22cbf04fa4dbe31d2261e2d"} Oct 30 17:04:15.487167 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:04:15.486512 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bc58j42p" Oct 30 17:04:15.529523 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:04:15.529470 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bc58j42p" podStartSLOduration=1.5294537799999999 podStartE2EDuration="1.52945378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:04:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:04:15.527181084 +0000 UTC m=+2482.754356185" watchObservedRunningTime="2026-10-30 17:04:15.52945378 +0000 UTC m=+2482.756628881" Oct 30 17:04:18.099660 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:04:18.099627 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60250cfb-qq4l5"] Oct 30 17:04:18.112035 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:04:18.112002 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60250cfb-qq4l5"] Oct 30 17:04:18.935702 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:04:18.935659 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebe98f2d-afe7-4684-a56e-5479ec69a6fa" path="/var/lib/kubelet/pods/ebe98f2d-afe7-4684-a56e-5479ec69a6fa/volumes" Oct 30 17:04:21.532273 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:04:21.532243 4493 generic.go:356] "Generic (PLEG): container finished" podID="3319a971-2f82-469a-8fc3-20687e710719" containerID="5efa55b141ec7f8db1da67e4712ca26a87c4fbe87d9df68e5a0d94b5ba6ccc52" exitCode=1 Oct 30 17:04:21.532718 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:04:21.532321 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-h6wgm" event={"ID":"3319a971-2f82-469a-8fc3-20687e710719","Type":"ContainerDied","Data":"5efa55b141ec7f8db1da67e4712ca26a87c4fbe87d9df68e5a0d94b5ba6ccc52"} Oct 30 17:04:21.532837 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:04:21.532743 4493 scope.go:122] "RemoveContainer" containerID="5efa55b141ec7f8db1da67e4712ca26a87c4fbe87d9df68e5a0d94b5ba6ccc52" Oct 30 17:04:22.541449 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:04:22.541416 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-h6wgm" event={"ID":"3319a971-2f82-469a-8fc3-20687e710719","Type":"ContainerStarted","Data":"970fca18c2417a139dd37df1bf79abb5771f7471f864f1abacd9b41fd180a5e8"} Oct 30 17:04:22.541876 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:04:22.541621 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-h6wgm" Oct 30 17:04:26.497071 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:04:26.497043 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bc58j42p" Oct 30 17:04:28.032518 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:04:28.032485 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d85db049-snhzz"] Oct 30 17:04:28.043825 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:04:28.043796 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d85db049-snhzz"] Oct 30 17:04:28.935960 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:04:28.935925 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="540a4a45-3790-4207-b753-ce03bafbe504" path="/var/lib/kubelet/pods/540a4a45-3790-4207-b753-ce03bafbe504/volumes" Oct 30 17:04:30.343183 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:04:30.343140 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 17:04:33.551173 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:04:33.551140 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-h6wgm" Oct 30 17:04:56.369368 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:04:56.369338 4493 scope.go:122] "RemoveContainer" containerID="e76985d62da397e085713d03d351cb09a96ee428ba73ef5e00b42f6b9d58a1fe" Oct 30 17:04:56.379569 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:04:56.379549 4493 scope.go:122] "RemoveContainer" containerID="f6aa55b50e871993f97da2007ea80c850e9d63ebc1fd3f200c62a2561b136a34" Oct 30 17:04:56.389409 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:04:56.389391 4493 scope.go:122] "RemoveContainer" containerID="6ec5b3263054683739d362a78303564ef55d57aaa8cd44fc650107acb9c5462e" Oct 30 17:04:56.401906 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:04:56.401364 4493 scope.go:122] "RemoveContainer" containerID="c218947ab011597840bde8598c1ec55ace97574b7cad4e41d565bf459f5e27a0" Oct 30 17:04:57.914806 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:04:57.908797 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 17:05:00.359070 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:05:00.359034 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 17:05:10.940378 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:10.940337 4493 generic.go:356] "Generic (PLEG): container finished" podID="e7475582-02dd-456b-a5d5-d894755aba74" containerID="c723df0cc3d721fbe627362f65e2b31c4f3b447d38484c601f0b610ff0fc7e58" exitCode=0 Oct 30 17:05:10.940866 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:10.940385 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw" event={"ID":"e7475582-02dd-456b-a5d5-d894755aba74","Type":"ContainerDied","Data":"c723df0cc3d721fbe627362f65e2b31c4f3b447d38484c601f0b610ff0fc7e58"} Oct 30 17:05:10.940866 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:10.940857 4493 scope.go:122] "RemoveContainer" containerID="c723df0cc3d721fbe627362f65e2b31c4f3b447d38484c601f0b610ff0fc7e58" Oct 30 17:05:10.942705 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:10.942673 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:05:10.942850 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:10.942764 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:05:10.942850 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:10.942840 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:05:11.329187 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:11.329095 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw" Oct 30 17:05:11.948996 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:11.948954 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw" event={"ID":"e7475582-02dd-456b-a5d5-d894755aba74","Type":"ContainerStarted","Data":"0f134a4f15ef6855db28d0d167317cb30d32595258bd3c610b5eb8266b17d493"} Oct 30 17:05:12.732845 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:12.732806 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw" Oct 30 17:05:12.734210 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:12.734186 4493 patch_prober.go:29] interesting pod/e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body= Oct 30 17:05:12.734340 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:12.734267 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw" podUID="e7475582-02dd-456b-a5d5-d894755aba74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" Oct 30 17:05:13.735220 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:13.735172 4493 patch_prober.go:29] interesting pod/e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body= Oct 30 17:05:13.735746 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:13.735250 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw" podUID="e7475582-02dd-456b-a5d5-d894755aba74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" Oct 30 17:05:14.733380 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:14.733344 4493 patch_prober.go:29] interesting pod/e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body= Oct 30 17:05:14.733600 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:14.733409 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw" podUID="e7475582-02dd-456b-a5d5-d894755aba74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" Oct 30 17:05:15.732720 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:15.732680 4493 patch_prober.go:29] interesting pod/e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body= Oct 30 17:05:15.733146 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:15.732742 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw" podUID="e7475582-02dd-456b-a5d5-d894755aba74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" Oct 30 17:05:15.940856 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:15.940818 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d1a2471-lfwx6"] Oct 30 17:05:15.941484 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:15.941468 4493 cpu_manager.go:414] "containerMap: removing container" podUID="16b79973-7449-41cf-8709-e8109aa31cd2" containerName="istio-proxy" Oct 30 17:05:15.941546 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:15.941485 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16b79973-7449-41cf-8709-e8109aa31cd2" containerName="istio-proxy" Oct 30 17:05:15.941595 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:15.941586 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="16b79973-7449-41cf-8709-e8109aa31cd2" containerName="istio-proxy" Oct 30 17:05:15.944073 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:15.944057 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d1a2471-lfwx6" Oct 30 17:05:15.970391 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:15.970288 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:05:15.970391 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:15.970364 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 17:05:15.970763 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:15.970741 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:05:15.974207 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:15.974184 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d1a2471-lfwx6"] Oct 30 17:05:16.061659 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:16.061591 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:05:16.136496 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:16.136460 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h8ls\" (UniqueName: \"kubernetes.io/projected/dab6fd8b-e841-4e04-889f-5d5c35a9cdba-kube-api-access-9h8ls\") pod \"maas-api-revoke-sub-2d1a2471-lfwx6\" (UID: \"dab6fd8b-e841-4e04-889f-5d5c35a9cdba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d1a2471-lfwx6" Oct 30 17:05:16.136924 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:16.136906 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dab6fd8b-e841-4e04-889f-5d5c35a9cdba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d1a2471-lfwx6\" (UID: \"dab6fd8b-e841-4e04-889f-5d5c35a9cdba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d1a2471-lfwx6" Oct 30 17:05:16.137116 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:16.137098 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dab6fd8b-e841-4e04-889f-5d5c35a9cdba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d1a2471-lfwx6\" (UID: \"dab6fd8b-e841-4e04-889f-5d5c35a9cdba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d1a2471-lfwx6" Oct 30 17:05:16.238119 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:16.238090 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h8ls\" (UniqueName: \"kubernetes.io/projected/dab6fd8b-e841-4e04-889f-5d5c35a9cdba-kube-api-access-9h8ls\") pod \"maas-api-revoke-sub-2d1a2471-lfwx6\" (UID: \"dab6fd8b-e841-4e04-889f-5d5c35a9cdba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d1a2471-lfwx6" Oct 30 17:05:16.238295 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:16.238195 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dab6fd8b-e841-4e04-889f-5d5c35a9cdba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d1a2471-lfwx6\" (UID: \"dab6fd8b-e841-4e04-889f-5d5c35a9cdba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d1a2471-lfwx6" Oct 30 17:05:16.238295 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:16.238251 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dab6fd8b-e841-4e04-889f-5d5c35a9cdba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d1a2471-lfwx6\" (UID: \"dab6fd8b-e841-4e04-889f-5d5c35a9cdba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d1a2471-lfwx6" Oct 30 17:05:16.238940 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:16.238917 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dab6fd8b-e841-4e04-889f-5d5c35a9cdba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d1a2471-lfwx6\" (UID: \"dab6fd8b-e841-4e04-889f-5d5c35a9cdba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d1a2471-lfwx6" Oct 30 17:05:16.242391 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:16.242262 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dab6fd8b-e841-4e04-889f-5d5c35a9cdba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d1a2471-lfwx6\" (UID: \"dab6fd8b-e841-4e04-889f-5d5c35a9cdba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d1a2471-lfwx6" Oct 30 17:05:16.261543 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:16.261518 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h8ls\" (UniqueName: \"kubernetes.io/projected/dab6fd8b-e841-4e04-889f-5d5c35a9cdba-kube-api-access-9h8ls\") pod \"maas-api-revoke-sub-2d1a2471-lfwx6\" (UID: \"dab6fd8b-e841-4e04-889f-5d5c35a9cdba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d1a2471-lfwx6" Oct 30 17:05:16.559459 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:16.559427 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d1a2471-lfwx6" Oct 30 17:05:16.734809 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:16.734301 4493 patch_prober.go:29] interesting pod/e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body= Oct 30 17:05:16.734809 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:16.734363 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw" podUID="e7475582-02dd-456b-a5d5-d894755aba74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" Oct 30 17:05:16.897253 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:16.897219 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d1a2471-lfwx6"] Oct 30 17:05:16.902520 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:05:16.902493 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddab6fd8b_e841_4e04_889f_5d5c35a9cdba.slice/crio-a2193c9dd5f82be9af5449fb16044f3330ff07dbb0481e77ac5c8fe3f2fa8dcc WatchSource:0}: Error finding container a2193c9dd5f82be9af5449fb16044f3330ff07dbb0481e77ac5c8fe3f2fa8dcc: Status 404 returned error can't find the container with id a2193c9dd5f82be9af5449fb16044f3330ff07dbb0481e77ac5c8fe3f2fa8dcc Oct 30 17:05:17.051431 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:17.051395 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d1a2471-lfwx6" event={"ID":"dab6fd8b-e841-4e04-889f-5d5c35a9cdba","Type":"ContainerStarted","Data":"a2193c9dd5f82be9af5449fb16044f3330ff07dbb0481e77ac5c8fe3f2fa8dcc"} Oct 30 17:05:17.738322 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:17.738281 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw" Oct 30 17:05:17.738718 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:17.738689 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw" Oct 30 17:05:17.739421 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:17.739398 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw" Oct 30 17:05:18.060151 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:18.060062 4493 generic.go:356] "Generic (PLEG): container finished" podID="dab6fd8b-e841-4e04-889f-5d5c35a9cdba" containerID="ba667739adfdec4ead3887b6573ad1accb61190d81df09ebbfbe9cfbeb33c3f8" exitCode=0 Oct 30 17:05:18.060315 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:18.060164 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d1a2471-lfwx6" event={"ID":"dab6fd8b-e841-4e04-889f-5d5c35a9cdba","Type":"ContainerDied","Data":"ba667739adfdec4ead3887b6573ad1accb61190d81df09ebbfbe9cfbeb33c3f8"} Oct 30 17:05:19.236522 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:19.236496 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d1a2471-lfwx6" Oct 30 17:05:19.285769 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:19.285740 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dab6fd8b-e841-4e04-889f-5d5c35a9cdba-kube-api-access-9h8ls\" (UniqueName: \"kubernetes.io/projected/dab6fd8b-e841-4e04-889f-5d5c35a9cdba-kube-api-access-9h8ls\") pod \"dab6fd8b-e841-4e04-889f-5d5c35a9cdba\" (UID: \"dab6fd8b-e841-4e04-889f-5d5c35a9cdba\") " Oct 30 17:05:19.285979 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:19.285862 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dab6fd8b-e841-4e04-889f-5d5c35a9cdba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dab6fd8b-e841-4e04-889f-5d5c35a9cdba-maas-api-openshift-ca\") pod \"dab6fd8b-e841-4e04-889f-5d5c35a9cdba\" (UID: \"dab6fd8b-e841-4e04-889f-5d5c35a9cdba\") " Oct 30 17:05:19.286544 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:19.286140 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dab6fd8b-e841-4e04-889f-5d5c35a9cdba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dab6fd8b-e841-4e04-889f-5d5c35a9cdba-maas-api-xks-ca\") pod \"dab6fd8b-e841-4e04-889f-5d5c35a9cdba\" (UID: \"dab6fd8b-e841-4e04-889f-5d5c35a9cdba\") " Oct 30 17:05:19.286544 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:19.286246 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dab6fd8b-e841-4e04-889f-5d5c35a9cdba-maas-api-openshift-ca" pod "dab6fd8b-e841-4e04-889f-5d5c35a9cdba" (UID: "dab6fd8b-e841-4e04-889f-5d5c35a9cdba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:05:19.286745 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:19.286697 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dab6fd8b-e841-4e04-889f-5d5c35a9cdba-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:05:19.288507 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:19.288481 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dab6fd8b-e841-4e04-889f-5d5c35a9cdba-kube-api-access-9h8ls" pod "dab6fd8b-e841-4e04-889f-5d5c35a9cdba" (UID: "dab6fd8b-e841-4e04-889f-5d5c35a9cdba"). InnerVolumeSpecName "kube-api-access-9h8ls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:05:19.288837 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:19.288813 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dab6fd8b-e841-4e04-889f-5d5c35a9cdba-maas-api-xks-ca" pod "dab6fd8b-e841-4e04-889f-5d5c35a9cdba" (UID: "dab6fd8b-e841-4e04-889f-5d5c35a9cdba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:05:19.388035 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:19.387956 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9h8ls\" (UniqueName: \"kubernetes.io/projected/dab6fd8b-e841-4e04-889f-5d5c35a9cdba-kube-api-access-9h8ls\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:05:19.388035 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:19.387990 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dab6fd8b-e841-4e04-889f-5d5c35a9cdba-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:05:20.083824 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:20.079727 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d1a2471-lfwx6" event={"ID":"dab6fd8b-e841-4e04-889f-5d5c35a9cdba","Type":"ContainerDied","Data":"a2193c9dd5f82be9af5449fb16044f3330ff07dbb0481e77ac5c8fe3f2fa8dcc"} Oct 30 17:05:20.083824 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:20.079776 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2193c9dd5f82be9af5449fb16044f3330ff07dbb0481e77ac5c8fe3f2fa8dcc" Oct 30 17:05:20.083824 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:20.079853 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d1a2471-lfwx6" Oct 30 17:05:25.895809 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:25.894879 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz"] Oct 30 17:05:25.895809 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:25.895667 4493 cpu_manager.go:414] "containerMap: removing container" podUID="dab6fd8b-e841-4e04-889f-5d5c35a9cdba" containerName="revoke-keys" Oct 30 17:05:25.895809 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:25.895681 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dab6fd8b-e841-4e04-889f-5d5c35a9cdba" containerName="revoke-keys" Oct 30 17:05:25.896278 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:25.895866 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="dab6fd8b-e841-4e04-889f-5d5c35a9cdba" containerName="revoke-keys" Oct 30 17:05:25.899826 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:25.899772 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz" Oct 30 17:05:25.915863 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:25.915772 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-43197b-openshift-default-dockercfg-qh2xc\"" Oct 30 17:05:25.936888 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:25.936855 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz"] Oct 30 17:05:25.989748 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:25.989715 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/009a69cb-8246-4711-a31d-37d9ee856749-istio-podinfo\") pod \"e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz\" (UID: \"009a69cb-8246-4711-a31d-37d9ee856749\") " pod="openshift-ingress/e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz" Oct 30 17:05:25.989748 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:25.989750 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/009a69cb-8246-4711-a31d-37d9ee856749-istio-token\") pod \"e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz\" (UID: \"009a69cb-8246-4711-a31d-37d9ee856749\") " pod="openshift-ingress/e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz" Oct 30 17:05:25.990024 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:25.989922 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/009a69cb-8246-4711-a31d-37d9ee856749-istiod-ca-cert\") pod \"e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz\" (UID: \"009a69cb-8246-4711-a31d-37d9ee856749\") " pod="openshift-ingress/e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz" Oct 30 17:05:25.990024 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:25.989958 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vckfw\" (UniqueName: \"kubernetes.io/projected/009a69cb-8246-4711-a31d-37d9ee856749-kube-api-access-vckfw\") pod \"e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz\" (UID: \"009a69cb-8246-4711-a31d-37d9ee856749\") " pod="openshift-ingress/e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz" Oct 30 17:05:25.990145 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:25.990119 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/009a69cb-8246-4711-a31d-37d9ee856749-workload-certs\") pod \"e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz\" (UID: \"009a69cb-8246-4711-a31d-37d9ee856749\") " pod="openshift-ingress/e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz" Oct 30 17:05:25.990214 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:25.990176 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/009a69cb-8246-4711-a31d-37d9ee856749-istio-envoy\") pod \"e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz\" (UID: \"009a69cb-8246-4711-a31d-37d9ee856749\") " pod="openshift-ingress/e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz" Oct 30 17:05:25.990430 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:25.990404 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/009a69cb-8246-4711-a31d-37d9ee856749-credential-socket\") pod \"e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz\" (UID: \"009a69cb-8246-4711-a31d-37d9ee856749\") " pod="openshift-ingress/e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz" Oct 30 17:05:25.990552 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:25.990460 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/009a69cb-8246-4711-a31d-37d9ee856749-workload-socket\") pod \"e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz\" (UID: \"009a69cb-8246-4711-a31d-37d9ee856749\") " pod="openshift-ingress/e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz" Oct 30 17:05:25.990552 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:25.990530 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/009a69cb-8246-4711-a31d-37d9ee856749-istio-data\") pod \"e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz\" (UID: \"009a69cb-8246-4711-a31d-37d9ee856749\") " pod="openshift-ingress/e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz" Oct 30 17:05:26.091720 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:26.091678 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/009a69cb-8246-4711-a31d-37d9ee856749-credential-socket\") pod \"e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz\" (UID: \"009a69cb-8246-4711-a31d-37d9ee856749\") " pod="openshift-ingress/e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz" Oct 30 17:05:26.091938 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:26.091739 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/009a69cb-8246-4711-a31d-37d9ee856749-workload-socket\") pod \"e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz\" (UID: \"009a69cb-8246-4711-a31d-37d9ee856749\") " pod="openshift-ingress/e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz" Oct 30 17:05:26.091938 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:26.091777 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/009a69cb-8246-4711-a31d-37d9ee856749-istio-data\") pod \"e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz\" (UID: \"009a69cb-8246-4711-a31d-37d9ee856749\") " pod="openshift-ingress/e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz" Oct 30 17:05:26.091938 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:26.091924 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/009a69cb-8246-4711-a31d-37d9ee856749-istio-podinfo\") pod \"e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz\" (UID: \"009a69cb-8246-4711-a31d-37d9ee856749\") " pod="openshift-ingress/e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz" Oct 30 17:05:26.092129 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:26.091949 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/009a69cb-8246-4711-a31d-37d9ee856749-istio-token\") pod \"e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz\" (UID: \"009a69cb-8246-4711-a31d-37d9ee856749\") " pod="openshift-ingress/e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz" Oct 30 17:05:26.092129 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:26.092005 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/009a69cb-8246-4711-a31d-37d9ee856749-istiod-ca-cert\") pod \"e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz\" (UID: \"009a69cb-8246-4711-a31d-37d9ee856749\") " pod="openshift-ingress/e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz" Oct 30 17:05:26.092129 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:26.092058 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vckfw\" (UniqueName: \"kubernetes.io/projected/009a69cb-8246-4711-a31d-37d9ee856749-kube-api-access-vckfw\") pod \"e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz\" (UID: \"009a69cb-8246-4711-a31d-37d9ee856749\") " pod="openshift-ingress/e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz" Oct 30 17:05:26.092129 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:26.092102 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/009a69cb-8246-4711-a31d-37d9ee856749-workload-certs\") pod \"e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz\" (UID: \"009a69cb-8246-4711-a31d-37d9ee856749\") " pod="openshift-ingress/e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz" Oct 30 17:05:26.092346 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:26.092133 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/009a69cb-8246-4711-a31d-37d9ee856749-credential-socket\") pod \"e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz\" (UID: \"009a69cb-8246-4711-a31d-37d9ee856749\") " pod="openshift-ingress/e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz" Oct 30 17:05:26.092346 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:26.092140 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/009a69cb-8246-4711-a31d-37d9ee856749-istio-envoy\") pod \"e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz\" (UID: \"009a69cb-8246-4711-a31d-37d9ee856749\") " pod="openshift-ingress/e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz" Oct 30 17:05:26.092346 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:26.092181 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/009a69cb-8246-4711-a31d-37d9ee856749-workload-socket\") pod \"e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz\" (UID: \"009a69cb-8246-4711-a31d-37d9ee856749\") " pod="openshift-ingress/e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz" Oct 30 17:05:26.092346 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:26.092278 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/009a69cb-8246-4711-a31d-37d9ee856749-istio-data\") pod \"e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz\" (UID: \"009a69cb-8246-4711-a31d-37d9ee856749\") " pod="openshift-ingress/e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz" Oct 30 17:05:26.092570 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:26.092511 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/009a69cb-8246-4711-a31d-37d9ee856749-workload-certs\") pod \"e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz\" (UID: \"009a69cb-8246-4711-a31d-37d9ee856749\") " pod="openshift-ingress/e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz" Oct 30 17:05:26.093109 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:26.093085 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/009a69cb-8246-4711-a31d-37d9ee856749-istiod-ca-cert\") pod \"e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz\" (UID: \"009a69cb-8246-4711-a31d-37d9ee856749\") " pod="openshift-ingress/e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz" Oct 30 17:05:26.094997 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:26.094957 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/009a69cb-8246-4711-a31d-37d9ee856749-istio-envoy\") pod \"e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz\" (UID: \"009a69cb-8246-4711-a31d-37d9ee856749\") " pod="openshift-ingress/e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz" Oct 30 17:05:26.095182 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:26.095158 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/009a69cb-8246-4711-a31d-37d9ee856749-istio-podinfo\") pod \"e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz\" (UID: \"009a69cb-8246-4711-a31d-37d9ee856749\") " pod="openshift-ingress/e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz" Oct 30 17:05:26.116347 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:26.116198 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vckfw\" (UniqueName: \"kubernetes.io/projected/009a69cb-8246-4711-a31d-37d9ee856749-kube-api-access-vckfw\") pod \"e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz\" (UID: \"009a69cb-8246-4711-a31d-37d9ee856749\") " pod="openshift-ingress/e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz" Oct 30 17:05:26.116484 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:26.116374 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/009a69cb-8246-4711-a31d-37d9ee856749-istio-token\") pod \"e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz\" (UID: \"009a69cb-8246-4711-a31d-37d9ee856749\") " pod="openshift-ingress/e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz" Oct 30 17:05:26.230857 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:26.228036 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz" Oct 30 17:05:26.654241 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:26.654207 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz"] Oct 30 17:05:26.658668 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:05:26.658636 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod009a69cb_8246_4711_a31d_37d9ee856749.slice/crio-d08654755ce83d0707c6c5d5df7ea4886eaef4654d37a3eabaa2a80d9a21a7ac WatchSource:0}: Error finding container d08654755ce83d0707c6c5d5df7ea4886eaef4654d37a3eabaa2a80d9a21a7ac: Status 404 returned error can't find the container with id d08654755ce83d0707c6c5d5df7ea4886eaef4654d37a3eabaa2a80d9a21a7ac Oct 30 17:05:26.662096 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:26.662056 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:05:26.662173 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:26.662153 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:05:26.662232 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:26.662208 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:05:27.136860 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:27.135947 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz" event={"ID":"009a69cb-8246-4711-a31d-37d9ee856749","Type":"ContainerStarted","Data":"891f9c6a6b5517919f373e475fa945315bea74d1b1e5833dd00eea5670c0dde6"} Oct 30 17:05:27.136860 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:27.136016 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz" event={"ID":"009a69cb-8246-4711-a31d-37d9ee856749","Type":"ContainerStarted","Data":"d08654755ce83d0707c6c5d5df7ea4886eaef4654d37a3eabaa2a80d9a21a7ac"} Oct 30 17:05:27.183539 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:27.183489 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz" podStartSLOduration=2.183474687 podStartE2EDuration="2.183474687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:05:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:05:27.181260669 +0000 UTC m=+2554.408435769" watchObservedRunningTime="2026-10-30 17:05:27.183474687 +0000 UTC m=+2554.410649790" Oct 30 17:05:27.228889 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:27.228845 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz" Oct 30 17:05:27.235628 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:27.235601 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz" Oct 30 17:05:28.157808 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:28.156776 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz" Oct 30 17:05:28.164714 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:28.164469 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz" Oct 30 17:05:30.342581 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:05:30.342539 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 17:05:30.389560 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:30.389525 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6301a11a-w4mbz"] Oct 30 17:05:30.393689 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:30.393672 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6301a11a-w4mbz" Oct 30 17:05:30.411117 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:30.411092 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:05:30.415900 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:30.415876 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6301a11a-w4mbz"] Oct 30 17:05:30.424833 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:30.423166 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:05:30.424833 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:30.423244 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 17:05:30.424833 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:30.423336 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:05:30.490200 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:30.490170 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/664a4424-09d2-463a-a9ce-be604190999f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6301a11a-w4mbz\" (UID: \"664a4424-09d2-463a-a9ce-be604190999f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6301a11a-w4mbz" Oct 30 17:05:30.490356 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:30.490246 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b8r8\" (UniqueName: \"kubernetes.io/projected/664a4424-09d2-463a-a9ce-be604190999f-kube-api-access-6b8r8\") pod \"maas-api-revoke-sub-6301a11a-w4mbz\" (UID: \"664a4424-09d2-463a-a9ce-be604190999f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6301a11a-w4mbz" Oct 30 17:05:30.490425 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:30.490403 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/664a4424-09d2-463a-a9ce-be604190999f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6301a11a-w4mbz\" (UID: \"664a4424-09d2-463a-a9ce-be604190999f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6301a11a-w4mbz" Oct 30 17:05:30.591797 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:30.591744 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/664a4424-09d2-463a-a9ce-be604190999f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6301a11a-w4mbz\" (UID: \"664a4424-09d2-463a-a9ce-be604190999f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6301a11a-w4mbz" Oct 30 17:05:30.591967 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:30.591836 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b8r8\" (UniqueName: \"kubernetes.io/projected/664a4424-09d2-463a-a9ce-be604190999f-kube-api-access-6b8r8\") pod \"maas-api-revoke-sub-6301a11a-w4mbz\" (UID: \"664a4424-09d2-463a-a9ce-be604190999f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6301a11a-w4mbz" Oct 30 17:05:30.591967 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:30.591916 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/664a4424-09d2-463a-a9ce-be604190999f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6301a11a-w4mbz\" (UID: \"664a4424-09d2-463a-a9ce-be604190999f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6301a11a-w4mbz" Oct 30 17:05:30.592382 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:30.592360 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/664a4424-09d2-463a-a9ce-be604190999f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6301a11a-w4mbz\" (UID: \"664a4424-09d2-463a-a9ce-be604190999f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6301a11a-w4mbz" Oct 30 17:05:30.594397 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:30.594334 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/664a4424-09d2-463a-a9ce-be604190999f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6301a11a-w4mbz\" (UID: \"664a4424-09d2-463a-a9ce-be604190999f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6301a11a-w4mbz" Oct 30 17:05:30.615797 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:30.615749 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b8r8\" (UniqueName: \"kubernetes.io/projected/664a4424-09d2-463a-a9ce-be604190999f-kube-api-access-6b8r8\") pod \"maas-api-revoke-sub-6301a11a-w4mbz\" (UID: \"664a4424-09d2-463a-a9ce-be604190999f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6301a11a-w4mbz" Oct 30 17:05:30.709004 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:30.708969 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6301a11a-w4mbz" Oct 30 17:05:31.037212 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:31.037187 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6301a11a-w4mbz"] Oct 30 17:05:31.038285 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:05:31.038259 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod664a4424_09d2_463a_a9ce_be604190999f.slice/crio-43b33a0f8e76c1768581b1d0a62d7d91a052bcbe9b3870163fb5ec2eac11cd15 WatchSource:0}: Error finding container 43b33a0f8e76c1768581b1d0a62d7d91a052bcbe9b3870163fb5ec2eac11cd15: Status 404 returned error can't find the container with id 43b33a0f8e76c1768581b1d0a62d7d91a052bcbe9b3870163fb5ec2eac11cd15 Oct 30 17:05:31.177081 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:31.177040 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6301a11a-w4mbz" event={"ID":"664a4424-09d2-463a-a9ce-be604190999f","Type":"ContainerStarted","Data":"43b33a0f8e76c1768581b1d0a62d7d91a052bcbe9b3870163fb5ec2eac11cd15"} Oct 30 17:05:32.185772 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:32.185736 4493 generic.go:356] "Generic (PLEG): container finished" podID="664a4424-09d2-463a-a9ce-be604190999f" containerID="b052fa23578e2658eb99f86c22ebe840fc105a3a96dbf6314d17ec09258f1cb9" exitCode=0 Oct 30 17:05:32.186284 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:32.185832 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6301a11a-w4mbz" event={"ID":"664a4424-09d2-463a-a9ce-be604190999f","Type":"ContainerDied","Data":"b052fa23578e2658eb99f86c22ebe840fc105a3a96dbf6314d17ec09258f1cb9"} Oct 30 17:05:33.375366 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:33.375337 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6301a11a-w4mbz" Oct 30 17:05:33.448562 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:33.448477 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/664a4424-09d2-463a-a9ce-be604190999f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/664a4424-09d2-463a-a9ce-be604190999f-maas-api-xks-ca\") pod \"664a4424-09d2-463a-a9ce-be604190999f\" (UID: \"664a4424-09d2-463a-a9ce-be604190999f\") " Oct 30 17:05:33.448562 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:33.448540 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/664a4424-09d2-463a-a9ce-be604190999f-kube-api-access-6b8r8\" (UniqueName: \"kubernetes.io/projected/664a4424-09d2-463a-a9ce-be604190999f-kube-api-access-6b8r8\") pod \"664a4424-09d2-463a-a9ce-be604190999f\" (UID: \"664a4424-09d2-463a-a9ce-be604190999f\") " Oct 30 17:05:33.448865 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:33.448682 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/664a4424-09d2-463a-a9ce-be604190999f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/664a4424-09d2-463a-a9ce-be604190999f-maas-api-openshift-ca\") pod \"664a4424-09d2-463a-a9ce-be604190999f\" (UID: \"664a4424-09d2-463a-a9ce-be604190999f\") " Oct 30 17:05:33.449263 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:33.449215 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/664a4424-09d2-463a-a9ce-be604190999f-maas-api-openshift-ca" pod "664a4424-09d2-463a-a9ce-be604190999f" (UID: "664a4424-09d2-463a-a9ce-be604190999f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:05:33.450429 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:33.450401 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/664a4424-09d2-463a-a9ce-be604190999f-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:05:33.450896 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:33.450873 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/664a4424-09d2-463a-a9ce-be604190999f-maas-api-xks-ca" pod "664a4424-09d2-463a-a9ce-be604190999f" (UID: "664a4424-09d2-463a-a9ce-be604190999f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:05:33.452496 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:33.452472 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/664a4424-09d2-463a-a9ce-be604190999f-kube-api-access-6b8r8" pod "664a4424-09d2-463a-a9ce-be604190999f" (UID: "664a4424-09d2-463a-a9ce-be604190999f"). InnerVolumeSpecName "kube-api-access-6b8r8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:05:33.552152 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:33.552110 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/664a4424-09d2-463a-a9ce-be604190999f-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:05:33.552152 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:33.552140 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6b8r8\" (UniqueName: \"kubernetes.io/projected/664a4424-09d2-463a-a9ce-be604190999f-kube-api-access-6b8r8\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:05:34.200649 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:34.200573 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6301a11a-w4mbz" Oct 30 17:05:34.200649 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:34.200610 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6301a11a-w4mbz" event={"ID":"664a4424-09d2-463a-a9ce-be604190999f","Type":"ContainerDied","Data":"43b33a0f8e76c1768581b1d0a62d7d91a052bcbe9b3870163fb5ec2eac11cd15"} Oct 30 17:05:34.200934 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:34.200654 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43b33a0f8e76c1768581b1d0a62d7d91a052bcbe9b3870163fb5ec2eac11cd15" Oct 30 17:05:36.034058 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:36.034027 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4674c49-tqlhw"] Oct 30 17:05:36.045877 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:36.045847 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4674c49-tqlhw"] Oct 30 17:05:36.934298 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:36.934269 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90406845-6ac6-46d0-ac71-99dfd34cc569" path="/var/lib/kubelet/pods/90406845-6ac6-46d0-ac71-99dfd34cc569/volumes" Oct 30 17:05:41.045037 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:41.045000 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99120b73-m4sg2"] Oct 30 17:05:41.045909 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:41.045881 4493 cpu_manager.go:414] "containerMap: removing container" podUID="664a4424-09d2-463a-a9ce-be604190999f" containerName="revoke-keys" Oct 30 17:05:41.045969 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:41.045912 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="664a4424-09d2-463a-a9ce-be604190999f" containerName="revoke-keys" Oct 30 17:05:41.046088 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:41.046074 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="664a4424-09d2-463a-a9ce-be604190999f" containerName="revoke-keys" Oct 30 17:05:41.050408 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:41.050391 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99120b73-m4sg2" Oct 30 17:05:41.070179 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:41.070158 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99120b73-m4sg2"] Oct 30 17:05:41.078617 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:41.078593 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:05:41.078751 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:41.078645 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 17:05:41.093448 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:41.093414 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:05:41.093580 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:41.093455 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:05:41.156399 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:41.156367 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac57c348-9196-42ad-9873-37c879019e21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99120b73-m4sg2\" (UID: \"ac57c348-9196-42ad-9873-37c879019e21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99120b73-m4sg2" Oct 30 17:05:41.156570 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:41.156492 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptf52\" (UniqueName: \"kubernetes.io/projected/ac57c348-9196-42ad-9873-37c879019e21-kube-api-access-ptf52\") pod \"maas-api-revoke-sub-99120b73-m4sg2\" (UID: \"ac57c348-9196-42ad-9873-37c879019e21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99120b73-m4sg2" Oct 30 17:05:41.156570 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:41.156531 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac57c348-9196-42ad-9873-37c879019e21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99120b73-m4sg2\" (UID: \"ac57c348-9196-42ad-9873-37c879019e21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99120b73-m4sg2" Oct 30 17:05:41.257658 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:41.257624 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac57c348-9196-42ad-9873-37c879019e21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99120b73-m4sg2\" (UID: \"ac57c348-9196-42ad-9873-37c879019e21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99120b73-m4sg2" Oct 30 17:05:41.257872 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:41.257702 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptf52\" (UniqueName: \"kubernetes.io/projected/ac57c348-9196-42ad-9873-37c879019e21-kube-api-access-ptf52\") pod \"maas-api-revoke-sub-99120b73-m4sg2\" (UID: \"ac57c348-9196-42ad-9873-37c879019e21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99120b73-m4sg2" Oct 30 17:05:41.257872 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:41.257738 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac57c348-9196-42ad-9873-37c879019e21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99120b73-m4sg2\" (UID: \"ac57c348-9196-42ad-9873-37c879019e21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99120b73-m4sg2" Oct 30 17:05:41.258448 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:41.258418 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac57c348-9196-42ad-9873-37c879019e21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99120b73-m4sg2\" (UID: \"ac57c348-9196-42ad-9873-37c879019e21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99120b73-m4sg2" Oct 30 17:05:41.260253 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:41.260230 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac57c348-9196-42ad-9873-37c879019e21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99120b73-m4sg2\" (UID: \"ac57c348-9196-42ad-9873-37c879019e21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99120b73-m4sg2" Oct 30 17:05:41.281143 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:41.281115 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptf52\" (UniqueName: \"kubernetes.io/projected/ac57c348-9196-42ad-9873-37c879019e21-kube-api-access-ptf52\") pod \"maas-api-revoke-sub-99120b73-m4sg2\" (UID: \"ac57c348-9196-42ad-9873-37c879019e21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99120b73-m4sg2" Oct 30 17:05:41.363538 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:41.363450 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99120b73-m4sg2" Oct 30 17:05:41.676449 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:41.676423 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99120b73-m4sg2"] Oct 30 17:05:41.677494 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:05:41.677471 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac57c348_9196_42ad_9873_37c879019e21.slice/crio-9bc296fe3da8f48cbe41d6a33f2873e91b775ad625b42db11d3c2811a97cc9d9 WatchSource:0}: Error finding container 9bc296fe3da8f48cbe41d6a33f2873e91b775ad625b42db11d3c2811a97cc9d9: Status 404 returned error can't find the container with id 9bc296fe3da8f48cbe41d6a33f2873e91b775ad625b42db11d3c2811a97cc9d9 Oct 30 17:05:42.283461 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:42.283366 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99120b73-m4sg2" event={"ID":"ac57c348-9196-42ad-9873-37c879019e21","Type":"ContainerStarted","Data":"78a9c0a768de39ccbfb96f9e83e15f4d7a8c516758cb8edbabb07d68e5e27439"} Oct 30 17:05:42.283849 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:42.283465 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99120b73-m4sg2" event={"ID":"ac57c348-9196-42ad-9873-37c879019e21","Type":"ContainerStarted","Data":"9bc296fe3da8f48cbe41d6a33f2873e91b775ad625b42db11d3c2811a97cc9d9"} Oct 30 17:05:42.323194 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:42.323136 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99120b73-m4sg2" podStartSLOduration=1.323120079 podStartE2EDuration="1.323120079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:05:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:05:42.32240721 +0000 UTC m=+2569.549582315" watchObservedRunningTime="2026-10-30 17:05:42.323120079 +0000 UTC m=+2569.550295180" Oct 30 17:05:43.293146 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:43.293113 4493 generic.go:356] "Generic (PLEG): container finished" podID="ac57c348-9196-42ad-9873-37c879019e21" containerID="78a9c0a768de39ccbfb96f9e83e15f4d7a8c516758cb8edbabb07d68e5e27439" exitCode=0 Oct 30 17:05:43.293626 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:43.293209 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99120b73-m4sg2" event={"ID":"ac57c348-9196-42ad-9873-37c879019e21","Type":"ContainerDied","Data":"78a9c0a768de39ccbfb96f9e83e15f4d7a8c516758cb8edbabb07d68e5e27439"} Oct 30 17:05:44.539742 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:44.539717 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99120b73-m4sg2" Oct 30 17:05:44.613864 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:44.611699 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac57c348-9196-42ad-9873-37c879019e21-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac57c348-9196-42ad-9873-37c879019e21-maas-api-xks-ca\") pod \"ac57c348-9196-42ad-9873-37c879019e21\" (UID: \"ac57c348-9196-42ad-9873-37c879019e21\") " Oct 30 17:05:44.613864 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:44.611835 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac57c348-9196-42ad-9873-37c879019e21-kube-api-access-ptf52\" (UniqueName: \"kubernetes.io/projected/ac57c348-9196-42ad-9873-37c879019e21-kube-api-access-ptf52\") pod \"ac57c348-9196-42ad-9873-37c879019e21\" (UID: \"ac57c348-9196-42ad-9873-37c879019e21\") " Oct 30 17:05:44.613864 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:44.611970 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac57c348-9196-42ad-9873-37c879019e21-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac57c348-9196-42ad-9873-37c879019e21-maas-api-openshift-ca\") pod \"ac57c348-9196-42ad-9873-37c879019e21\" (UID: \"ac57c348-9196-42ad-9873-37c879019e21\") " Oct 30 17:05:44.631042 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:44.622243 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac57c348-9196-42ad-9873-37c879019e21-maas-api-openshift-ca" pod "ac57c348-9196-42ad-9873-37c879019e21" (UID: "ac57c348-9196-42ad-9873-37c879019e21"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:05:44.706279 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:44.705953 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac57c348-9196-42ad-9873-37c879019e21-maas-api-xks-ca" pod "ac57c348-9196-42ad-9873-37c879019e21" (UID: "ac57c348-9196-42ad-9873-37c879019e21"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:05:44.720932 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:44.720896 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac57c348-9196-42ad-9873-37c879019e21-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:05:44.721156 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:44.721136 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac57c348-9196-42ad-9873-37c879019e21-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:05:44.721944 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:44.721911 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac57c348-9196-42ad-9873-37c879019e21-kube-api-access-ptf52" pod "ac57c348-9196-42ad-9873-37c879019e21" (UID: "ac57c348-9196-42ad-9873-37c879019e21"). InnerVolumeSpecName "kube-api-access-ptf52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:05:44.822612 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:44.822577 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptf52\" (UniqueName: \"kubernetes.io/projected/ac57c348-9196-42ad-9873-37c879019e21-kube-api-access-ptf52\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:05:45.311749 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:45.311672 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99120b73-m4sg2" Oct 30 17:05:45.311929 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:45.311749 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99120b73-m4sg2" event={"ID":"ac57c348-9196-42ad-9873-37c879019e21","Type":"ContainerDied","Data":"9bc296fe3da8f48cbe41d6a33f2873e91b775ad625b42db11d3c2811a97cc9d9"} Oct 30 17:05:45.311929 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:45.311821 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9bc296fe3da8f48cbe41d6a33f2873e91b775ad625b42db11d3c2811a97cc9d9" Oct 30 17:05:47.035839 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:47.035805 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4aeb7b96-6tfgm"] Oct 30 17:05:47.048215 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:47.048153 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4aeb7b96-6tfgm"] Oct 30 17:05:48.935079 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:48.935045 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de79a823-3142-4ce4-a646-09072a736b9f" path="/var/lib/kubelet/pods/de79a823-3142-4ce4-a646-09072a736b9f/volumes" Oct 30 17:05:49.448734 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:49.448689 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 17:05:55.454840 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:55.454804 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71b91dc5-48xgq"] Oct 30 17:05:55.455576 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:55.455561 4493 cpu_manager.go:414] "containerMap: removing container" podUID="ac57c348-9196-42ad-9873-37c879019e21" containerName="revoke-keys" Oct 30 17:05:55.455641 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:55.455579 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac57c348-9196-42ad-9873-37c879019e21" containerName="revoke-keys" Oct 30 17:05:55.455798 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:55.455772 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac57c348-9196-42ad-9873-37c879019e21" containerName="revoke-keys" Oct 30 17:05:55.459697 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:55.459677 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71b91dc5-48xgq" Oct 30 17:05:55.476093 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:55.476064 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:05:55.487908 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:55.487882 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:05:55.488742 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:55.488718 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 17:05:55.488950 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:55.488821 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:05:55.491777 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:55.491756 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71b91dc5-48xgq"] Oct 30 17:05:55.500802 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:55.500764 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32ae81e0-de61-47ec-809c-a1ec55b181a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71b91dc5-48xgq\" (UID: \"32ae81e0-de61-47ec-809c-a1ec55b181a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71b91dc5-48xgq" Oct 30 17:05:55.500914 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:55.500888 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32ae81e0-de61-47ec-809c-a1ec55b181a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71b91dc5-48xgq\" (UID: \"32ae81e0-de61-47ec-809c-a1ec55b181a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71b91dc5-48xgq" Oct 30 17:05:55.501045 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:55.501031 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk886\" (UniqueName: \"kubernetes.io/projected/32ae81e0-de61-47ec-809c-a1ec55b181a6-kube-api-access-fk886\") pod \"maas-api-revoke-sub-71b91dc5-48xgq\" (UID: \"32ae81e0-de61-47ec-809c-a1ec55b181a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71b91dc5-48xgq" Oct 30 17:05:55.602176 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:55.602144 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32ae81e0-de61-47ec-809c-a1ec55b181a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71b91dc5-48xgq\" (UID: \"32ae81e0-de61-47ec-809c-a1ec55b181a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71b91dc5-48xgq" Oct 30 17:05:55.602362 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:55.602285 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk886\" (UniqueName: \"kubernetes.io/projected/32ae81e0-de61-47ec-809c-a1ec55b181a6-kube-api-access-fk886\") pod \"maas-api-revoke-sub-71b91dc5-48xgq\" (UID: \"32ae81e0-de61-47ec-809c-a1ec55b181a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71b91dc5-48xgq" Oct 30 17:05:55.602428 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:55.602381 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32ae81e0-de61-47ec-809c-a1ec55b181a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71b91dc5-48xgq\" (UID: \"32ae81e0-de61-47ec-809c-a1ec55b181a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71b91dc5-48xgq" Oct 30 17:05:55.602874 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:55.602853 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32ae81e0-de61-47ec-809c-a1ec55b181a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71b91dc5-48xgq\" (UID: \"32ae81e0-de61-47ec-809c-a1ec55b181a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71b91dc5-48xgq" Oct 30 17:05:55.604988 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:55.604966 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32ae81e0-de61-47ec-809c-a1ec55b181a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71b91dc5-48xgq\" (UID: \"32ae81e0-de61-47ec-809c-a1ec55b181a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71b91dc5-48xgq" Oct 30 17:05:55.625727 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:55.625701 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk886\" (UniqueName: \"kubernetes.io/projected/32ae81e0-de61-47ec-809c-a1ec55b181a6-kube-api-access-fk886\") pod \"maas-api-revoke-sub-71b91dc5-48xgq\" (UID: \"32ae81e0-de61-47ec-809c-a1ec55b181a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71b91dc5-48xgq" Oct 30 17:05:55.773193 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:55.773112 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71b91dc5-48xgq" Oct 30 17:05:56.087923 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:56.087885 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71b91dc5-48xgq"] Oct 30 17:05:56.089286 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:05:56.089242 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32ae81e0_de61_47ec_809c_a1ec55b181a6.slice/crio-636a4c19a228c1dd0eeff04adf496685a7a8fd1eabe466fa2c90ddc7ae0b2427 WatchSource:0}: Error finding container 636a4c19a228c1dd0eeff04adf496685a7a8fd1eabe466fa2c90ddc7ae0b2427: Status 404 returned error can't find the container with id 636a4c19a228c1dd0eeff04adf496685a7a8fd1eabe466fa2c90ddc7ae0b2427 Oct 30 17:05:56.395322 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:56.395290 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71b91dc5-48xgq" event={"ID":"32ae81e0-de61-47ec-809c-a1ec55b181a6","Type":"ContainerStarted","Data":"636a4c19a228c1dd0eeff04adf496685a7a8fd1eabe466fa2c90ddc7ae0b2427"} Oct 30 17:05:56.455944 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:56.455914 4493 scope.go:122] "RemoveContainer" containerID="d41ea7dcfc2e4df9b104a2de4077ce6800523b4a23fd2430458a813de2b1137a" Oct 30 17:05:56.467595 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:56.467570 4493 scope.go:122] "RemoveContainer" containerID="94d41bea71d99e1e7ab57cd727a4be213eb4d72e6bb769aee6750a5c21837dbe" Oct 30 17:05:57.406824 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:57.406773 4493 generic.go:356] "Generic (PLEG): container finished" podID="32ae81e0-de61-47ec-809c-a1ec55b181a6" containerID="56354649c121f92c56a239868751842cc87dd67ac06435c92869fd0613f8a1e8" exitCode=0 Oct 30 17:05:57.406985 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:57.406893 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71b91dc5-48xgq" event={"ID":"32ae81e0-de61-47ec-809c-a1ec55b181a6","Type":"ContainerDied","Data":"56354649c121f92c56a239868751842cc87dd67ac06435c92869fd0613f8a1e8"} Oct 30 17:05:58.566468 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:58.566447 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71b91dc5-48xgq" Oct 30 17:05:58.641913 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:58.641874 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32ae81e0-de61-47ec-809c-a1ec55b181a6-kube-api-access-fk886\" (UniqueName: \"kubernetes.io/projected/32ae81e0-de61-47ec-809c-a1ec55b181a6-kube-api-access-fk886\") pod \"32ae81e0-de61-47ec-809c-a1ec55b181a6\" (UID: \"32ae81e0-de61-47ec-809c-a1ec55b181a6\") " Oct 30 17:05:58.642105 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:58.641931 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32ae81e0-de61-47ec-809c-a1ec55b181a6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32ae81e0-de61-47ec-809c-a1ec55b181a6-maas-api-xks-ca\") pod \"32ae81e0-de61-47ec-809c-a1ec55b181a6\" (UID: \"32ae81e0-de61-47ec-809c-a1ec55b181a6\") " Oct 30 17:05:58.642105 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:58.642033 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32ae81e0-de61-47ec-809c-a1ec55b181a6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32ae81e0-de61-47ec-809c-a1ec55b181a6-maas-api-openshift-ca\") pod \"32ae81e0-de61-47ec-809c-a1ec55b181a6\" (UID: \"32ae81e0-de61-47ec-809c-a1ec55b181a6\") " Oct 30 17:05:58.642673 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:58.642645 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32ae81e0-de61-47ec-809c-a1ec55b181a6-maas-api-openshift-ca" pod "32ae81e0-de61-47ec-809c-a1ec55b181a6" (UID: "32ae81e0-de61-47ec-809c-a1ec55b181a6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:05:58.644967 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:58.644936 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32ae81e0-de61-47ec-809c-a1ec55b181a6-maas-api-xks-ca" pod "32ae81e0-de61-47ec-809c-a1ec55b181a6" (UID: "32ae81e0-de61-47ec-809c-a1ec55b181a6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:05:58.645089 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:58.645018 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32ae81e0-de61-47ec-809c-a1ec55b181a6-kube-api-access-fk886" pod "32ae81e0-de61-47ec-809c-a1ec55b181a6" (UID: "32ae81e0-de61-47ec-809c-a1ec55b181a6"). InnerVolumeSpecName "kube-api-access-fk886". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:05:58.743713 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:58.743653 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fk886\" (UniqueName: \"kubernetes.io/projected/32ae81e0-de61-47ec-809c-a1ec55b181a6-kube-api-access-fk886\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:05:58.743713 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:58.743679 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32ae81e0-de61-47ec-809c-a1ec55b181a6-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:05:58.743713 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:58.743691 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32ae81e0-de61-47ec-809c-a1ec55b181a6-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:05:59.424054 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:59.424026 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71b91dc5-48xgq" Oct 30 17:05:59.424346 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:59.424052 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71b91dc5-48xgq" event={"ID":"32ae81e0-de61-47ec-809c-a1ec55b181a6","Type":"ContainerDied","Data":"636a4c19a228c1dd0eeff04adf496685a7a8fd1eabe466fa2c90ddc7ae0b2427"} Oct 30 17:05:59.424494 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:59.424376 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="636a4c19a228c1dd0eeff04adf496685a7a8fd1eabe466fa2c90ddc7ae0b2427" Oct 30 17:05:59.652928 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:59.652898 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b87b52b0-kqjxx"] Oct 30 17:05:59.653459 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:59.653444 4493 cpu_manager.go:414] "containerMap: removing container" podUID="32ae81e0-de61-47ec-809c-a1ec55b181a6" containerName="revoke-keys" Oct 30 17:05:59.653508 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:59.653461 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32ae81e0-de61-47ec-809c-a1ec55b181a6" containerName="revoke-keys" Oct 30 17:05:59.653578 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:59.653569 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="32ae81e0-de61-47ec-809c-a1ec55b181a6" containerName="revoke-keys" Oct 30 17:05:59.657171 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:59.657150 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b87b52b0-kqjxx" Oct 30 17:05:59.672516 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:59.672495 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:05:59.677245 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:59.677167 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:05:59.682843 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:59.682750 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b87b52b0-kqjxx"] Oct 30 17:05:59.684279 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:59.684259 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 17:05:59.757083 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:59.757046 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8d4ac15-01f8-42c9-83aa-dbe69955b0d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b87b52b0-kqjxx\" (UID: \"b8d4ac15-01f8-42c9-83aa-dbe69955b0d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b87b52b0-kqjxx" Oct 30 17:05:59.757245 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:59.757104 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g2q4\" (UniqueName: \"kubernetes.io/projected/b8d4ac15-01f8-42c9-83aa-dbe69955b0d8-kube-api-access-6g2q4\") pod \"maas-api-revoke-sub-b87b52b0-kqjxx\" (UID: \"b8d4ac15-01f8-42c9-83aa-dbe69955b0d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b87b52b0-kqjxx" Oct 30 17:05:59.757245 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:59.757222 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8d4ac15-01f8-42c9-83aa-dbe69955b0d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b87b52b0-kqjxx\" (UID: \"b8d4ac15-01f8-42c9-83aa-dbe69955b0d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b87b52b0-kqjxx" Oct 30 17:05:59.773424 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:59.773393 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:05:59.858296 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:59.858261 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8d4ac15-01f8-42c9-83aa-dbe69955b0d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b87b52b0-kqjxx\" (UID: \"b8d4ac15-01f8-42c9-83aa-dbe69955b0d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b87b52b0-kqjxx" Oct 30 17:05:59.858468 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:59.858305 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g2q4\" (UniqueName: \"kubernetes.io/projected/b8d4ac15-01f8-42c9-83aa-dbe69955b0d8-kube-api-access-6g2q4\") pod \"maas-api-revoke-sub-b87b52b0-kqjxx\" (UID: \"b8d4ac15-01f8-42c9-83aa-dbe69955b0d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b87b52b0-kqjxx" Oct 30 17:05:59.858468 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:59.858353 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8d4ac15-01f8-42c9-83aa-dbe69955b0d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b87b52b0-kqjxx\" (UID: \"b8d4ac15-01f8-42c9-83aa-dbe69955b0d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b87b52b0-kqjxx" Oct 30 17:05:59.859158 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:59.859073 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8d4ac15-01f8-42c9-83aa-dbe69955b0d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b87b52b0-kqjxx\" (UID: \"b8d4ac15-01f8-42c9-83aa-dbe69955b0d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b87b52b0-kqjxx" Oct 30 17:05:59.860949 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:59.860924 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8d4ac15-01f8-42c9-83aa-dbe69955b0d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b87b52b0-kqjxx\" (UID: \"b8d4ac15-01f8-42c9-83aa-dbe69955b0d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b87b52b0-kqjxx" Oct 30 17:05:59.881251 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:59.881224 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g2q4\" (UniqueName: \"kubernetes.io/projected/b8d4ac15-01f8-42c9-83aa-dbe69955b0d8-kube-api-access-6g2q4\") pod \"maas-api-revoke-sub-b87b52b0-kqjxx\" (UID: \"b8d4ac15-01f8-42c9-83aa-dbe69955b0d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b87b52b0-kqjxx" Oct 30 17:05:59.977623 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:05:59.977543 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b87b52b0-kqjxx" Oct 30 17:06:00.287201 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:00.287169 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b87b52b0-kqjxx"] Oct 30 17:06:00.288087 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:06:00.288062 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8d4ac15_01f8_42c9_83aa_dbe69955b0d8.slice/crio-5650c6fcd3b0feeb79f608f6e3a66713810211026d71e8afbcc178664caaf754 WatchSource:0}: Error finding container 5650c6fcd3b0feeb79f608f6e3a66713810211026d71e8afbcc178664caaf754: Status 404 returned error can't find the container with id 5650c6fcd3b0feeb79f608f6e3a66713810211026d71e8afbcc178664caaf754 Oct 30 17:06:00.343421 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:06:00.343387 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 17:06:00.432516 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:00.432480 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b87b52b0-kqjxx" event={"ID":"b8d4ac15-01f8-42c9-83aa-dbe69955b0d8","Type":"ContainerStarted","Data":"5650c6fcd3b0feeb79f608f6e3a66713810211026d71e8afbcc178664caaf754"} Oct 30 17:06:01.441817 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:01.441751 4493 generic.go:356] "Generic (PLEG): container finished" podID="b8d4ac15-01f8-42c9-83aa-dbe69955b0d8" containerID="800b402f1f1906601d64101df2d5630e62bfde1851485129f70f18e5bb9520dd" exitCode=0 Oct 30 17:06:01.442338 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:01.441842 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b87b52b0-kqjxx" event={"ID":"b8d4ac15-01f8-42c9-83aa-dbe69955b0d8","Type":"ContainerDied","Data":"800b402f1f1906601d64101df2d5630e62bfde1851485129f70f18e5bb9520dd"} Oct 30 17:06:02.606462 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:02.606434 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b87b52b0-kqjxx" Oct 30 17:06:02.693411 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:02.693372 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8d4ac15-01f8-42c9-83aa-dbe69955b0d8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8d4ac15-01f8-42c9-83aa-dbe69955b0d8-maas-api-xks-ca\") pod \"b8d4ac15-01f8-42c9-83aa-dbe69955b0d8\" (UID: \"b8d4ac15-01f8-42c9-83aa-dbe69955b0d8\") " Oct 30 17:06:02.693618 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:02.693449 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8d4ac15-01f8-42c9-83aa-dbe69955b0d8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8d4ac15-01f8-42c9-83aa-dbe69955b0d8-maas-api-openshift-ca\") pod \"b8d4ac15-01f8-42c9-83aa-dbe69955b0d8\" (UID: \"b8d4ac15-01f8-42c9-83aa-dbe69955b0d8\") " Oct 30 17:06:02.693618 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:02.693474 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8d4ac15-01f8-42c9-83aa-dbe69955b0d8-kube-api-access-6g2q4\" (UniqueName: \"kubernetes.io/projected/b8d4ac15-01f8-42c9-83aa-dbe69955b0d8-kube-api-access-6g2q4\") pod \"b8d4ac15-01f8-42c9-83aa-dbe69955b0d8\" (UID: \"b8d4ac15-01f8-42c9-83aa-dbe69955b0d8\") " Oct 30 17:06:02.693909 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:02.693874 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8d4ac15-01f8-42c9-83aa-dbe69955b0d8-maas-api-openshift-ca" pod "b8d4ac15-01f8-42c9-83aa-dbe69955b0d8" (UID: "b8d4ac15-01f8-42c9-83aa-dbe69955b0d8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:06:02.695876 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:02.695851 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8d4ac15-01f8-42c9-83aa-dbe69955b0d8-maas-api-xks-ca" pod "b8d4ac15-01f8-42c9-83aa-dbe69955b0d8" (UID: "b8d4ac15-01f8-42c9-83aa-dbe69955b0d8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:06:02.696014 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:02.695849 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8d4ac15-01f8-42c9-83aa-dbe69955b0d8-kube-api-access-6g2q4" pod "b8d4ac15-01f8-42c9-83aa-dbe69955b0d8" (UID: "b8d4ac15-01f8-42c9-83aa-dbe69955b0d8"). InnerVolumeSpecName "kube-api-access-6g2q4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:06:02.795114 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:02.795083 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8d4ac15-01f8-42c9-83aa-dbe69955b0d8-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:06:02.795114 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:02.795110 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8d4ac15-01f8-42c9-83aa-dbe69955b0d8-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:06:02.795114 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:02.795120 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6g2q4\" (UniqueName: \"kubernetes.io/projected/b8d4ac15-01f8-42c9-83aa-dbe69955b0d8-kube-api-access-6g2q4\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:06:03.460035 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:03.460004 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b87b52b0-kqjxx" Oct 30 17:06:03.460238 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:03.460042 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b87b52b0-kqjxx" event={"ID":"b8d4ac15-01f8-42c9-83aa-dbe69955b0d8","Type":"ContainerDied","Data":"5650c6fcd3b0feeb79f608f6e3a66713810211026d71e8afbcc178664caaf754"} Oct 30 17:06:03.460238 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:03.460087 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5650c6fcd3b0feeb79f608f6e3a66713810211026d71e8afbcc178664caaf754" Oct 30 17:06:04.774398 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:04.774356 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bc58j42p"] Oct 30 17:06:09.775923 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:09.775861 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bc58j42p" podUID="82f0f9de-fb45-48b1-9a13-9d9149c893a3" containerName="payload-processing" containerID="cri-o://39cb612607a51558dc58245e07720e8f26880794068e32614311d68b8342859b" gracePeriod=25 Oct 30 17:06:10.065611 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:10.065586 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bc58j42p" Oct 30 17:06:10.221920 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:10.221854 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82f0f9de-fb45-48b1-9a13-9d9149c893a3-provider-credentials\" (UniqueName: \"kubernetes.io/projected/82f0f9de-fb45-48b1-9a13-9d9149c893a3-provider-credentials\") pod \"82f0f9de-fb45-48b1-9a13-9d9149c893a3\" (UID: \"82f0f9de-fb45-48b1-9a13-9d9149c893a3\") " Oct 30 17:06:10.222132 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:10.222079 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82f0f9de-fb45-48b1-9a13-9d9149c893a3-kube-api-access-ljllb\" (UniqueName: \"kubernetes.io/projected/82f0f9de-fb45-48b1-9a13-9d9149c893a3-kube-api-access-ljllb\") pod \"82f0f9de-fb45-48b1-9a13-9d9149c893a3\" (UID: \"82f0f9de-fb45-48b1-9a13-9d9149c893a3\") " Oct 30 17:06:10.222203 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:10.222160 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82f0f9de-fb45-48b1-9a13-9d9149c893a3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82f0f9de-fb45-48b1-9a13-9d9149c893a3-plugins-config-volume\") pod \"82f0f9de-fb45-48b1-9a13-9d9149c893a3\" (UID: \"82f0f9de-fb45-48b1-9a13-9d9149c893a3\") " Oct 30 17:06:10.223031 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:10.222267 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82f0f9de-fb45-48b1-9a13-9d9149c893a3-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/82f0f9de-fb45-48b1-9a13-9d9149c893a3-routing-overlay-volume\") pod \"82f0f9de-fb45-48b1-9a13-9d9149c893a3\" (UID: \"82f0f9de-fb45-48b1-9a13-9d9149c893a3\") " Oct 30 17:06:10.225677 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:10.225648 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82f0f9de-fb45-48b1-9a13-9d9149c893a3-routing-overlay-volume" pod "82f0f9de-fb45-48b1-9a13-9d9149c893a3" (UID: "82f0f9de-fb45-48b1-9a13-9d9149c893a3"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:06:10.226316 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:10.226292 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82f0f9de-fb45-48b1-9a13-9d9149c893a3-plugins-config-volume" pod "82f0f9de-fb45-48b1-9a13-9d9149c893a3" (UID: "82f0f9de-fb45-48b1-9a13-9d9149c893a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:06:10.228407 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:10.228378 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82f0f9de-fb45-48b1-9a13-9d9149c893a3-kube-api-access-ljllb" pod "82f0f9de-fb45-48b1-9a13-9d9149c893a3" (UID: "82f0f9de-fb45-48b1-9a13-9d9149c893a3"). InnerVolumeSpecName "kube-api-access-ljllb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:06:10.228591 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:10.228571 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82f0f9de-fb45-48b1-9a13-9d9149c893a3-provider-credentials" pod "82f0f9de-fb45-48b1-9a13-9d9149c893a3" (UID: "82f0f9de-fb45-48b1-9a13-9d9149c893a3"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:06:10.324562 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:10.324472 4493 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/82f0f9de-fb45-48b1-9a13-9d9149c893a3-routing-overlay-volume\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:06:10.324562 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:10.324502 4493 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/82f0f9de-fb45-48b1-9a13-9d9149c893a3-provider-credentials\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:06:10.324562 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:10.324512 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljllb\" (UniqueName: \"kubernetes.io/projected/82f0f9de-fb45-48b1-9a13-9d9149c893a3-kube-api-access-ljllb\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:06:10.324562 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:10.324520 4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82f0f9de-fb45-48b1-9a13-9d9149c893a3-plugins-config-volume\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:06:10.513022 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:10.512986 4493 generic.go:356] "Generic (PLEG): container finished" podID="82f0f9de-fb45-48b1-9a13-9d9149c893a3" containerID="39cb612607a51558dc58245e07720e8f26880794068e32614311d68b8342859b" exitCode=0 Oct 30 17:06:10.513217 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:10.513061 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bc58j42p" Oct 30 17:06:10.513217 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:10.513084 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bc58j42p" event={"ID":"82f0f9de-fb45-48b1-9a13-9d9149c893a3","Type":"ContainerDied","Data":"39cb612607a51558dc58245e07720e8f26880794068e32614311d68b8342859b"} Oct 30 17:06:10.513217 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:10.513190 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bc58j42p" event={"ID":"82f0f9de-fb45-48b1-9a13-9d9149c893a3","Type":"ContainerDied","Data":"0726f68122bddca26e20da2a9c436de89d930ccef22cbf04fa4dbe31d2261e2d"} Oct 30 17:06:10.513217 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:10.513208 4493 scope.go:122] "RemoveContainer" containerID="39cb612607a51558dc58245e07720e8f26880794068e32614311d68b8342859b" Oct 30 17:06:10.524635 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:10.524615 4493 scope.go:122] "RemoveContainer" containerID="39cb612607a51558dc58245e07720e8f26880794068e32614311d68b8342859b" Oct 30 17:06:10.526259 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:06:10.526211 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39cb612607a51558dc58245e07720e8f26880794068e32614311d68b8342859b\": container with ID starting with 39cb612607a51558dc58245e07720e8f26880794068e32614311d68b8342859b not found: ID does not exist" containerID="39cb612607a51558dc58245e07720e8f26880794068e32614311d68b8342859b" Oct 30 17:06:10.526408 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:10.526255 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39cb612607a51558dc58245e07720e8f26880794068e32614311d68b8342859b"} err="failed to get container status \"39cb612607a51558dc58245e07720e8f26880794068e32614311d68b8342859b\": rpc error: code = NotFound desc = could not find container \"39cb612607a51558dc58245e07720e8f26880794068e32614311d68b8342859b\": container with ID starting with 39cb612607a51558dc58245e07720e8f26880794068e32614311d68b8342859b not found: ID does not exist" Oct 30 17:06:10.565423 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:10.565390 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bc58j42p"] Oct 30 17:06:10.577041 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:10.576972 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bc58j42p"] Oct 30 17:06:10.952855 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:10.952543 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82f0f9de-fb45-48b1-9a13-9d9149c893a3" path="/var/lib/kubelet/pods/82f0f9de-fb45-48b1-9a13-9d9149c893a3/volumes" Oct 30 17:06:11.228675 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:11.228576 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5dbf38c-4wrxh"] Oct 30 17:06:11.229163 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:11.229144 4493 cpu_manager.go:414] "containerMap: removing container" podUID="82f0f9de-fb45-48b1-9a13-9d9149c893a3" containerName="payload-processing" Oct 30 17:06:11.229163 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:11.229162 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82f0f9de-fb45-48b1-9a13-9d9149c893a3" containerName="payload-processing" Oct 30 17:06:11.229335 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:11.229196 4493 cpu_manager.go:414] "containerMap: removing container" podUID="b8d4ac15-01f8-42c9-83aa-dbe69955b0d8" containerName="revoke-keys" Oct 30 17:06:11.229335 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:11.229203 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8d4ac15-01f8-42c9-83aa-dbe69955b0d8" containerName="revoke-keys" Oct 30 17:06:11.229335 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:11.229287 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="82f0f9de-fb45-48b1-9a13-9d9149c893a3" containerName="payload-processing" Oct 30 17:06:11.229335 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:11.229302 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8d4ac15-01f8-42c9-83aa-dbe69955b0d8" containerName="revoke-keys" Oct 30 17:06:11.234266 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:11.234238 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5dbf38c-4wrxh" Oct 30 17:06:11.261603 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:11.261563 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5dbf38c-4wrxh"] Oct 30 17:06:11.262829 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:11.262806 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:06:11.263003 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:11.262849 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 17:06:11.263084 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:11.262860 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:06:11.263084 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:11.262887 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:06:11.352992 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:11.352959 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f844002e-4790-452b-88bf-7e33af30001e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5dbf38c-4wrxh\" (UID: \"f844002e-4790-452b-88bf-7e33af30001e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5dbf38c-4wrxh" Oct 30 17:06:11.353147 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:11.353104 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbhkl\" (UniqueName: \"kubernetes.io/projected/f844002e-4790-452b-88bf-7e33af30001e-kube-api-access-dbhkl\") pod \"maas-api-revoke-sub-b5dbf38c-4wrxh\" (UID: \"f844002e-4790-452b-88bf-7e33af30001e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5dbf38c-4wrxh" Oct 30 17:06:11.353147 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:11.353136 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f844002e-4790-452b-88bf-7e33af30001e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5dbf38c-4wrxh\" (UID: \"f844002e-4790-452b-88bf-7e33af30001e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5dbf38c-4wrxh" Oct 30 17:06:11.454094 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:11.454063 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbhkl\" (UniqueName: \"kubernetes.io/projected/f844002e-4790-452b-88bf-7e33af30001e-kube-api-access-dbhkl\") pod \"maas-api-revoke-sub-b5dbf38c-4wrxh\" (UID: \"f844002e-4790-452b-88bf-7e33af30001e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5dbf38c-4wrxh" Oct 30 17:06:11.454289 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:11.454101 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f844002e-4790-452b-88bf-7e33af30001e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5dbf38c-4wrxh\" (UID: \"f844002e-4790-452b-88bf-7e33af30001e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5dbf38c-4wrxh" Oct 30 17:06:11.454289 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:11.454250 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f844002e-4790-452b-88bf-7e33af30001e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5dbf38c-4wrxh\" (UID: \"f844002e-4790-452b-88bf-7e33af30001e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5dbf38c-4wrxh" Oct 30 17:06:11.454798 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:11.454763 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f844002e-4790-452b-88bf-7e33af30001e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5dbf38c-4wrxh\" (UID: \"f844002e-4790-452b-88bf-7e33af30001e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5dbf38c-4wrxh" Oct 30 17:06:11.456980 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:11.456955 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f844002e-4790-452b-88bf-7e33af30001e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5dbf38c-4wrxh\" (UID: \"f844002e-4790-452b-88bf-7e33af30001e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5dbf38c-4wrxh" Oct 30 17:06:11.484881 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:11.483928 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbhkl\" (UniqueName: \"kubernetes.io/projected/f844002e-4790-452b-88bf-7e33af30001e-kube-api-access-dbhkl\") pod \"maas-api-revoke-sub-b5dbf38c-4wrxh\" (UID: \"f844002e-4790-452b-88bf-7e33af30001e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5dbf38c-4wrxh" Oct 30 17:06:11.551839 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:11.551455 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5dbf38c-4wrxh" Oct 30 17:06:11.923413 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:11.923385 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5dbf38c-4wrxh"] Oct 30 17:06:11.925015 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:06:11.924983 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf844002e_4790_452b_88bf_7e33af30001e.slice/crio-2d2539cd8974d33fe8acdd891497915682b59da235113ec9e144f335dd4604f9 WatchSource:0}: Error finding container 2d2539cd8974d33fe8acdd891497915682b59da235113ec9e144f335dd4604f9: Status 404 returned error can't find the container with id 2d2539cd8974d33fe8acdd891497915682b59da235113ec9e144f335dd4604f9 Oct 30 17:06:12.580895 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:12.580813 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5dbf38c-4wrxh" event={"ID":"f844002e-4790-452b-88bf-7e33af30001e","Type":"ContainerStarted","Data":"6c0ab7b5540f82977a83506f613bb1b0f1ab4355c61c78c09a10d8f64ff4aa56"} Oct 30 17:06:12.580895 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:12.580871 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5dbf38c-4wrxh" event={"ID":"f844002e-4790-452b-88bf-7e33af30001e","Type":"ContainerStarted","Data":"2d2539cd8974d33fe8acdd891497915682b59da235113ec9e144f335dd4604f9"} Oct 30 17:06:12.621370 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:12.621322 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5dbf38c-4wrxh" podStartSLOduration=1.621308998 podStartE2EDuration="1.621308998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:06:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:06:12.619950924 +0000 UTC m=+2599.847126024" watchObservedRunningTime="2026-10-30 17:06:12.621308998 +0000 UTC m=+2599.848484098" Oct 30 17:06:13.589648 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:13.589605 4493 generic.go:356] "Generic (PLEG): container finished" podID="f844002e-4790-452b-88bf-7e33af30001e" containerID="6c0ab7b5540f82977a83506f613bb1b0f1ab4355c61c78c09a10d8f64ff4aa56" exitCode=0 Oct 30 17:06:13.590074 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:13.589654 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5dbf38c-4wrxh" event={"ID":"f844002e-4790-452b-88bf-7e33af30001e","Type":"ContainerDied","Data":"6c0ab7b5540f82977a83506f613bb1b0f1ab4355c61c78c09a10d8f64ff4aa56"} Oct 30 17:06:14.323065 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:14.323033 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5a6e039-m5zsq"] Oct 30 17:06:14.327533 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:14.327511 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a6e039-m5zsq" Oct 30 17:06:14.353000 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:14.352940 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5a6e039-m5zsq"] Oct 30 17:06:14.426614 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:14.426589 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd2d5a07-ba42-482e-8a22-d5dec65ddeef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5a6e039-m5zsq\" (UID: \"fd2d5a07-ba42-482e-8a22-d5dec65ddeef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a6e039-m5zsq" Oct 30 17:06:14.426820 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:14.426637 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd2d5a07-ba42-482e-8a22-d5dec65ddeef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5a6e039-m5zsq\" (UID: \"fd2d5a07-ba42-482e-8a22-d5dec65ddeef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a6e039-m5zsq" Oct 30 17:06:14.426913 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:14.426889 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg5kp\" (UniqueName: \"kubernetes.io/projected/fd2d5a07-ba42-482e-8a22-d5dec65ddeef-kube-api-access-vg5kp\") pod \"maas-api-revoke-sub-a5a6e039-m5zsq\" (UID: \"fd2d5a07-ba42-482e-8a22-d5dec65ddeef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a6e039-m5zsq" Oct 30 17:06:14.528232 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:14.528194 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg5kp\" (UniqueName: \"kubernetes.io/projected/fd2d5a07-ba42-482e-8a22-d5dec65ddeef-kube-api-access-vg5kp\") pod \"maas-api-revoke-sub-a5a6e039-m5zsq\" (UID: \"fd2d5a07-ba42-482e-8a22-d5dec65ddeef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a6e039-m5zsq" Oct 30 17:06:14.528446 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:14.528282 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd2d5a07-ba42-482e-8a22-d5dec65ddeef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5a6e039-m5zsq\" (UID: \"fd2d5a07-ba42-482e-8a22-d5dec65ddeef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a6e039-m5zsq" Oct 30 17:06:14.528446 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:14.528353 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd2d5a07-ba42-482e-8a22-d5dec65ddeef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5a6e039-m5zsq\" (UID: \"fd2d5a07-ba42-482e-8a22-d5dec65ddeef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a6e039-m5zsq" Oct 30 17:06:14.529199 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:14.529174 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd2d5a07-ba42-482e-8a22-d5dec65ddeef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5a6e039-m5zsq\" (UID: \"fd2d5a07-ba42-482e-8a22-d5dec65ddeef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a6e039-m5zsq" Oct 30 17:06:14.531112 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:14.531080 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd2d5a07-ba42-482e-8a22-d5dec65ddeef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5a6e039-m5zsq\" (UID: \"fd2d5a07-ba42-482e-8a22-d5dec65ddeef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a6e039-m5zsq" Oct 30 17:06:14.552704 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:14.552677 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg5kp\" (UniqueName: \"kubernetes.io/projected/fd2d5a07-ba42-482e-8a22-d5dec65ddeef-kube-api-access-vg5kp\") pod \"maas-api-revoke-sub-a5a6e039-m5zsq\" (UID: \"fd2d5a07-ba42-482e-8a22-d5dec65ddeef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a6e039-m5zsq" Oct 30 17:06:14.641912 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:14.641867 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a6e039-m5zsq" Oct 30 17:06:14.780231 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:14.779898 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5dbf38c-4wrxh" Oct 30 17:06:14.834990 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:14.834961 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f844002e-4790-452b-88bf-7e33af30001e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f844002e-4790-452b-88bf-7e33af30001e-maas-api-openshift-ca\") pod \"f844002e-4790-452b-88bf-7e33af30001e\" (UID: \"f844002e-4790-452b-88bf-7e33af30001e\") " Oct 30 17:06:14.835192 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:14.835049 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f844002e-4790-452b-88bf-7e33af30001e-kube-api-access-dbhkl\" (UniqueName: \"kubernetes.io/projected/f844002e-4790-452b-88bf-7e33af30001e-kube-api-access-dbhkl\") pod \"f844002e-4790-452b-88bf-7e33af30001e\" (UID: \"f844002e-4790-452b-88bf-7e33af30001e\") " Oct 30 17:06:14.835192 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:14.835174 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f844002e-4790-452b-88bf-7e33af30001e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f844002e-4790-452b-88bf-7e33af30001e-maas-api-xks-ca\") pod \"f844002e-4790-452b-88bf-7e33af30001e\" (UID: \"f844002e-4790-452b-88bf-7e33af30001e\") " Oct 30 17:06:14.835365 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:14.835332 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f844002e-4790-452b-88bf-7e33af30001e-maas-api-openshift-ca" pod "f844002e-4790-452b-88bf-7e33af30001e" (UID: "f844002e-4790-452b-88bf-7e33af30001e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:06:14.835718 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:14.835697 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f844002e-4790-452b-88bf-7e33af30001e-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:06:14.837654 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:14.837628 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f844002e-4790-452b-88bf-7e33af30001e-kube-api-access-dbhkl" pod "f844002e-4790-452b-88bf-7e33af30001e" (UID: "f844002e-4790-452b-88bf-7e33af30001e"). InnerVolumeSpecName "kube-api-access-dbhkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:06:14.838219 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:14.838198 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f844002e-4790-452b-88bf-7e33af30001e-maas-api-xks-ca" pod "f844002e-4790-452b-88bf-7e33af30001e" (UID: "f844002e-4790-452b-88bf-7e33af30001e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:06:14.937246 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:14.937212 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbhkl\" (UniqueName: \"kubernetes.io/projected/f844002e-4790-452b-88bf-7e33af30001e-kube-api-access-dbhkl\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:06:14.937246 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:14.937242 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f844002e-4790-452b-88bf-7e33af30001e-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:06:14.966843 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:14.966814 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5a6e039-m5zsq"] Oct 30 17:06:14.967559 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:06:14.967525 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd2d5a07_ba42_482e_8a22_d5dec65ddeef.slice/crio-ccb6e13b27c2ee8cfe4f6adc5e4ec2f5901ffa7e7456e69c13686783ec236763 WatchSource:0}: Error finding container ccb6e13b27c2ee8cfe4f6adc5e4ec2f5901ffa7e7456e69c13686783ec236763: Status 404 returned error can't find the container with id ccb6e13b27c2ee8cfe4f6adc5e4ec2f5901ffa7e7456e69c13686783ec236763 Oct 30 17:06:15.605956 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:15.605919 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a6e039-m5zsq" event={"ID":"fd2d5a07-ba42-482e-8a22-d5dec65ddeef","Type":"ContainerStarted","Data":"77129fa5a1b5bd04b207ca258f437a95bbf0871bec89d90ceec39827fd436cb9"} Oct 30 17:06:15.606138 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:15.605984 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a6e039-m5zsq" event={"ID":"fd2d5a07-ba42-482e-8a22-d5dec65ddeef","Type":"ContainerStarted","Data":"ccb6e13b27c2ee8cfe4f6adc5e4ec2f5901ffa7e7456e69c13686783ec236763"} Oct 30 17:06:15.607657 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:15.607625 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5dbf38c-4wrxh" Oct 30 17:06:15.607996 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:15.607947 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5dbf38c-4wrxh" event={"ID":"f844002e-4790-452b-88bf-7e33af30001e","Type":"ContainerDied","Data":"2d2539cd8974d33fe8acdd891497915682b59da235113ec9e144f335dd4604f9"} Oct 30 17:06:15.608121 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:15.607997 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d2539cd8974d33fe8acdd891497915682b59da235113ec9e144f335dd4604f9" Oct 30 17:06:15.652114 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:15.652054 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a6e039-m5zsq" podStartSLOduration=1.6520346460000002 podStartE2EDuration="1.652034646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:06:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:06:15.648675741 +0000 UTC m=+2602.875850842" watchObservedRunningTime="2026-10-30 17:06:15.652034646 +0000 UTC m=+2602.879209748" Oct 30 17:06:16.620328 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:16.620289 4493 generic.go:356] "Generic (PLEG): container finished" podID="fd2d5a07-ba42-482e-8a22-d5dec65ddeef" containerID="77129fa5a1b5bd04b207ca258f437a95bbf0871bec89d90ceec39827fd436cb9" exitCode=0 Oct 30 17:06:16.620548 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:16.620380 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a6e039-m5zsq" event={"ID":"fd2d5a07-ba42-482e-8a22-d5dec65ddeef","Type":"ContainerDied","Data":"77129fa5a1b5bd04b207ca258f437a95bbf0871bec89d90ceec39827fd436cb9"} Oct 30 17:06:17.780151 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:17.780126 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a6e039-m5zsq" Oct 30 17:06:17.874296 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:17.874266 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd2d5a07-ba42-482e-8a22-d5dec65ddeef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd2d5a07-ba42-482e-8a22-d5dec65ddeef-maas-api-xks-ca\") pod \"fd2d5a07-ba42-482e-8a22-d5dec65ddeef\" (UID: \"fd2d5a07-ba42-482e-8a22-d5dec65ddeef\") " Oct 30 17:06:17.874493 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:17.874360 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd2d5a07-ba42-482e-8a22-d5dec65ddeef-kube-api-access-vg5kp\" (UniqueName: \"kubernetes.io/projected/fd2d5a07-ba42-482e-8a22-d5dec65ddeef-kube-api-access-vg5kp\") pod \"fd2d5a07-ba42-482e-8a22-d5dec65ddeef\" (UID: \"fd2d5a07-ba42-482e-8a22-d5dec65ddeef\") " Oct 30 17:06:17.874609 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:17.874590 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd2d5a07-ba42-482e-8a22-d5dec65ddeef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd2d5a07-ba42-482e-8a22-d5dec65ddeef-maas-api-openshift-ca\") pod \"fd2d5a07-ba42-482e-8a22-d5dec65ddeef\" (UID: \"fd2d5a07-ba42-482e-8a22-d5dec65ddeef\") " Oct 30 17:06:17.875126 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:17.875024 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd2d5a07-ba42-482e-8a22-d5dec65ddeef-maas-api-openshift-ca" pod "fd2d5a07-ba42-482e-8a22-d5dec65ddeef" (UID: "fd2d5a07-ba42-482e-8a22-d5dec65ddeef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:06:17.875331 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:17.875307 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd2d5a07-ba42-482e-8a22-d5dec65ddeef-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:06:17.876588 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:17.876560 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd2d5a07-ba42-482e-8a22-d5dec65ddeef-maas-api-xks-ca" pod "fd2d5a07-ba42-482e-8a22-d5dec65ddeef" (UID: "fd2d5a07-ba42-482e-8a22-d5dec65ddeef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:06:17.876703 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:17.876677 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd2d5a07-ba42-482e-8a22-d5dec65ddeef-kube-api-access-vg5kp" pod "fd2d5a07-ba42-482e-8a22-d5dec65ddeef" (UID: "fd2d5a07-ba42-482e-8a22-d5dec65ddeef"). InnerVolumeSpecName "kube-api-access-vg5kp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:06:17.977007 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:17.976962 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg5kp\" (UniqueName: \"kubernetes.io/projected/fd2d5a07-ba42-482e-8a22-d5dec65ddeef-kube-api-access-vg5kp\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:06:17.977007 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:17.977002 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd2d5a07-ba42-482e-8a22-d5dec65ddeef-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:06:18.640022 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:18.639990 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a6e039-m5zsq" Oct 30 17:06:18.640249 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:18.639990 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a6e039-m5zsq" event={"ID":"fd2d5a07-ba42-482e-8a22-d5dec65ddeef","Type":"ContainerDied","Data":"ccb6e13b27c2ee8cfe4f6adc5e4ec2f5901ffa7e7456e69c13686783ec236763"} Oct 30 17:06:18.640249 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:18.640131 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ccb6e13b27c2ee8cfe4f6adc5e4ec2f5901ffa7e7456e69c13686783ec236763" Oct 30 17:06:26.202824 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:26.190891 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a7f50df-xftrt"] Oct 30 17:06:26.202824 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:26.191734 4493 cpu_manager.go:414] "containerMap: removing container" podUID="fd2d5a07-ba42-482e-8a22-d5dec65ddeef" containerName="revoke-keys" Oct 30 17:06:26.202824 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:26.191752 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd2d5a07-ba42-482e-8a22-d5dec65ddeef" containerName="revoke-keys" Oct 30 17:06:26.202824 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:26.191767 4493 cpu_manager.go:414] "containerMap: removing container" podUID="f844002e-4790-452b-88bf-7e33af30001e" containerName="revoke-keys" Oct 30 17:06:26.202824 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:26.191774 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f844002e-4790-452b-88bf-7e33af30001e" containerName="revoke-keys" Oct 30 17:06:26.202824 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:26.193021 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd2d5a07-ba42-482e-8a22-d5dec65ddeef" containerName="revoke-keys" Oct 30 17:06:26.202824 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:26.193043 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="f844002e-4790-452b-88bf-7e33af30001e" containerName="revoke-keys" Oct 30 17:06:26.292226 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:26.292185 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a7f50df-xftrt" Oct 30 17:06:26.299372 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:26.299329 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a7f50df-xftrt"] Oct 30 17:06:26.307438 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:26.307392 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:06:26.319546 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:26.319487 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:06:26.319546 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:26.319488 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 17:06:26.319546 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:26.319544 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:06:26.387131 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:26.387096 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndkc5\" (UniqueName: \"kubernetes.io/projected/136bd37d-7029-4bad-8663-091b30bcf7d4-kube-api-access-ndkc5\") pod \"maas-api-revoke-sub-3a7f50df-xftrt\" (UID: \"136bd37d-7029-4bad-8663-091b30bcf7d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a7f50df-xftrt" Oct 30 17:06:26.387321 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:26.387272 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/136bd37d-7029-4bad-8663-091b30bcf7d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a7f50df-xftrt\" (UID: \"136bd37d-7029-4bad-8663-091b30bcf7d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a7f50df-xftrt" Oct 30 17:06:26.387321 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:26.387302 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/136bd37d-7029-4bad-8663-091b30bcf7d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a7f50df-xftrt\" (UID: \"136bd37d-7029-4bad-8663-091b30bcf7d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a7f50df-xftrt" Oct 30 17:06:26.489064 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:26.488976 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndkc5\" (UniqueName: \"kubernetes.io/projected/136bd37d-7029-4bad-8663-091b30bcf7d4-kube-api-access-ndkc5\") pod \"maas-api-revoke-sub-3a7f50df-xftrt\" (UID: \"136bd37d-7029-4bad-8663-091b30bcf7d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a7f50df-xftrt" Oct 30 17:06:26.489230 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:26.489130 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/136bd37d-7029-4bad-8663-091b30bcf7d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a7f50df-xftrt\" (UID: \"136bd37d-7029-4bad-8663-091b30bcf7d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a7f50df-xftrt" Oct 30 17:06:26.489230 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:26.489157 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/136bd37d-7029-4bad-8663-091b30bcf7d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a7f50df-xftrt\" (UID: \"136bd37d-7029-4bad-8663-091b30bcf7d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a7f50df-xftrt" Oct 30 17:06:26.489776 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:26.489757 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/136bd37d-7029-4bad-8663-091b30bcf7d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a7f50df-xftrt\" (UID: \"136bd37d-7029-4bad-8663-091b30bcf7d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a7f50df-xftrt" Oct 30 17:06:26.491706 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:26.491686 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/136bd37d-7029-4bad-8663-091b30bcf7d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a7f50df-xftrt\" (UID: \"136bd37d-7029-4bad-8663-091b30bcf7d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a7f50df-xftrt" Oct 30 17:06:26.512508 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:26.512470 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndkc5\" (UniqueName: \"kubernetes.io/projected/136bd37d-7029-4bad-8663-091b30bcf7d4-kube-api-access-ndkc5\") pod \"maas-api-revoke-sub-3a7f50df-xftrt\" (UID: \"136bd37d-7029-4bad-8663-091b30bcf7d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a7f50df-xftrt" Oct 30 17:06:26.614373 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:26.614333 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a7f50df-xftrt" Oct 30 17:06:26.955271 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:26.955246 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a7f50df-xftrt"] Oct 30 17:06:26.956340 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:06:26.956314 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod136bd37d_7029_4bad_8663_091b30bcf7d4.slice/crio-54dd7c8cf0704cea955af7c5e5a41dd6f7f0b6de84a8a59e1182e0b51906e2e9 WatchSource:0}: Error finding container 54dd7c8cf0704cea955af7c5e5a41dd6f7f0b6de84a8a59e1182e0b51906e2e9: Status 404 returned error can't find the container with id 54dd7c8cf0704cea955af7c5e5a41dd6f7f0b6de84a8a59e1182e0b51906e2e9 Oct 30 17:06:27.752412 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:27.752374 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a7f50df-xftrt" event={"ID":"136bd37d-7029-4bad-8663-091b30bcf7d4","Type":"ContainerStarted","Data":"cf4e0bcc91e29125473957f539063d00433d62e34aea80f9c03dcf1ca7000d58"} Oct 30 17:06:27.752951 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:27.752434 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a7f50df-xftrt" event={"ID":"136bd37d-7029-4bad-8663-091b30bcf7d4","Type":"ContainerStarted","Data":"54dd7c8cf0704cea955af7c5e5a41dd6f7f0b6de84a8a59e1182e0b51906e2e9"} Oct 30 17:06:27.794116 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:27.794061 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a7f50df-xftrt" podStartSLOduration=1.794046858 podStartE2EDuration="1.794046858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:06:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:06:27.792317617 +0000 UTC m=+2615.019492717" watchObservedRunningTime="2026-10-30 17:06:27.794046858 +0000 UTC m=+2615.021221977" Oct 30 17:06:28.763008 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:28.762972 4493 generic.go:356] "Generic (PLEG): container finished" podID="136bd37d-7029-4bad-8663-091b30bcf7d4" containerID="cf4e0bcc91e29125473957f539063d00433d62e34aea80f9c03dcf1ca7000d58" exitCode=0 Oct 30 17:06:28.763423 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:28.763016 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a7f50df-xftrt" event={"ID":"136bd37d-7029-4bad-8663-091b30bcf7d4","Type":"ContainerDied","Data":"cf4e0bcc91e29125473957f539063d00433d62e34aea80f9c03dcf1ca7000d58"} Oct 30 17:06:29.938604 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:29.938568 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a7f50df-xftrt" Oct 30 17:06:30.042272 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:30.042179 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/136bd37d-7029-4bad-8663-091b30bcf7d4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/136bd37d-7029-4bad-8663-091b30bcf7d4-maas-api-openshift-ca\") pod \"136bd37d-7029-4bad-8663-091b30bcf7d4\" (UID: \"136bd37d-7029-4bad-8663-091b30bcf7d4\") " Oct 30 17:06:30.042449 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:30.042280 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/136bd37d-7029-4bad-8663-091b30bcf7d4-kube-api-access-ndkc5\" (UniqueName: \"kubernetes.io/projected/136bd37d-7029-4bad-8663-091b30bcf7d4-kube-api-access-ndkc5\") pod \"136bd37d-7029-4bad-8663-091b30bcf7d4\" (UID: \"136bd37d-7029-4bad-8663-091b30bcf7d4\") " Oct 30 17:06:30.042449 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:30.042349 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/136bd37d-7029-4bad-8663-091b30bcf7d4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/136bd37d-7029-4bad-8663-091b30bcf7d4-maas-api-xks-ca\") pod \"136bd37d-7029-4bad-8663-091b30bcf7d4\" (UID: \"136bd37d-7029-4bad-8663-091b30bcf7d4\") " Oct 30 17:06:30.042577 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:30.042534 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/136bd37d-7029-4bad-8663-091b30bcf7d4-maas-api-openshift-ca" pod "136bd37d-7029-4bad-8663-091b30bcf7d4" (UID: "136bd37d-7029-4bad-8663-091b30bcf7d4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:06:30.042988 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:30.042962 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/136bd37d-7029-4bad-8663-091b30bcf7d4-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:06:30.044660 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:30.044637 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/136bd37d-7029-4bad-8663-091b30bcf7d4-maas-api-xks-ca" pod "136bd37d-7029-4bad-8663-091b30bcf7d4" (UID: "136bd37d-7029-4bad-8663-091b30bcf7d4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:06:30.044740 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:30.044694 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/136bd37d-7029-4bad-8663-091b30bcf7d4-kube-api-access-ndkc5" pod "136bd37d-7029-4bad-8663-091b30bcf7d4" (UID: "136bd37d-7029-4bad-8663-091b30bcf7d4"). InnerVolumeSpecName "kube-api-access-ndkc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:06:30.144774 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:30.144730 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndkc5\" (UniqueName: \"kubernetes.io/projected/136bd37d-7029-4bad-8663-091b30bcf7d4-kube-api-access-ndkc5\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:06:30.144774 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:30.144776 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/136bd37d-7029-4bad-8663-091b30bcf7d4-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:06:30.342987 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:06:30.342904 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 17:06:30.779078 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:30.779043 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a7f50df-xftrt" Oct 30 17:06:30.779078 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:30.779072 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a7f50df-xftrt" event={"ID":"136bd37d-7029-4bad-8663-091b30bcf7d4","Type":"ContainerDied","Data":"54dd7c8cf0704cea955af7c5e5a41dd6f7f0b6de84a8a59e1182e0b51906e2e9"} Oct 30 17:06:30.779309 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:30.779119 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54dd7c8cf0704cea955af7c5e5a41dd6f7f0b6de84a8a59e1182e0b51906e2e9" Oct 30 17:06:38.026819 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:38.025849 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efa5edd8-bcdch"] Oct 30 17:06:38.032948 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:38.032922 4493 cpu_manager.go:414] "containerMap: removing container" podUID="136bd37d-7029-4bad-8663-091b30bcf7d4" containerName="revoke-keys" Oct 30 17:06:38.033115 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:38.033102 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="136bd37d-7029-4bad-8663-091b30bcf7d4" containerName="revoke-keys" Oct 30 17:06:38.033477 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:38.033462 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="136bd37d-7029-4bad-8663-091b30bcf7d4" containerName="revoke-keys" Oct 30 17:06:38.039303 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:38.039268 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa5edd8-bcdch" Oct 30 17:06:38.077713 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:38.076720 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:06:38.077713 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:38.076869 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:06:38.077713 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:38.076977 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 17:06:38.077713 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:38.077027 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:06:38.123834 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:38.108392 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0262a990-e664-4be8-8925-6858a7ac443e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efa5edd8-bcdch\" (UID: \"0262a990-e664-4be8-8925-6858a7ac443e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa5edd8-bcdch" Oct 30 17:06:38.123834 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:38.108561 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znwx4\" (UniqueName: \"kubernetes.io/projected/0262a990-e664-4be8-8925-6858a7ac443e-kube-api-access-znwx4\") pod \"maas-api-revoke-sub-efa5edd8-bcdch\" (UID: \"0262a990-e664-4be8-8925-6858a7ac443e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa5edd8-bcdch" Oct 30 17:06:38.123834 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:38.108623 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0262a990-e664-4be8-8925-6858a7ac443e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efa5edd8-bcdch\" (UID: \"0262a990-e664-4be8-8925-6858a7ac443e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa5edd8-bcdch" Oct 30 17:06:38.146557 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:38.146524 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efa5edd8-bcdch"] Oct 30 17:06:38.214700 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:38.214653 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0262a990-e664-4be8-8925-6858a7ac443e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efa5edd8-bcdch\" (UID: \"0262a990-e664-4be8-8925-6858a7ac443e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa5edd8-bcdch" Oct 30 17:06:38.214924 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:38.214751 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znwx4\" (UniqueName: \"kubernetes.io/projected/0262a990-e664-4be8-8925-6858a7ac443e-kube-api-access-znwx4\") pod \"maas-api-revoke-sub-efa5edd8-bcdch\" (UID: \"0262a990-e664-4be8-8925-6858a7ac443e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa5edd8-bcdch" Oct 30 17:06:38.214924 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:38.214817 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0262a990-e664-4be8-8925-6858a7ac443e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efa5edd8-bcdch\" (UID: \"0262a990-e664-4be8-8925-6858a7ac443e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa5edd8-bcdch" Oct 30 17:06:38.229811 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:38.229488 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0262a990-e664-4be8-8925-6858a7ac443e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efa5edd8-bcdch\" (UID: \"0262a990-e664-4be8-8925-6858a7ac443e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa5edd8-bcdch" Oct 30 17:06:38.236026 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:38.235267 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0262a990-e664-4be8-8925-6858a7ac443e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efa5edd8-bcdch\" (UID: \"0262a990-e664-4be8-8925-6858a7ac443e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa5edd8-bcdch" Oct 30 17:06:38.240837 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:38.240526 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znwx4\" (UniqueName: \"kubernetes.io/projected/0262a990-e664-4be8-8925-6858a7ac443e-kube-api-access-znwx4\") pod \"maas-api-revoke-sub-efa5edd8-bcdch\" (UID: \"0262a990-e664-4be8-8925-6858a7ac443e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa5edd8-bcdch" Oct 30 17:06:38.416820 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:38.414345 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa5edd8-bcdch" Oct 30 17:06:38.823229 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:38.823200 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efa5edd8-bcdch"] Oct 30 17:06:38.825619 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:06:38.825588 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0262a990_e664_4be8_8925_6858a7ac443e.slice/crio-1d827dde8d12fd05a1780efff866fb9d09ced0fb55facbc09cbdb00401d8aefe WatchSource:0}: Error finding container 1d827dde8d12fd05a1780efff866fb9d09ced0fb55facbc09cbdb00401d8aefe: Status 404 returned error can't find the container with id 1d827dde8d12fd05a1780efff866fb9d09ced0fb55facbc09cbdb00401d8aefe Oct 30 17:06:38.855267 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:38.855236 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa5edd8-bcdch" event={"ID":"0262a990-e664-4be8-8925-6858a7ac443e","Type":"ContainerStarted","Data":"1d827dde8d12fd05a1780efff866fb9d09ced0fb55facbc09cbdb00401d8aefe"} Oct 30 17:06:39.863348 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:39.863317 4493 generic.go:356] "Generic (PLEG): container finished" podID="0262a990-e664-4be8-8925-6858a7ac443e" containerID="84791a8c1aa9ee3e28869fe2e5d77926d098fcc1e2f671fa87ce882043ca9082" exitCode=0 Oct 30 17:06:39.863683 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:39.863401 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa5edd8-bcdch" event={"ID":"0262a990-e664-4be8-8925-6858a7ac443e","Type":"ContainerDied","Data":"84791a8c1aa9ee3e28869fe2e5d77926d098fcc1e2f671fa87ce882043ca9082"} Oct 30 17:06:40.723037 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:40.722994 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 17:06:41.039070 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:41.039042 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa5edd8-bcdch" Oct 30 17:06:41.067204 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:41.067169 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0262a990-e664-4be8-8925-6858a7ac443e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0262a990-e664-4be8-8925-6858a7ac443e-maas-api-xks-ca\") pod \"0262a990-e664-4be8-8925-6858a7ac443e\" (UID: \"0262a990-e664-4be8-8925-6858a7ac443e\") " Oct 30 17:06:41.067388 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:41.067338 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0262a990-e664-4be8-8925-6858a7ac443e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0262a990-e664-4be8-8925-6858a7ac443e-maas-api-openshift-ca\") pod \"0262a990-e664-4be8-8925-6858a7ac443e\" (UID: \"0262a990-e664-4be8-8925-6858a7ac443e\") " Oct 30 17:06:41.067388 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:41.067381 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0262a990-e664-4be8-8925-6858a7ac443e-kube-api-access-znwx4\" (UniqueName: \"kubernetes.io/projected/0262a990-e664-4be8-8925-6858a7ac443e-kube-api-access-znwx4\") pod \"0262a990-e664-4be8-8925-6858a7ac443e\" (UID: \"0262a990-e664-4be8-8925-6858a7ac443e\") " Oct 30 17:06:41.067915 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:41.067882 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0262a990-e664-4be8-8925-6858a7ac443e-maas-api-openshift-ca" pod "0262a990-e664-4be8-8925-6858a7ac443e" (UID: "0262a990-e664-4be8-8925-6858a7ac443e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:06:41.069711 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:41.069680 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0262a990-e664-4be8-8925-6858a7ac443e-maas-api-xks-ca" pod "0262a990-e664-4be8-8925-6858a7ac443e" (UID: "0262a990-e664-4be8-8925-6858a7ac443e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:06:41.070235 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:41.070213 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0262a990-e664-4be8-8925-6858a7ac443e-kube-api-access-znwx4" pod "0262a990-e664-4be8-8925-6858a7ac443e" (UID: "0262a990-e664-4be8-8925-6858a7ac443e"). InnerVolumeSpecName "kube-api-access-znwx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:06:41.170041 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:41.170012 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0262a990-e664-4be8-8925-6858a7ac443e-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:06:41.170041 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:41.170039 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znwx4\" (UniqueName: \"kubernetes.io/projected/0262a990-e664-4be8-8925-6858a7ac443e-kube-api-access-znwx4\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:06:41.170041 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:41.170051 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0262a990-e664-4be8-8925-6858a7ac443e-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:06:41.885558 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:41.885513 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa5edd8-bcdch" event={"ID":"0262a990-e664-4be8-8925-6858a7ac443e","Type":"ContainerDied","Data":"1d827dde8d12fd05a1780efff866fb9d09ced0fb55facbc09cbdb00401d8aefe"} Oct 30 17:06:41.885775 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:41.885577 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d827dde8d12fd05a1780efff866fb9d09ced0fb55facbc09cbdb00401d8aefe" Oct 30 17:06:41.885775 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:41.885651 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa5edd8-bcdch" Oct 30 17:06:52.019130 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:52.019092 4493 generic.go:356] "Generic (PLEG): container finished" podID="009a69cb-8246-4711-a31d-37d9ee856749" containerID="891f9c6a6b5517919f373e475fa945315bea74d1b1e5833dd00eea5670c0dde6" exitCode=0 Oct 30 17:06:52.019687 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:52.019172 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz" event={"ID":"009a69cb-8246-4711-a31d-37d9ee856749","Type":"ContainerDied","Data":"891f9c6a6b5517919f373e475fa945315bea74d1b1e5833dd00eea5670c0dde6"} Oct 30 17:06:52.019753 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:52.019683 4493 scope.go:122] "RemoveContainer" containerID="891f9c6a6b5517919f373e475fa945315bea74d1b1e5833dd00eea5670c0dde6" Oct 30 17:06:52.021981 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:52.021944 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:06:52.022125 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:52.022032 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:06:52.022125 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:52.022095 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:06:53.040678 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:53.040643 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz" event={"ID":"009a69cb-8246-4711-a31d-37d9ee856749","Type":"ContainerStarted","Data":"95cd765793bd4b262c39ddb7f3beba27afb8b6751ab0631132e70151908bfa8f"} Oct 30 17:06:53.228803 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:53.228752 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz" Oct 30 17:06:53.229183 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:53.229160 4493 patch_prober.go:29] interesting pod/e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body= Oct 30 17:06:53.229367 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:53.229343 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz" podUID="009a69cb-8246-4711-a31d-37d9ee856749" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" Oct 30 17:06:54.229260 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:54.229222 4493 patch_prober.go:29] interesting pod/e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body= Oct 30 17:06:54.229837 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:54.229763 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz" podUID="009a69cb-8246-4711-a31d-37d9ee856749" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" Oct 30 17:06:55.230259 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:55.230217 4493 patch_prober.go:29] interesting pod/e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body= Oct 30 17:06:55.230840 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:55.230296 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz" podUID="009a69cb-8246-4711-a31d-37d9ee856749" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" Oct 30 17:06:56.229809 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:56.229173 4493 patch_prober.go:29] interesting pod/e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body= Oct 30 17:06:56.229809 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:56.229234 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz" podUID="009a69cb-8246-4711-a31d-37d9ee856749" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" Oct 30 17:06:56.497729 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:56.497638 4493 scope.go:122] "RemoveContainer" containerID="57e3bad5a687545701e35ee329eef30575ad539de2677c94a64acd943979f9f3" Oct 30 17:06:56.530639 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:56.530610 4493 scope.go:122] "RemoveContainer" containerID="fb14f05513ed9b5a9aa042e31de214fc0bede369020345715ecbd0448f3422ba" Oct 30 17:06:57.232214 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:57.232180 4493 patch_prober.go:29] interesting pod/e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body= Oct 30 17:06:57.232534 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:57.232504 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz" podUID="009a69cb-8246-4711-a31d-37d9ee856749" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" Oct 30 17:06:58.156496 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:58.156466 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz" Oct 30 17:06:58.228755 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:58.228716 4493 patch_prober.go:29] interesting pod/e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body= Oct 30 17:06:58.228960 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:58.228771 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz" podUID="009a69cb-8246-4711-a31d-37d9ee856749" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" Oct 30 17:06:59.234336 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:59.234309 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz" Oct 30 17:06:59.235526 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:06:59.235504 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz" Oct 30 17:07:00.342118 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:07:00.342084 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 17:07:14.703176 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:07:14.703101 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6586pd4p2"] Oct 30 17:07:14.703653 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:07:14.703640 4493 cpu_manager.go:414] "containerMap: removing container" podUID="0262a990-e664-4be8-8925-6858a7ac443e" containerName="revoke-keys" Oct 30 17:07:14.703699 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:07:14.703654 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0262a990-e664-4be8-8925-6858a7ac443e" containerName="revoke-keys" Oct 30 17:07:14.703779 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:07:14.703770 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="0262a990-e664-4be8-8925-6858a7ac443e" containerName="revoke-keys" Oct 30 17:07:14.707715 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:07:14.707686 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6586pd4p2" Oct 30 17:07:14.733586 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:07:14.733550 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d38e4c56-0173-4fd0-a839-bd456e17e03a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-43197b-6586pd4p2\" (UID: \"d38e4c56-0173-4fd0-a839-bd456e17e03a\") " pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6586pd4p2" Oct 30 17:07:14.733808 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:07:14.733708 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d38e4c56-0173-4fd0-a839-bd456e17e03a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-43197b-6586pd4p2\" (UID: \"d38e4c56-0173-4fd0-a839-bd456e17e03a\") " pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6586pd4p2" Oct 30 17:07:14.733897 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:07:14.733872 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csdng\" (UniqueName: \"kubernetes.io/projected/d38e4c56-0173-4fd0-a839-bd456e17e03a-kube-api-access-csdng\") pod \"payload-processing-external-model-e2e-extmod-b-43197b-6586pd4p2\" (UID: \"d38e4c56-0173-4fd0-a839-bd456e17e03a\") " pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6586pd4p2" Oct 30 17:07:14.733957 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:07:14.733906 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d38e4c56-0173-4fd0-a839-bd456e17e03a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-43197b-6586pd4p2\" (UID: \"d38e4c56-0173-4fd0-a839-bd456e17e03a\") " pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6586pd4p2" Oct 30 17:07:14.752843 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:07:14.739807 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-43197b\"/\"payload-processing-external-model-e2e--e2ae9620-dockercfg-rjgsg\"" Oct 30 17:07:14.752843 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:07:14.739948 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-43197b\"/\"kube-root-ca.crt\"" Oct 30 17:07:14.752843 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:07:14.740011 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-43197b\"/\"openshift-service-ca.crt\"" Oct 30 17:07:14.752843 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:07:14.742926 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-43197b\"/\"payload-processing-external-model-plugins-e2e-extmod-b-43197b\"" Oct 30 17:07:14.752843 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:07:14.743008 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-43197b\"/\"routing-overlay\"" Oct 30 17:07:14.752843 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:07:14.743606 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-43197b\"/\"agc-cred-af511487c8db\"" Oct 30 17:07:14.752843 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:07:14.747183 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6586pd4p2"] Oct 30 17:07:14.841884 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:07:14.839592 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csdng\" (UniqueName: \"kubernetes.io/projected/d38e4c56-0173-4fd0-a839-bd456e17e03a-kube-api-access-csdng\") pod \"payload-processing-external-model-e2e-extmod-b-43197b-6586pd4p2\" (UID: \"d38e4c56-0173-4fd0-a839-bd456e17e03a\") " pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6586pd4p2" Oct 30 17:07:14.842186 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:07:14.842155 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d38e4c56-0173-4fd0-a839-bd456e17e03a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-43197b-6586pd4p2\" (UID: \"d38e4c56-0173-4fd0-a839-bd456e17e03a\") " pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6586pd4p2" Oct 30 17:07:14.865902 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:07:14.865856 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d38e4c56-0173-4fd0-a839-bd456e17e03a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-43197b-6586pd4p2\" (UID: \"d38e4c56-0173-4fd0-a839-bd456e17e03a\") " pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6586pd4p2" Oct 30 17:07:14.866324 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:07:14.866299 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d38e4c56-0173-4fd0-a839-bd456e17e03a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-43197b-6586pd4p2\" (UID: \"d38e4c56-0173-4fd0-a839-bd456e17e03a\") " pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6586pd4p2" Oct 30 17:07:14.890809 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:07:14.867695 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d38e4c56-0173-4fd0-a839-bd456e17e03a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-43197b-6586pd4p2\" (UID: \"d38e4c56-0173-4fd0-a839-bd456e17e03a\") " pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6586pd4p2" Oct 30 17:07:14.891223 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:07:14.869110 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d38e4c56-0173-4fd0-a839-bd456e17e03a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-43197b-6586pd4p2\" (UID: \"d38e4c56-0173-4fd0-a839-bd456e17e03a\") " pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6586pd4p2" Oct 30 17:07:14.913810 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:07:14.906340 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d38e4c56-0173-4fd0-a839-bd456e17e03a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-43197b-6586pd4p2\" (UID: \"d38e4c56-0173-4fd0-a839-bd456e17e03a\") " pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6586pd4p2" Oct 30 17:07:14.938477 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:07:14.934475 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csdng\" (UniqueName: \"kubernetes.io/projected/d38e4c56-0173-4fd0-a839-bd456e17e03a-kube-api-access-csdng\") pod \"payload-processing-external-model-e2e-extmod-b-43197b-6586pd4p2\" (UID: \"d38e4c56-0173-4fd0-a839-bd456e17e03a\") " pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6586pd4p2" Oct 30 17:07:15.024293 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:07:15.024215 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6586pd4p2" Oct 30 17:07:15.330848 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:07:15.330818 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6586pd4p2"] Oct 30 17:07:15.332169 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:07:15.332142 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd38e4c56_0173_4fd0_a839_bd456e17e03a.slice/crio-0e7a8bb12ae9c4c750a789d45119d98e4c262f6f775b53a4b463256d0a5b5436 WatchSource:0}: Error finding container 0e7a8bb12ae9c4c750a789d45119d98e4c262f6f775b53a4b463256d0a5b5436: Status 404 returned error can't find the container with id 0e7a8bb12ae9c4c750a789d45119d98e4c262f6f775b53a4b463256d0a5b5436 Oct 30 17:07:16.297808 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:07:16.291022 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6586pd4p2" event={"ID":"d38e4c56-0173-4fd0-a839-bd456e17e03a","Type":"ContainerStarted","Data":"841db2f2344b81ecf045e6b3d9aaad7bf542e16a79da745832bcd46a7941475c"} Oct 30 17:07:16.297808 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:07:16.291098 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6586pd4p2" event={"ID":"d38e4c56-0173-4fd0-a839-bd456e17e03a","Type":"ContainerStarted","Data":"0e7a8bb12ae9c4c750a789d45119d98e4c262f6f775b53a4b463256d0a5b5436"} Oct 30 17:07:16.297808 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:07:16.291403 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6586pd4p2" Oct 30 17:07:16.340016 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:07:16.339962 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6586pd4p2" podStartSLOduration=2.339947808 podStartE2EDuration="2.339947808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:07:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:07:16.338332254 +0000 UTC m=+2663.565507353" watchObservedRunningTime="2026-10-30 17:07:16.339947808 +0000 UTC m=+2663.567122908" Oct 30 17:07:27.300804 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:07:27.300764 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6586pd4p2" Oct 30 17:07:30.341749 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:07:30.341707 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 17:07:32.530568 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:07:32.530535 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 17:07:36.587740 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:07:36.587705 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79201fa2-p88h9"] Oct 30 17:07:36.592042 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:07:36.592023 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79201fa2-p88h9" Oct 30 17:07:36.620986 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:07:36.620959 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:07:36.620986 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:07:36.620965 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 17:07:36.621205 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:07:36.620990 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:07:36.621205 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:07:36.621010 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:07:36.627498 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:07:36.627471 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79201fa2-p88h9"] Oct 30 17:07:36.709713 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:07:36.709682 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-946qh\" (UniqueName: \"kubernetes.io/projected/2408796b-b549-4ac6-a71f-67da68b91a68-kube-api-access-946qh\") pod \"maas-api-revoke-sub-79201fa2-p88h9\" (UID: \"2408796b-b549-4ac6-a71f-67da68b91a68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79201fa2-p88h9" Oct 30 17:07:36.709913 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:07:36.709838 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2408796b-b549-4ac6-a71f-67da68b91a68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79201fa2-p88h9\" (UID: \"2408796b-b549-4ac6-a71f-67da68b91a68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79201fa2-p88h9" Oct 30 17:07:36.709913 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:07:36.709893 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2408796b-b549-4ac6-a71f-67da68b91a68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79201fa2-p88h9\" (UID: \"2408796b-b549-4ac6-a71f-67da68b91a68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79201fa2-p88h9" Oct 30 17:07:36.811245 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:07:36.811207 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-946qh\" (UniqueName: \"kubernetes.io/projected/2408796b-b549-4ac6-a71f-67da68b91a68-kube-api-access-946qh\") pod \"maas-api-revoke-sub-79201fa2-p88h9\" (UID: \"2408796b-b549-4ac6-a71f-67da68b91a68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79201fa2-p88h9" Oct 30 17:07:36.811453 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:07:36.811341 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2408796b-b549-4ac6-a71f-67da68b91a68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79201fa2-p88h9\" (UID: \"2408796b-b549-4ac6-a71f-67da68b91a68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79201fa2-p88h9" Oct 30 17:07:36.811453 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:07:36.811398 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2408796b-b549-4ac6-a71f-67da68b91a68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79201fa2-p88h9\" (UID: \"2408796b-b549-4ac6-a71f-67da68b91a68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79201fa2-p88h9" Oct 30 17:07:36.812301 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:07:36.812275 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2408796b-b549-4ac6-a71f-67da68b91a68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79201fa2-p88h9\" (UID: \"2408796b-b549-4ac6-a71f-67da68b91a68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79201fa2-p88h9" Oct 30 17:07:36.814086 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:07:36.814060 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2408796b-b549-4ac6-a71f-67da68b91a68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79201fa2-p88h9\" (UID: \"2408796b-b549-4ac6-a71f-67da68b91a68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79201fa2-p88h9" Oct 30 17:07:36.834174 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:07:36.834148 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-946qh\" (UniqueName: \"kubernetes.io/projected/2408796b-b549-4ac6-a71f-67da68b91a68-kube-api-access-946qh\") pod \"maas-api-revoke-sub-79201fa2-p88h9\" (UID: \"2408796b-b549-4ac6-a71f-67da68b91a68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79201fa2-p88h9" Oct 30 17:07:36.906624 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:07:36.906595 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79201fa2-p88h9" Oct 30 17:07:37.219556 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:07:37.219527 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79201fa2-p88h9"] Oct 30 17:07:37.221592 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:07:37.221564 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2408796b_b549_4ac6_a71f_67da68b91a68.slice/crio-93037b9be45095598011e2cf9c30495fd878dcbdfcc2af263e12265b08fae4ca WatchSource:0}: Error finding container 93037b9be45095598011e2cf9c30495fd878dcbdfcc2af263e12265b08fae4ca: Status 404 returned error can't find the container with id 93037b9be45095598011e2cf9c30495fd878dcbdfcc2af263e12265b08fae4ca Oct 30 17:07:37.485307 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:07:37.485212 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79201fa2-p88h9" event={"ID":"2408796b-b549-4ac6-a71f-67da68b91a68","Type":"ContainerStarted","Data":"93037b9be45095598011e2cf9c30495fd878dcbdfcc2af263e12265b08fae4ca"} Oct 30 17:07:38.494742 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:07:38.494644 4493 generic.go:356] "Generic (PLEG): container finished" podID="2408796b-b549-4ac6-a71f-67da68b91a68" containerID="26123186e843d2f0aa42d61ec1dd3877077c137d7fb39d975ad3670007c56f51" exitCode=0 Oct 30 17:07:38.495124 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:07:38.494735 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79201fa2-p88h9" event={"ID":"2408796b-b549-4ac6-a71f-67da68b91a68","Type":"ContainerDied","Data":"26123186e843d2f0aa42d61ec1dd3877077c137d7fb39d975ad3670007c56f51"} Oct 30 17:07:39.671726 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:07:39.671697 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79201fa2-p88h9" Oct 30 17:07:39.763530 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:07:39.763428 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2408796b-b549-4ac6-a71f-67da68b91a68-kube-api-access-946qh\" (UniqueName: \"kubernetes.io/projected/2408796b-b549-4ac6-a71f-67da68b91a68-kube-api-access-946qh\") pod \"2408796b-b549-4ac6-a71f-67da68b91a68\" (UID: \"2408796b-b549-4ac6-a71f-67da68b91a68\") " Oct 30 17:07:39.763719 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:07:39.763565 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2408796b-b549-4ac6-a71f-67da68b91a68-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2408796b-b549-4ac6-a71f-67da68b91a68-maas-api-xks-ca\") pod \"2408796b-b549-4ac6-a71f-67da68b91a68\" (UID: \"2408796b-b549-4ac6-a71f-67da68b91a68\") " Oct 30 17:07:39.763719 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:07:39.763622 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2408796b-b549-4ac6-a71f-67da68b91a68-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2408796b-b549-4ac6-a71f-67da68b91a68-maas-api-openshift-ca\") pod \"2408796b-b549-4ac6-a71f-67da68b91a68\" (UID: \"2408796b-b549-4ac6-a71f-67da68b91a68\") " Oct 30 17:07:39.764338 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:07:39.764019 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2408796b-b549-4ac6-a71f-67da68b91a68-maas-api-openshift-ca" pod "2408796b-b549-4ac6-a71f-67da68b91a68" (UID: "2408796b-b549-4ac6-a71f-67da68b91a68"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:07:39.764338 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:07:39.764296 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2408796b-b549-4ac6-a71f-67da68b91a68-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:07:39.766158 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:07:39.766130 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2408796b-b549-4ac6-a71f-67da68b91a68-maas-api-xks-ca" pod "2408796b-b549-4ac6-a71f-67da68b91a68" (UID: "2408796b-b549-4ac6-a71f-67da68b91a68"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:07:39.766269 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:07:39.766185 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2408796b-b549-4ac6-a71f-67da68b91a68-kube-api-access-946qh" pod "2408796b-b549-4ac6-a71f-67da68b91a68" (UID: "2408796b-b549-4ac6-a71f-67da68b91a68"). InnerVolumeSpecName "kube-api-access-946qh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:07:39.865840 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:07:39.865776 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-946qh\" (UniqueName: \"kubernetes.io/projected/2408796b-b549-4ac6-a71f-67da68b91a68-kube-api-access-946qh\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:07:39.865840 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:07:39.865836 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2408796b-b549-4ac6-a71f-67da68b91a68-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:07:40.510291 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:07:40.510259 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79201fa2-p88h9" Oct 30 17:07:40.510291 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:07:40.510287 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79201fa2-p88h9" event={"ID":"2408796b-b549-4ac6-a71f-67da68b91a68","Type":"ContainerDied","Data":"93037b9be45095598011e2cf9c30495fd878dcbdfcc2af263e12265b08fae4ca"} Oct 30 17:07:40.510518 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:07:40.510335 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93037b9be45095598011e2cf9c30495fd878dcbdfcc2af263e12265b08fae4ca" Oct 30 17:07:54.716518 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:07:54.716492 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-56v9s_d3a1c417-2698-4462-b7ff-1d2e66d7a10f/ovn-acl-logging/0.log" Oct 30 17:07:54.724683 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:07:54.724658 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-56v9s_d3a1c417-2698-4462-b7ff-1d2e66d7a10f/ovn-acl-logging/0.log" Oct 30 17:08:00.342653 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:08:00.342606 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 17:08:19.482505 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:19.482470 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1ff3a7a-h52f7"] Oct 30 17:08:19.483259 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:19.483237 4493 cpu_manager.go:414] "containerMap: removing container" podUID="2408796b-b549-4ac6-a71f-67da68b91a68" containerName="revoke-keys" Oct 30 17:08:19.483370 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:19.483260 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2408796b-b549-4ac6-a71f-67da68b91a68" containerName="revoke-keys" Oct 30 17:08:19.483443 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:19.483427 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="2408796b-b549-4ac6-a71f-67da68b91a68" containerName="revoke-keys" Oct 30 17:08:19.487442 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:19.487420 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1ff3a7a-h52f7" Oct 30 17:08:19.504328 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:19.504292 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:08:19.516509 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:19.516481 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:08:19.516670 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:19.516568 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:08:19.516670 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:19.516634 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 17:08:19.523133 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:19.523108 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1ff3a7a-h52f7"] Oct 30 17:08:19.557261 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:19.557219 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43d23a76-3bc5-44a6-b830-3e838afe638e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1ff3a7a-h52f7\" (UID: \"43d23a76-3bc5-44a6-b830-3e838afe638e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1ff3a7a-h52f7" Oct 30 17:08:19.557466 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:19.557276 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43d23a76-3bc5-44a6-b830-3e838afe638e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1ff3a7a-h52f7\" (UID: \"43d23a76-3bc5-44a6-b830-3e838afe638e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1ff3a7a-h52f7" Oct 30 17:08:19.557466 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:19.557375 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5858\" (UniqueName: \"kubernetes.io/projected/43d23a76-3bc5-44a6-b830-3e838afe638e-kube-api-access-w5858\") pod \"maas-api-revoke-sub-b1ff3a7a-h52f7\" (UID: \"43d23a76-3bc5-44a6-b830-3e838afe638e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1ff3a7a-h52f7" Oct 30 17:08:19.658749 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:19.658711 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43d23a76-3bc5-44a6-b830-3e838afe638e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1ff3a7a-h52f7\" (UID: \"43d23a76-3bc5-44a6-b830-3e838afe638e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1ff3a7a-h52f7" Oct 30 17:08:19.658749 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:19.658756 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43d23a76-3bc5-44a6-b830-3e838afe638e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1ff3a7a-h52f7\" (UID: \"43d23a76-3bc5-44a6-b830-3e838afe638e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1ff3a7a-h52f7" Oct 30 17:08:19.659031 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:19.658886 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5858\" (UniqueName: \"kubernetes.io/projected/43d23a76-3bc5-44a6-b830-3e838afe638e-kube-api-access-w5858\") pod \"maas-api-revoke-sub-b1ff3a7a-h52f7\" (UID: \"43d23a76-3bc5-44a6-b830-3e838afe638e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1ff3a7a-h52f7" Oct 30 17:08:19.659473 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:19.659453 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43d23a76-3bc5-44a6-b830-3e838afe638e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1ff3a7a-h52f7\" (UID: \"43d23a76-3bc5-44a6-b830-3e838afe638e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1ff3a7a-h52f7" Oct 30 17:08:19.661433 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:19.661406 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43d23a76-3bc5-44a6-b830-3e838afe638e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1ff3a7a-h52f7\" (UID: \"43d23a76-3bc5-44a6-b830-3e838afe638e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1ff3a7a-h52f7" Oct 30 17:08:19.682431 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:19.682405 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5858\" (UniqueName: \"kubernetes.io/projected/43d23a76-3bc5-44a6-b830-3e838afe638e-kube-api-access-w5858\") pod \"maas-api-revoke-sub-b1ff3a7a-h52f7\" (UID: \"43d23a76-3bc5-44a6-b830-3e838afe638e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1ff3a7a-h52f7" Oct 30 17:08:19.802607 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:19.802527 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1ff3a7a-h52f7" Oct 30 17:08:20.119272 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:20.119232 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1ff3a7a-h52f7"] Oct 30 17:08:20.119699 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:08:20.119671 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43d23a76_3bc5_44a6_b830_3e838afe638e.slice/crio-9ca1dee840572dd8243908c6b90f18ea76d5ca464fafc0fad19ada6ddbebf99c WatchSource:0}: Error finding container 9ca1dee840572dd8243908c6b90f18ea76d5ca464fafc0fad19ada6ddbebf99c: Status 404 returned error can't find the container with id 9ca1dee840572dd8243908c6b90f18ea76d5ca464fafc0fad19ada6ddbebf99c Oct 30 17:08:20.848076 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:20.847945 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1ff3a7a-h52f7" event={"ID":"43d23a76-3bc5-44a6-b830-3e838afe638e","Type":"ContainerStarted","Data":"9b1052ea46c2b365c5cd42ac33c75b7597c8c56aa3a7e708c8e93a87751fe8e3"} Oct 30 17:08:20.848665 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:20.848639 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1ff3a7a-h52f7" event={"ID":"43d23a76-3bc5-44a6-b830-3e838afe638e","Type":"ContainerStarted","Data":"9ca1dee840572dd8243908c6b90f18ea76d5ca464fafc0fad19ada6ddbebf99c"} Oct 30 17:08:20.891603 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:20.891551 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1ff3a7a-h52f7" podStartSLOduration=1.891537267 podStartE2EDuration="1.891537267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:08:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:08:20.888379848 +0000 UTC m=+2728.115554948" watchObservedRunningTime="2026-10-30 17:08:20.891537267 +0000 UTC m=+2728.118712367" Oct 30 17:08:21.857475 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:21.857440 4493 generic.go:356] "Generic (PLEG): container finished" podID="43d23a76-3bc5-44a6-b830-3e838afe638e" containerID="9b1052ea46c2b365c5cd42ac33c75b7597c8c56aa3a7e708c8e93a87751fe8e3" exitCode=0 Oct 30 17:08:21.857978 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:21.857535 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1ff3a7a-h52f7" event={"ID":"43d23a76-3bc5-44a6-b830-3e838afe638e","Type":"ContainerDied","Data":"9b1052ea46c2b365c5cd42ac33c75b7597c8c56aa3a7e708c8e93a87751fe8e3"} Oct 30 17:08:23.040066 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:23.040041 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1ff3a7a-h52f7" Oct 30 17:08:23.098892 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:23.098856 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43d23a76-3bc5-44a6-b830-3e838afe638e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43d23a76-3bc5-44a6-b830-3e838afe638e-maas-api-openshift-ca\") pod \"43d23a76-3bc5-44a6-b830-3e838afe638e\" (UID: \"43d23a76-3bc5-44a6-b830-3e838afe638e\") " Oct 30 17:08:23.099064 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:23.098935 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43d23a76-3bc5-44a6-b830-3e838afe638e-kube-api-access-w5858\" (UniqueName: \"kubernetes.io/projected/43d23a76-3bc5-44a6-b830-3e838afe638e-kube-api-access-w5858\") pod \"43d23a76-3bc5-44a6-b830-3e838afe638e\" (UID: \"43d23a76-3bc5-44a6-b830-3e838afe638e\") " Oct 30 17:08:23.099064 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:23.099050 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43d23a76-3bc5-44a6-b830-3e838afe638e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43d23a76-3bc5-44a6-b830-3e838afe638e-maas-api-xks-ca\") pod \"43d23a76-3bc5-44a6-b830-3e838afe638e\" (UID: \"43d23a76-3bc5-44a6-b830-3e838afe638e\") " Oct 30 17:08:23.099291 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:23.099266 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43d23a76-3bc5-44a6-b830-3e838afe638e-maas-api-openshift-ca" pod "43d23a76-3bc5-44a6-b830-3e838afe638e" (UID: "43d23a76-3bc5-44a6-b830-3e838afe638e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:08:23.099705 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:23.099686 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43d23a76-3bc5-44a6-b830-3e838afe638e-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:08:23.102523 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:23.102493 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43d23a76-3bc5-44a6-b830-3e838afe638e-maas-api-xks-ca" pod "43d23a76-3bc5-44a6-b830-3e838afe638e" (UID: "43d23a76-3bc5-44a6-b830-3e838afe638e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:08:23.102652 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:23.102599 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43d23a76-3bc5-44a6-b830-3e838afe638e-kube-api-access-w5858" pod "43d23a76-3bc5-44a6-b830-3e838afe638e" (UID: "43d23a76-3bc5-44a6-b830-3e838afe638e"). InnerVolumeSpecName "kube-api-access-w5858". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:08:23.201226 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:23.201196 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43d23a76-3bc5-44a6-b830-3e838afe638e-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:08:23.201226 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:23.201224 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5858\" (UniqueName: \"kubernetes.io/projected/43d23a76-3bc5-44a6-b830-3e838afe638e-kube-api-access-w5858\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:08:23.875424 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:23.875394 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1ff3a7a-h52f7" Oct 30 17:08:23.875624 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:23.875426 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1ff3a7a-h52f7" event={"ID":"43d23a76-3bc5-44a6-b830-3e838afe638e","Type":"ContainerDied","Data":"9ca1dee840572dd8243908c6b90f18ea76d5ca464fafc0fad19ada6ddbebf99c"} Oct 30 17:08:23.875624 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:23.875475 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ca1dee840572dd8243908c6b90f18ea76d5ca464fafc0fad19ada6ddbebf99c" Oct 30 17:08:30.343306 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:08:30.343256 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 17:08:35.358637 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:35.358437 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 17:08:40.494533 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:40.494496 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d1960dd-55b85"] Oct 30 17:08:40.495077 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:40.495063 4493 cpu_manager.go:414] "containerMap: removing container" podUID="43d23a76-3bc5-44a6-b830-3e838afe638e" containerName="revoke-keys" Oct 30 17:08:40.495133 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:40.495080 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43d23a76-3bc5-44a6-b830-3e838afe638e" containerName="revoke-keys" Oct 30 17:08:40.495196 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:40.495187 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="43d23a76-3bc5-44a6-b830-3e838afe638e" containerName="revoke-keys" Oct 30 17:08:40.499081 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:40.499066 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d1960dd-55b85" Oct 30 17:08:40.532335 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:40.528270 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 17:08:40.532335 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:40.528339 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:08:40.532335 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:40.528396 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:08:40.535026 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:40.534994 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d1960dd-55b85"] Oct 30 17:08:40.535171 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:40.535122 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:08:40.608716 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:40.608679 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45117888-42f0-4c94-a55b-892b703d1889-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d1960dd-55b85\" (UID: \"45117888-42f0-4c94-a55b-892b703d1889\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d1960dd-55b85" Oct 30 17:08:40.608716 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:40.608718 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45117888-42f0-4c94-a55b-892b703d1889-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d1960dd-55b85\" (UID: \"45117888-42f0-4c94-a55b-892b703d1889\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d1960dd-55b85" Oct 30 17:08:40.608955 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:40.608904 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8fj4\" (UniqueName: \"kubernetes.io/projected/45117888-42f0-4c94-a55b-892b703d1889-kube-api-access-b8fj4\") pod \"maas-api-revoke-sub-7d1960dd-55b85\" (UID: \"45117888-42f0-4c94-a55b-892b703d1889\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d1960dd-55b85" Oct 30 17:08:40.710069 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:40.710029 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45117888-42f0-4c94-a55b-892b703d1889-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d1960dd-55b85\" (UID: \"45117888-42f0-4c94-a55b-892b703d1889\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d1960dd-55b85" Oct 30 17:08:40.710069 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:40.710070 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45117888-42f0-4c94-a55b-892b703d1889-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d1960dd-55b85\" (UID: \"45117888-42f0-4c94-a55b-892b703d1889\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d1960dd-55b85" Oct 30 17:08:40.710326 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:40.710143 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8fj4\" (UniqueName: \"kubernetes.io/projected/45117888-42f0-4c94-a55b-892b703d1889-kube-api-access-b8fj4\") pod \"maas-api-revoke-sub-7d1960dd-55b85\" (UID: \"45117888-42f0-4c94-a55b-892b703d1889\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d1960dd-55b85" Oct 30 17:08:40.710799 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:40.710759 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45117888-42f0-4c94-a55b-892b703d1889-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d1960dd-55b85\" (UID: \"45117888-42f0-4c94-a55b-892b703d1889\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d1960dd-55b85" Oct 30 17:08:40.712582 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:40.712561 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45117888-42f0-4c94-a55b-892b703d1889-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d1960dd-55b85\" (UID: \"45117888-42f0-4c94-a55b-892b703d1889\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d1960dd-55b85" Oct 30 17:08:40.733069 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:40.733041 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8fj4\" (UniqueName: \"kubernetes.io/projected/45117888-42f0-4c94-a55b-892b703d1889-kube-api-access-b8fj4\") pod \"maas-api-revoke-sub-7d1960dd-55b85\" (UID: \"45117888-42f0-4c94-a55b-892b703d1889\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d1960dd-55b85" Oct 30 17:08:40.821681 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:40.821595 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d1960dd-55b85" Oct 30 17:08:41.130906 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:41.130874 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d1960dd-55b85"] Oct 30 17:08:42.038398 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:42.038272 4493 generic.go:356] "Generic (PLEG): container finished" podID="45117888-42f0-4c94-a55b-892b703d1889" containerID="47c52c1f3d0a8e8d452180fc23d1164ff75cb42bfdb6a62066aa1c2a671d41ac" exitCode=0 Oct 30 17:08:42.038846 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:42.038579 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d1960dd-55b85" event={"ID":"45117888-42f0-4c94-a55b-892b703d1889","Type":"ContainerDied","Data":"47c52c1f3d0a8e8d452180fc23d1164ff75cb42bfdb6a62066aa1c2a671d41ac"} Oct 30 17:08:42.038846 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:42.038645 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d1960dd-55b85" event={"ID":"45117888-42f0-4c94-a55b-892b703d1889","Type":"ContainerStarted","Data":"493ba824941dd18c85d21418b941b4edaa7c5b735868566c27b54247661ec6a6"} Oct 30 17:08:43.214672 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:43.214647 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d1960dd-55b85" Oct 30 17:08:43.349946 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:43.349859 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45117888-42f0-4c94-a55b-892b703d1889-kube-api-access-b8fj4\" (UniqueName: \"kubernetes.io/projected/45117888-42f0-4c94-a55b-892b703d1889-kube-api-access-b8fj4\") pod \"45117888-42f0-4c94-a55b-892b703d1889\" (UID: \"45117888-42f0-4c94-a55b-892b703d1889\") " Oct 30 17:08:43.349946 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:43.349912 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45117888-42f0-4c94-a55b-892b703d1889-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45117888-42f0-4c94-a55b-892b703d1889-maas-api-openshift-ca\") pod \"45117888-42f0-4c94-a55b-892b703d1889\" (UID: \"45117888-42f0-4c94-a55b-892b703d1889\") " Oct 30 17:08:43.350191 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:43.350089 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45117888-42f0-4c94-a55b-892b703d1889-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45117888-42f0-4c94-a55b-892b703d1889-maas-api-xks-ca\") pod \"45117888-42f0-4c94-a55b-892b703d1889\" (UID: \"45117888-42f0-4c94-a55b-892b703d1889\") " Oct 30 17:08:43.350320 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:43.350299 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45117888-42f0-4c94-a55b-892b703d1889-maas-api-openshift-ca" pod "45117888-42f0-4c94-a55b-892b703d1889" (UID: "45117888-42f0-4c94-a55b-892b703d1889"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:08:43.350549 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:43.350531 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45117888-42f0-4c94-a55b-892b703d1889-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:08:43.352153 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:43.352126 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45117888-42f0-4c94-a55b-892b703d1889-maas-api-xks-ca" pod "45117888-42f0-4c94-a55b-892b703d1889" (UID: "45117888-42f0-4c94-a55b-892b703d1889"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:08:43.352153 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:43.352130 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45117888-42f0-4c94-a55b-892b703d1889-kube-api-access-b8fj4" pod "45117888-42f0-4c94-a55b-892b703d1889" (UID: "45117888-42f0-4c94-a55b-892b703d1889"). InnerVolumeSpecName "kube-api-access-b8fj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:08:43.451760 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:43.451725 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45117888-42f0-4c94-a55b-892b703d1889-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:08:43.451760 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:43.451752 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8fj4\" (UniqueName: \"kubernetes.io/projected/45117888-42f0-4c94-a55b-892b703d1889-kube-api-access-b8fj4\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:08:44.055977 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:44.055940 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d1960dd-55b85" event={"ID":"45117888-42f0-4c94-a55b-892b703d1889","Type":"ContainerDied","Data":"493ba824941dd18c85d21418b941b4edaa7c5b735868566c27b54247661ec6a6"} Oct 30 17:08:44.056186 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:44.056004 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="493ba824941dd18c85d21418b941b4edaa7c5b735868566c27b54247661ec6a6" Oct 30 17:08:44.056186 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:44.055953 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d1960dd-55b85" Oct 30 17:08:51.247890 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:51.247852 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90d54b1f-nkmqg"] Oct 30 17:08:51.248746 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:51.248724 4493 cpu_manager.go:414] "containerMap: removing container" podUID="45117888-42f0-4c94-a55b-892b703d1889" containerName="revoke-keys" Oct 30 17:08:51.248888 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:51.248747 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45117888-42f0-4c94-a55b-892b703d1889" containerName="revoke-keys" Oct 30 17:08:51.248980 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:51.248966 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="45117888-42f0-4c94-a55b-892b703d1889" containerName="revoke-keys" Oct 30 17:08:51.253052 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:51.253018 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d54b1f-nkmqg" Oct 30 17:08:51.272495 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:51.272468 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90d54b1f-nkmqg"] Oct 30 17:08:51.279181 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:51.279160 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:08:51.279328 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:51.279281 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:08:51.279387 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:51.279372 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:08:51.279424 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:51.279384 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 17:08:51.351990 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:51.351950 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlvv9\" (UniqueName: \"kubernetes.io/projected/decec274-c815-4e33-948b-7966e1f420b0-kube-api-access-rlvv9\") pod \"maas-api-revoke-sub-90d54b1f-nkmqg\" (UID: \"decec274-c815-4e33-948b-7966e1f420b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d54b1f-nkmqg" Oct 30 17:08:51.352177 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:51.352083 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/decec274-c815-4e33-948b-7966e1f420b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90d54b1f-nkmqg\" (UID: \"decec274-c815-4e33-948b-7966e1f420b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d54b1f-nkmqg" Oct 30 17:08:51.352177 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:51.352132 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/decec274-c815-4e33-948b-7966e1f420b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90d54b1f-nkmqg\" (UID: \"decec274-c815-4e33-948b-7966e1f420b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d54b1f-nkmqg" Oct 30 17:08:51.453753 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:51.453714 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlvv9\" (UniqueName: \"kubernetes.io/projected/decec274-c815-4e33-948b-7966e1f420b0-kube-api-access-rlvv9\") pod \"maas-api-revoke-sub-90d54b1f-nkmqg\" (UID: \"decec274-c815-4e33-948b-7966e1f420b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d54b1f-nkmqg" Oct 30 17:08:51.453946 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:51.453846 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/decec274-c815-4e33-948b-7966e1f420b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90d54b1f-nkmqg\" (UID: \"decec274-c815-4e33-948b-7966e1f420b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d54b1f-nkmqg" Oct 30 17:08:51.453946 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:51.453916 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/decec274-c815-4e33-948b-7966e1f420b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90d54b1f-nkmqg\" (UID: \"decec274-c815-4e33-948b-7966e1f420b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d54b1f-nkmqg" Oct 30 17:08:51.454577 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:51.454550 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/decec274-c815-4e33-948b-7966e1f420b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90d54b1f-nkmqg\" (UID: \"decec274-c815-4e33-948b-7966e1f420b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d54b1f-nkmqg" Oct 30 17:08:51.456579 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:51.456555 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/decec274-c815-4e33-948b-7966e1f420b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90d54b1f-nkmqg\" (UID: \"decec274-c815-4e33-948b-7966e1f420b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d54b1f-nkmqg" Oct 30 17:08:51.476842 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:51.476819 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlvv9\" (UniqueName: \"kubernetes.io/projected/decec274-c815-4e33-948b-7966e1f420b0-kube-api-access-rlvv9\") pod \"maas-api-revoke-sub-90d54b1f-nkmqg\" (UID: \"decec274-c815-4e33-948b-7966e1f420b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d54b1f-nkmqg" Oct 30 17:08:51.566605 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:51.566527 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d54b1f-nkmqg" Oct 30 17:08:51.895630 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:51.895242 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90d54b1f-nkmqg"] Oct 30 17:08:51.912266 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:08:51.912234 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddecec274_c815_4e33_948b_7966e1f420b0.slice/crio-9c57942ff8e4e275cc0ce12125da36627ae8a87930553e3a0985eaea1e2c043a WatchSource:0}: Error finding container 9c57942ff8e4e275cc0ce12125da36627ae8a87930553e3a0985eaea1e2c043a: Status 404 returned error can't find the container with id 9c57942ff8e4e275cc0ce12125da36627ae8a87930553e3a0985eaea1e2c043a Oct 30 17:08:52.116559 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:52.116521 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d54b1f-nkmqg" event={"ID":"decec274-c815-4e33-948b-7966e1f420b0","Type":"ContainerStarted","Data":"9c57942ff8e4e275cc0ce12125da36627ae8a87930553e3a0985eaea1e2c043a"} Oct 30 17:08:53.131974 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:53.131934 4493 generic.go:356] "Generic (PLEG): container finished" podID="decec274-c815-4e33-948b-7966e1f420b0" containerID="3a9ef4f86f495b704cdbff0652445bd70f294f07ac41d6bc8a00281ddd2f57d4" exitCode=0 Oct 30 17:08:53.132464 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:53.132108 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d54b1f-nkmqg" event={"ID":"decec274-c815-4e33-948b-7966e1f420b0","Type":"ContainerDied","Data":"3a9ef4f86f495b704cdbff0652445bd70f294f07ac41d6bc8a00281ddd2f57d4"} Oct 30 17:08:53.377024 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:53.376927 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6586pd4p2"] Oct 30 17:08:54.320983 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:54.320953 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d54b1f-nkmqg" Oct 30 17:08:54.395737 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:54.395651 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/decec274-c815-4e33-948b-7966e1f420b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/decec274-c815-4e33-948b-7966e1f420b0-maas-api-xks-ca\") pod \"decec274-c815-4e33-948b-7966e1f420b0\" (UID: \"decec274-c815-4e33-948b-7966e1f420b0\") " Oct 30 17:08:54.395932 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:54.395855 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/decec274-c815-4e33-948b-7966e1f420b0-kube-api-access-rlvv9\" (UniqueName: \"kubernetes.io/projected/decec274-c815-4e33-948b-7966e1f420b0-kube-api-access-rlvv9\") pod \"decec274-c815-4e33-948b-7966e1f420b0\" (UID: \"decec274-c815-4e33-948b-7966e1f420b0\") " Oct 30 17:08:54.395932 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:54.395910 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/decec274-c815-4e33-948b-7966e1f420b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/decec274-c815-4e33-948b-7966e1f420b0-maas-api-openshift-ca\") pod \"decec274-c815-4e33-948b-7966e1f420b0\" (UID: \"decec274-c815-4e33-948b-7966e1f420b0\") " Oct 30 17:08:54.396315 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:54.396281 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/decec274-c815-4e33-948b-7966e1f420b0-maas-api-openshift-ca" pod "decec274-c815-4e33-948b-7966e1f420b0" (UID: "decec274-c815-4e33-948b-7966e1f420b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:08:54.396611 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:54.396442 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/decec274-c815-4e33-948b-7966e1f420b0-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:08:54.398028 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:54.397999 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/decec274-c815-4e33-948b-7966e1f420b0-maas-api-xks-ca" pod "decec274-c815-4e33-948b-7966e1f420b0" (UID: "decec274-c815-4e33-948b-7966e1f420b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:08:54.398271 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:54.398251 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/decec274-c815-4e33-948b-7966e1f420b0-kube-api-access-rlvv9" pod "decec274-c815-4e33-948b-7966e1f420b0" (UID: "decec274-c815-4e33-948b-7966e1f420b0"). InnerVolumeSpecName "kube-api-access-rlvv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:08:54.497985 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:54.497947 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlvv9\" (UniqueName: \"kubernetes.io/projected/decec274-c815-4e33-948b-7966e1f420b0-kube-api-access-rlvv9\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:08:54.497985 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:54.497979 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/decec274-c815-4e33-948b-7966e1f420b0-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:08:55.156127 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:55.155254 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d54b1f-nkmqg" event={"ID":"decec274-c815-4e33-948b-7966e1f420b0","Type":"ContainerDied","Data":"9c57942ff8e4e275cc0ce12125da36627ae8a87930553e3a0985eaea1e2c043a"} Oct 30 17:08:55.156127 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:55.155341 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c57942ff8e4e275cc0ce12125da36627ae8a87930553e3a0985eaea1e2c043a" Oct 30 17:08:55.156127 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:55.155445 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90d54b1f-nkmqg" Oct 30 17:08:56.579701 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:56.579673 4493 scope.go:122] "RemoveContainer" containerID="a9ed7bb3ceaf6984c5e8f08d5098cac6a78c4d7f7b4a0ed77b6dac62df2309f2" Oct 30 17:08:58.377590 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:58.377535 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6586pd4p2" podUID="d38e4c56-0173-4fd0-a839-bd456e17e03a" containerName="payload-processing" containerID="cri-o://841db2f2344b81ecf045e6b3d9aaad7bf542e16a79da745832bcd46a7941475c" gracePeriod=25 Oct 30 17:08:58.683975 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:58.683951 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6586pd4p2" Oct 30 17:08:58.864888 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:58.864856 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d38e4c56-0173-4fd0-a839-bd456e17e03a-kube-api-access-csdng\" (UniqueName: \"kubernetes.io/projected/d38e4c56-0173-4fd0-a839-bd456e17e03a-kube-api-access-csdng\") pod \"d38e4c56-0173-4fd0-a839-bd456e17e03a\" (UID: \"d38e4c56-0173-4fd0-a839-bd456e17e03a\") " Oct 30 17:08:58.865076 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:58.864992 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d38e4c56-0173-4fd0-a839-bd456e17e03a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d38e4c56-0173-4fd0-a839-bd456e17e03a-plugins-config-volume\") pod \"d38e4c56-0173-4fd0-a839-bd456e17e03a\" (UID: \"d38e4c56-0173-4fd0-a839-bd456e17e03a\") " Oct 30 17:08:58.865076 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:58.865067 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d38e4c56-0173-4fd0-a839-bd456e17e03a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d38e4c56-0173-4fd0-a839-bd456e17e03a-provider-credentials\") pod \"d38e4c56-0173-4fd0-a839-bd456e17e03a\" (UID: \"d38e4c56-0173-4fd0-a839-bd456e17e03a\") " Oct 30 17:08:58.865185 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:58.865109 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d38e4c56-0173-4fd0-a839-bd456e17e03a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d38e4c56-0173-4fd0-a839-bd456e17e03a-routing-overlay-volume\") pod \"d38e4c56-0173-4fd0-a839-bd456e17e03a\" (UID: \"d38e4c56-0173-4fd0-a839-bd456e17e03a\") " Oct 30 17:08:58.865396 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:58.865366 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d38e4c56-0173-4fd0-a839-bd456e17e03a-plugins-config-volume" pod "d38e4c56-0173-4fd0-a839-bd456e17e03a" (UID: "d38e4c56-0173-4fd0-a839-bd456e17e03a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:08:58.865505 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:58.865487 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d38e4c56-0173-4fd0-a839-bd456e17e03a-routing-overlay-volume" pod "d38e4c56-0173-4fd0-a839-bd456e17e03a" (UID: "d38e4c56-0173-4fd0-a839-bd456e17e03a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:08:58.865614 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:58.865598 4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d38e4c56-0173-4fd0-a839-bd456e17e03a-plugins-config-volume\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:08:58.867378 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:58.867357 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d38e4c56-0173-4fd0-a839-bd456e17e03a-kube-api-access-csdng" pod "d38e4c56-0173-4fd0-a839-bd456e17e03a" (UID: "d38e4c56-0173-4fd0-a839-bd456e17e03a"). InnerVolumeSpecName "kube-api-access-csdng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:08:58.867487 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:58.867387 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d38e4c56-0173-4fd0-a839-bd456e17e03a-provider-credentials" pod "d38e4c56-0173-4fd0-a839-bd456e17e03a" (UID: "d38e4c56-0173-4fd0-a839-bd456e17e03a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:08:58.966700 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:58.966624 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csdng\" (UniqueName: \"kubernetes.io/projected/d38e4c56-0173-4fd0-a839-bd456e17e03a-kube-api-access-csdng\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:08:58.966700 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:58.966654 4493 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d38e4c56-0173-4fd0-a839-bd456e17e03a-provider-credentials\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:08:58.966700 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:58.966664 4493 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d38e4c56-0173-4fd0-a839-bd456e17e03a-routing-overlay-volume\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:08:59.186384 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:59.186352 4493 generic.go:356] "Generic (PLEG): container finished" podID="d38e4c56-0173-4fd0-a839-bd456e17e03a" containerID="841db2f2344b81ecf045e6b3d9aaad7bf542e16a79da745832bcd46a7941475c" exitCode=0 Oct 30 17:08:59.186578 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:59.186409 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6586pd4p2" Oct 30 17:08:59.186578 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:59.186434 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6586pd4p2" event={"ID":"d38e4c56-0173-4fd0-a839-bd456e17e03a","Type":"ContainerDied","Data":"841db2f2344b81ecf045e6b3d9aaad7bf542e16a79da745832bcd46a7941475c"} Oct 30 17:08:59.186578 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:59.186508 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6586pd4p2" event={"ID":"d38e4c56-0173-4fd0-a839-bd456e17e03a","Type":"ContainerDied","Data":"0e7a8bb12ae9c4c750a789d45119d98e4c262f6f775b53a4b463256d0a5b5436"} Oct 30 17:08:59.186578 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:59.186544 4493 scope.go:122] "RemoveContainer" containerID="841db2f2344b81ecf045e6b3d9aaad7bf542e16a79da745832bcd46a7941475c" Oct 30 17:08:59.196656 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:59.196638 4493 scope.go:122] "RemoveContainer" containerID="841db2f2344b81ecf045e6b3d9aaad7bf542e16a79da745832bcd46a7941475c" Oct 30 17:08:59.196970 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:08:59.196948 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"841db2f2344b81ecf045e6b3d9aaad7bf542e16a79da745832bcd46a7941475c\": container with ID starting with 841db2f2344b81ecf045e6b3d9aaad7bf542e16a79da745832bcd46a7941475c not found: ID does not exist" containerID="841db2f2344b81ecf045e6b3d9aaad7bf542e16a79da745832bcd46a7941475c" Oct 30 17:08:59.197062 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:59.196977 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"841db2f2344b81ecf045e6b3d9aaad7bf542e16a79da745832bcd46a7941475c"} err="failed to get container status \"841db2f2344b81ecf045e6b3d9aaad7bf542e16a79da745832bcd46a7941475c\": rpc error: code = NotFound desc = could not find container \"841db2f2344b81ecf045e6b3d9aaad7bf542e16a79da745832bcd46a7941475c\": container with ID starting with 841db2f2344b81ecf045e6b3d9aaad7bf542e16a79da745832bcd46a7941475c not found: ID does not exist" Oct 30 17:08:59.239893 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:59.239813 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6586pd4p2"] Oct 30 17:08:59.243756 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:08:59.243725 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6586pd4p2"] Oct 30 17:09:00.344075 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:09:00.344025 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 17:09:00.936719 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:00.936681 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d38e4c56-0173-4fd0-a839-bd456e17e03a" path="/var/lib/kubelet/pods/d38e4c56-0173-4fd0-a839-bd456e17e03a/volumes" Oct 30 17:09:04.376625 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:04.376588 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a3d9d25-j5cs8"] Oct 30 17:09:04.377221 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:04.377206 4493 cpu_manager.go:414] "containerMap: removing container" podUID="d38e4c56-0173-4fd0-a839-bd456e17e03a" containerName="payload-processing" Oct 30 17:09:04.377263 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:04.377223 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d38e4c56-0173-4fd0-a839-bd456e17e03a" containerName="payload-processing" Oct 30 17:09:04.377263 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:04.377248 4493 cpu_manager.go:414] "containerMap: removing container" podUID="decec274-c815-4e33-948b-7966e1f420b0" containerName="revoke-keys" Oct 30 17:09:04.377263 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:04.377254 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="decec274-c815-4e33-948b-7966e1f420b0" containerName="revoke-keys" Oct 30 17:09:04.377379 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:04.377327 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="d38e4c56-0173-4fd0-a839-bd456e17e03a" containerName="payload-processing" Oct 30 17:09:04.377379 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:04.377340 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="decec274-c815-4e33-948b-7966e1f420b0" containerName="revoke-keys" Oct 30 17:09:04.382575 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:04.382550 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3d9d25-j5cs8" Oct 30 17:09:04.414810 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:04.411826 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:09:04.414810 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:04.412935 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:09:04.414810 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:04.413026 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:09:04.414810 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:04.413160 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 17:09:04.418633 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:04.418604 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a3d9d25-j5cs8"] Oct 30 17:09:04.478010 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:04.477978 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c41b6331-1e79-4bf4-8f1c-7728a330202b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a3d9d25-j5cs8\" (UID: \"c41b6331-1e79-4bf4-8f1c-7728a330202b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3d9d25-j5cs8" Oct 30 17:09:04.478157 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:04.478091 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c41b6331-1e79-4bf4-8f1c-7728a330202b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a3d9d25-j5cs8\" (UID: \"c41b6331-1e79-4bf4-8f1c-7728a330202b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3d9d25-j5cs8" Oct 30 17:09:04.478220 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:04.478157 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhc2p\" (UniqueName: \"kubernetes.io/projected/c41b6331-1e79-4bf4-8f1c-7728a330202b-kube-api-access-dhc2p\") pod \"maas-api-revoke-sub-8a3d9d25-j5cs8\" (UID: \"c41b6331-1e79-4bf4-8f1c-7728a330202b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3d9d25-j5cs8" Oct 30 17:09:04.579855 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:04.579816 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c41b6331-1e79-4bf4-8f1c-7728a330202b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a3d9d25-j5cs8\" (UID: \"c41b6331-1e79-4bf4-8f1c-7728a330202b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3d9d25-j5cs8" Oct 30 17:09:04.580041 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:04.579915 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c41b6331-1e79-4bf4-8f1c-7728a330202b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a3d9d25-j5cs8\" (UID: \"c41b6331-1e79-4bf4-8f1c-7728a330202b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3d9d25-j5cs8" Oct 30 17:09:04.580041 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:04.579963 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhc2p\" (UniqueName: \"kubernetes.io/projected/c41b6331-1e79-4bf4-8f1c-7728a330202b-kube-api-access-dhc2p\") pod \"maas-api-revoke-sub-8a3d9d25-j5cs8\" (UID: \"c41b6331-1e79-4bf4-8f1c-7728a330202b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3d9d25-j5cs8" Oct 30 17:09:04.580527 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:04.580503 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c41b6331-1e79-4bf4-8f1c-7728a330202b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a3d9d25-j5cs8\" (UID: \"c41b6331-1e79-4bf4-8f1c-7728a330202b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3d9d25-j5cs8" Oct 30 17:09:04.582389 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:04.582368 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c41b6331-1e79-4bf4-8f1c-7728a330202b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a3d9d25-j5cs8\" (UID: \"c41b6331-1e79-4bf4-8f1c-7728a330202b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3d9d25-j5cs8" Oct 30 17:09:04.603206 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:04.603179 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhc2p\" (UniqueName: \"kubernetes.io/projected/c41b6331-1e79-4bf4-8f1c-7728a330202b-kube-api-access-dhc2p\") pod \"maas-api-revoke-sub-8a3d9d25-j5cs8\" (UID: \"c41b6331-1e79-4bf4-8f1c-7728a330202b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3d9d25-j5cs8" Oct 30 17:09:04.697346 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:04.697320 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3d9d25-j5cs8" Oct 30 17:09:05.007501 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:05.007472 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a3d9d25-j5cs8"] Oct 30 17:09:05.008437 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:09:05.008409 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc41b6331_1e79_4bf4_8f1c_7728a330202b.slice/crio-70713ff551d0f7114ae04aabfa828e18c70593960df6180a0d1ee606c14f2557 WatchSource:0}: Error finding container 70713ff551d0f7114ae04aabfa828e18c70593960df6180a0d1ee606c14f2557: Status 404 returned error can't find the container with id 70713ff551d0f7114ae04aabfa828e18c70593960df6180a0d1ee606c14f2557 Oct 30 17:09:05.287650 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:05.287571 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3d9d25-j5cs8" event={"ID":"c41b6331-1e79-4bf4-8f1c-7728a330202b","Type":"ContainerStarted","Data":"70713ff551d0f7114ae04aabfa828e18c70593960df6180a0d1ee606c14f2557"} Oct 30 17:09:05.336703 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:05.336667 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-66f5hf8rc"] Oct 30 17:09:05.343125 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:05.343101 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-66f5hf8rc" Oct 30 17:09:05.359537 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:05.359508 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-3106d6\"/\"routing-overlay\"" Oct 30 17:09:05.359722 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:05.359704 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-3106d6\"/\"kube-root-ca.crt\"" Oct 30 17:09:05.360818 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:05.360800 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-66f5hf8rc"] Oct 30 17:09:05.368465 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:05.368441 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-3106d6\"/\"openshift-service-ca.crt\"" Oct 30 17:09:05.368465 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:05.368463 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-3106d6\"/\"payload-processing-external-model-e2e--ac3a616d-dockercfg-nw26z\"" Oct 30 17:09:05.368630 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:05.368447 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-3106d6\"/\"payload-processing-external-model-plugins-e2e-extmod-a-3106d6\"" Oct 30 17:09:05.368630 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:05.368466 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-3106d6\"/\"agc-cred-6be2aaced4b8\"" Oct 30 17:09:05.390700 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:05.390670 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75qx4\" (UniqueName: \"kubernetes.io/projected/97f6ef13-6bb7-4cac-9e89-fc299de8850d-kube-api-access-75qx4\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-66f5hf8rc\" (UID: \"97f6ef13-6bb7-4cac-9e89-fc299de8850d\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-66f5hf8rc" Oct 30 17:09:05.391110 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:05.390741 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/97f6ef13-6bb7-4cac-9e89-fc299de8850d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-66f5hf8rc\" (UID: \"97f6ef13-6bb7-4cac-9e89-fc299de8850d\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-66f5hf8rc" Oct 30 17:09:05.391110 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:05.390951 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97f6ef13-6bb7-4cac-9e89-fc299de8850d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-66f5hf8rc\" (UID: \"97f6ef13-6bb7-4cac-9e89-fc299de8850d\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-66f5hf8rc" Oct 30 17:09:05.391239 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:05.391179 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/97f6ef13-6bb7-4cac-9e89-fc299de8850d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-66f5hf8rc\" (UID: \"97f6ef13-6bb7-4cac-9e89-fc299de8850d\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-66f5hf8rc" Oct 30 17:09:05.492264 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:05.492234 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/97f6ef13-6bb7-4cac-9e89-fc299de8850d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-66f5hf8rc\" (UID: \"97f6ef13-6bb7-4cac-9e89-fc299de8850d\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-66f5hf8rc" Oct 30 17:09:05.492433 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:05.492317 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97f6ef13-6bb7-4cac-9e89-fc299de8850d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-66f5hf8rc\" (UID: \"97f6ef13-6bb7-4cac-9e89-fc299de8850d\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-66f5hf8rc" Oct 30 17:09:05.492515 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:05.492497 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/97f6ef13-6bb7-4cac-9e89-fc299de8850d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-66f5hf8rc\" (UID: \"97f6ef13-6bb7-4cac-9e89-fc299de8850d\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-66f5hf8rc" Oct 30 17:09:05.492580 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:05.492538 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75qx4\" (UniqueName: \"kubernetes.io/projected/97f6ef13-6bb7-4cac-9e89-fc299de8850d-kube-api-access-75qx4\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-66f5hf8rc\" (UID: \"97f6ef13-6bb7-4cac-9e89-fc299de8850d\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-66f5hf8rc" Oct 30 17:09:05.493020 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:05.492996 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/97f6ef13-6bb7-4cac-9e89-fc299de8850d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-66f5hf8rc\" (UID: \"97f6ef13-6bb7-4cac-9e89-fc299de8850d\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-66f5hf8rc" Oct 30 17:09:05.493128 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:05.493000 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97f6ef13-6bb7-4cac-9e89-fc299de8850d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-66f5hf8rc\" (UID: \"97f6ef13-6bb7-4cac-9e89-fc299de8850d\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-66f5hf8rc" Oct 30 17:09:05.495142 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:05.495125 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/97f6ef13-6bb7-4cac-9e89-fc299de8850d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-66f5hf8rc\" (UID: \"97f6ef13-6bb7-4cac-9e89-fc299de8850d\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-66f5hf8rc" Oct 30 17:09:05.517159 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:05.517134 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75qx4\" (UniqueName: \"kubernetes.io/projected/97f6ef13-6bb7-4cac-9e89-fc299de8850d-kube-api-access-75qx4\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-66f5hf8rc\" (UID: \"97f6ef13-6bb7-4cac-9e89-fc299de8850d\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-66f5hf8rc" Oct 30 17:09:05.658035 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:05.658004 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-66f5hf8rc" Oct 30 17:09:05.975733 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:05.975700 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-66f5hf8rc"] Oct 30 17:09:05.976369 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:09:05.976342 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97f6ef13_6bb7_4cac_9e89_fc299de8850d.slice/crio-e56f20b4f17240ca336929a8bd62c6f03b7a571e0e39d6e8a3538eda74343492 WatchSource:0}: Error finding container e56f20b4f17240ca336929a8bd62c6f03b7a571e0e39d6e8a3538eda74343492: Status 404 returned error can't find the container with id e56f20b4f17240ca336929a8bd62c6f03b7a571e0e39d6e8a3538eda74343492 Oct 30 17:09:06.316891 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:06.313433 4493 generic.go:356] "Generic (PLEG): container finished" podID="c41b6331-1e79-4bf4-8f1c-7728a330202b" containerID="54ba4a27e1edb2a0aedb7ce3b6311acdabb02860cc4ce55dd6cf0261785a44f9" exitCode=0 Oct 30 17:09:06.316891 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:06.313505 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3d9d25-j5cs8" event={"ID":"c41b6331-1e79-4bf4-8f1c-7728a330202b","Type":"ContainerDied","Data":"54ba4a27e1edb2a0aedb7ce3b6311acdabb02860cc4ce55dd6cf0261785a44f9"} Oct 30 17:09:06.316891 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:06.315812 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-66f5hf8rc" event={"ID":"97f6ef13-6bb7-4cac-9e89-fc299de8850d","Type":"ContainerStarted","Data":"e5a55e06e36c650cd3478c137a8edc1cb7378116b6187f2f15179d356d62502f"} Oct 30 17:09:06.316891 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:06.315858 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-66f5hf8rc" event={"ID":"97f6ef13-6bb7-4cac-9e89-fc299de8850d","Type":"ContainerStarted","Data":"e56f20b4f17240ca336929a8bd62c6f03b7a571e0e39d6e8a3538eda74343492"} Oct 30 17:09:06.316891 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:06.315898 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-66f5hf8rc" Oct 30 17:09:06.395416 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:06.395356 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-66f5hf8rc" podStartSLOduration=1.395341315 podStartE2EDuration="1.395341315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:09:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:09:06.394198667 +0000 UTC m=+2773.621373767" watchObservedRunningTime="2026-10-30 17:09:06.395341315 +0000 UTC m=+2773.622516415" Oct 30 17:09:07.493985 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:07.493955 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3d9d25-j5cs8" Oct 30 17:09:07.622301 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:07.622201 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c41b6331-1e79-4bf4-8f1c-7728a330202b-kube-api-access-dhc2p\" (UniqueName: \"kubernetes.io/projected/c41b6331-1e79-4bf4-8f1c-7728a330202b-kube-api-access-dhc2p\") pod \"c41b6331-1e79-4bf4-8f1c-7728a330202b\" (UID: \"c41b6331-1e79-4bf4-8f1c-7728a330202b\") " Oct 30 17:09:07.622301 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:07.622259 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c41b6331-1e79-4bf4-8f1c-7728a330202b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c41b6331-1e79-4bf4-8f1c-7728a330202b-maas-api-xks-ca\") pod \"c41b6331-1e79-4bf4-8f1c-7728a330202b\" (UID: \"c41b6331-1e79-4bf4-8f1c-7728a330202b\") " Oct 30 17:09:07.622584 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:07.622491 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c41b6331-1e79-4bf4-8f1c-7728a330202b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c41b6331-1e79-4bf4-8f1c-7728a330202b-maas-api-openshift-ca\") pod \"c41b6331-1e79-4bf4-8f1c-7728a330202b\" (UID: \"c41b6331-1e79-4bf4-8f1c-7728a330202b\") " Oct 30 17:09:07.622924 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:07.622899 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c41b6331-1e79-4bf4-8f1c-7728a330202b-maas-api-openshift-ca" pod "c41b6331-1e79-4bf4-8f1c-7728a330202b" (UID: "c41b6331-1e79-4bf4-8f1c-7728a330202b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:09:07.623057 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:07.623030 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c41b6331-1e79-4bf4-8f1c-7728a330202b-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:09:07.624452 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:07.624422 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c41b6331-1e79-4bf4-8f1c-7728a330202b-kube-api-access-dhc2p" pod "c41b6331-1e79-4bf4-8f1c-7728a330202b" (UID: "c41b6331-1e79-4bf4-8f1c-7728a330202b"). InnerVolumeSpecName "kube-api-access-dhc2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:09:07.624558 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:07.624491 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c41b6331-1e79-4bf4-8f1c-7728a330202b-maas-api-xks-ca" pod "c41b6331-1e79-4bf4-8f1c-7728a330202b" (UID: "c41b6331-1e79-4bf4-8f1c-7728a330202b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:09:07.724656 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:07.724624 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c41b6331-1e79-4bf4-8f1c-7728a330202b-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:09:07.724656 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:07.724654 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhc2p\" (UniqueName: \"kubernetes.io/projected/c41b6331-1e79-4bf4-8f1c-7728a330202b-kube-api-access-dhc2p\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:09:08.340597 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:08.340566 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3d9d25-j5cs8" Oct 30 17:09:08.340597 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:08.340571 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a3d9d25-j5cs8" event={"ID":"c41b6331-1e79-4bf4-8f1c-7728a330202b","Type":"ContainerDied","Data":"70713ff551d0f7114ae04aabfa828e18c70593960df6180a0d1ee606c14f2557"} Oct 30 17:09:08.340866 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:08.340626 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70713ff551d0f7114ae04aabfa828e18c70593960df6180a0d1ee606c14f2557" Oct 30 17:09:16.906261 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:16.906225 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-66f5hf8rc"] Oct 30 17:09:16.909697 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:16.909674 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-66f5hf8rc" Oct 30 17:09:21.907451 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:21.907396 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-66f5hf8rc" podUID="97f6ef13-6bb7-4cac-9e89-fc299de8850d" containerName="payload-processing" containerID="cri-o://e5a55e06e36c650cd3478c137a8edc1cb7378116b6187f2f15179d356d62502f" gracePeriod=25 Oct 30 17:09:22.182087 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:22.182061 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-66f5hf8rc" Oct 30 17:09:22.353270 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:22.353226 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97f6ef13-6bb7-4cac-9e89-fc299de8850d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/97f6ef13-6bb7-4cac-9e89-fc299de8850d-routing-overlay-volume\") pod \"97f6ef13-6bb7-4cac-9e89-fc299de8850d\" (UID: \"97f6ef13-6bb7-4cac-9e89-fc299de8850d\") " Oct 30 17:09:22.353572 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:22.353295 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97f6ef13-6bb7-4cac-9e89-fc299de8850d-kube-api-access-75qx4\" (UniqueName: \"kubernetes.io/projected/97f6ef13-6bb7-4cac-9e89-fc299de8850d-kube-api-access-75qx4\") pod \"97f6ef13-6bb7-4cac-9e89-fc299de8850d\" (UID: \"97f6ef13-6bb7-4cac-9e89-fc299de8850d\") " Oct 30 17:09:22.353572 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:22.353332 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97f6ef13-6bb7-4cac-9e89-fc299de8850d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97f6ef13-6bb7-4cac-9e89-fc299de8850d-plugins-config-volume\") pod \"97f6ef13-6bb7-4cac-9e89-fc299de8850d\" (UID: \"97f6ef13-6bb7-4cac-9e89-fc299de8850d\") " Oct 30 17:09:22.353572 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:22.353410 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97f6ef13-6bb7-4cac-9e89-fc299de8850d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/97f6ef13-6bb7-4cac-9e89-fc299de8850d-provider-credentials\") pod \"97f6ef13-6bb7-4cac-9e89-fc299de8850d\" (UID: \"97f6ef13-6bb7-4cac-9e89-fc299de8850d\") " Oct 30 17:09:22.353761 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:22.353662 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97f6ef13-6bb7-4cac-9e89-fc299de8850d-routing-overlay-volume" pod "97f6ef13-6bb7-4cac-9e89-fc299de8850d" (UID: "97f6ef13-6bb7-4cac-9e89-fc299de8850d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:09:22.354068 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:22.354046 4493 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/97f6ef13-6bb7-4cac-9e89-fc299de8850d-routing-overlay-volume\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:09:22.354163 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:22.354098 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97f6ef13-6bb7-4cac-9e89-fc299de8850d-plugins-config-volume" pod "97f6ef13-6bb7-4cac-9e89-fc299de8850d" (UID: "97f6ef13-6bb7-4cac-9e89-fc299de8850d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:09:22.356302 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:22.356280 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97f6ef13-6bb7-4cac-9e89-fc299de8850d-kube-api-access-75qx4" pod "97f6ef13-6bb7-4cac-9e89-fc299de8850d" (UID: "97f6ef13-6bb7-4cac-9e89-fc299de8850d"). InnerVolumeSpecName "kube-api-access-75qx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:09:22.356302 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:22.356291 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97f6ef13-6bb7-4cac-9e89-fc299de8850d-provider-credentials" pod "97f6ef13-6bb7-4cac-9e89-fc299de8850d" (UID: "97f6ef13-6bb7-4cac-9e89-fc299de8850d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:09:22.454854 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:22.454753 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75qx4\" (UniqueName: \"kubernetes.io/projected/97f6ef13-6bb7-4cac-9e89-fc299de8850d-kube-api-access-75qx4\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:09:22.454854 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:22.454796 4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97f6ef13-6bb7-4cac-9e89-fc299de8850d-plugins-config-volume\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:09:22.454854 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:22.454810 4493 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/97f6ef13-6bb7-4cac-9e89-fc299de8850d-provider-credentials\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:09:22.509485 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:22.509451 4493 generic.go:356] "Generic (PLEG): container finished" podID="97f6ef13-6bb7-4cac-9e89-fc299de8850d" containerID="e5a55e06e36c650cd3478c137a8edc1cb7378116b6187f2f15179d356d62502f" exitCode=0 Oct 30 17:09:22.509685 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:22.509498 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-66f5hf8rc" Oct 30 17:09:22.509685 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:22.509525 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-66f5hf8rc" event={"ID":"97f6ef13-6bb7-4cac-9e89-fc299de8850d","Type":"ContainerDied","Data":"e5a55e06e36c650cd3478c137a8edc1cb7378116b6187f2f15179d356d62502f"} Oct 30 17:09:22.509685 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:22.509588 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-66f5hf8rc" event={"ID":"97f6ef13-6bb7-4cac-9e89-fc299de8850d","Type":"ContainerDied","Data":"e56f20b4f17240ca336929a8bd62c6f03b7a571e0e39d6e8a3538eda74343492"} Oct 30 17:09:22.509685 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:22.509617 4493 scope.go:122] "RemoveContainer" containerID="e5a55e06e36c650cd3478c137a8edc1cb7378116b6187f2f15179d356d62502f" Oct 30 17:09:22.520549 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:22.520528 4493 scope.go:122] "RemoveContainer" containerID="e5a55e06e36c650cd3478c137a8edc1cb7378116b6187f2f15179d356d62502f" Oct 30 17:09:22.520884 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:09:22.520858 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5a55e06e36c650cd3478c137a8edc1cb7378116b6187f2f15179d356d62502f\": container with ID starting with e5a55e06e36c650cd3478c137a8edc1cb7378116b6187f2f15179d356d62502f not found: ID does not exist" containerID="e5a55e06e36c650cd3478c137a8edc1cb7378116b6187f2f15179d356d62502f" Oct 30 17:09:22.520976 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:22.520895 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5a55e06e36c650cd3478c137a8edc1cb7378116b6187f2f15179d356d62502f"} err="failed to get container status \"e5a55e06e36c650cd3478c137a8edc1cb7378116b6187f2f15179d356d62502f\": rpc error: code = NotFound desc = could not find container \"e5a55e06e36c650cd3478c137a8edc1cb7378116b6187f2f15179d356d62502f\": container with ID starting with e5a55e06e36c650cd3478c137a8edc1cb7378116b6187f2f15179d356d62502f not found: ID does not exist" Oct 30 17:09:22.576315 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:22.576288 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-66f5hf8rc"] Oct 30 17:09:22.587715 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:22.587683 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-66f5hf8rc"] Oct 30 17:09:22.933080 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:22.933050 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97f6ef13-6bb7-4cac-9e89-fc299de8850d" path="/var/lib/kubelet/pods/97f6ef13-6bb7-4cac-9e89-fc299de8850d/volumes" Oct 30 17:09:27.586212 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:27.586167 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 17:09:30.342945 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:09:30.342908 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 17:09:32.830324 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:32.830286 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5fff639-nw275"] Oct 30 17:09:32.831878 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:32.831849 4493 cpu_manager.go:414] "containerMap: removing container" podUID="c41b6331-1e79-4bf4-8f1c-7728a330202b" containerName="revoke-keys" Oct 30 17:09:32.832061 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:32.832047 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c41b6331-1e79-4bf4-8f1c-7728a330202b" containerName="revoke-keys" Oct 30 17:09:32.832193 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:32.832181 4493 cpu_manager.go:414] "containerMap: removing container" podUID="97f6ef13-6bb7-4cac-9e89-fc299de8850d" containerName="payload-processing" Oct 30 17:09:32.832275 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:32.832265 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97f6ef13-6bb7-4cac-9e89-fc299de8850d" containerName="payload-processing" Oct 30 17:09:32.832523 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:32.832509 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="c41b6331-1e79-4bf4-8f1c-7728a330202b" containerName="revoke-keys" Oct 30 17:09:32.832627 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:32.832615 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="97f6ef13-6bb7-4cac-9e89-fc299de8850d" containerName="payload-processing" Oct 30 17:09:32.842045 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:32.842014 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5fff639-nw275" Oct 30 17:09:32.856935 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:32.856911 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:09:32.861173 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:32.861151 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5fff639-nw275"] Oct 30 17:09:32.869060 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:32.869022 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:09:32.869177 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:32.869104 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 17:09:32.869177 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:32.869124 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:09:32.941552 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:32.941516 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86a15b69-0911-4261-a2a4-08198bfb9142-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5fff639-nw275\" (UID: \"86a15b69-0911-4261-a2a4-08198bfb9142\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5fff639-nw275" Oct 30 17:09:32.941722 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:32.941641 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gkdg\" (UniqueName: \"kubernetes.io/projected/86a15b69-0911-4261-a2a4-08198bfb9142-kube-api-access-7gkdg\") pod \"maas-api-revoke-sub-e5fff639-nw275\" (UID: \"86a15b69-0911-4261-a2a4-08198bfb9142\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5fff639-nw275" Oct 30 17:09:32.941722 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:32.941690 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86a15b69-0911-4261-a2a4-08198bfb9142-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5fff639-nw275\" (UID: \"86a15b69-0911-4261-a2a4-08198bfb9142\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5fff639-nw275" Oct 30 17:09:33.043416 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:33.043371 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gkdg\" (UniqueName: \"kubernetes.io/projected/86a15b69-0911-4261-a2a4-08198bfb9142-kube-api-access-7gkdg\") pod \"maas-api-revoke-sub-e5fff639-nw275\" (UID: \"86a15b69-0911-4261-a2a4-08198bfb9142\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5fff639-nw275" Oct 30 17:09:33.044103 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:33.043467 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86a15b69-0911-4261-a2a4-08198bfb9142-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5fff639-nw275\" (UID: \"86a15b69-0911-4261-a2a4-08198bfb9142\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5fff639-nw275" Oct 30 17:09:33.044103 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:33.043753 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86a15b69-0911-4261-a2a4-08198bfb9142-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5fff639-nw275\" (UID: \"86a15b69-0911-4261-a2a4-08198bfb9142\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5fff639-nw275" Oct 30 17:09:33.044657 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:33.044634 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86a15b69-0911-4261-a2a4-08198bfb9142-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5fff639-nw275\" (UID: \"86a15b69-0911-4261-a2a4-08198bfb9142\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5fff639-nw275" Oct 30 17:09:33.046230 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:33.046207 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86a15b69-0911-4261-a2a4-08198bfb9142-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5fff639-nw275\" (UID: \"86a15b69-0911-4261-a2a4-08198bfb9142\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5fff639-nw275" Oct 30 17:09:33.069408 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:33.069385 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gkdg\" (UniqueName: \"kubernetes.io/projected/86a15b69-0911-4261-a2a4-08198bfb9142-kube-api-access-7gkdg\") pod \"maas-api-revoke-sub-e5fff639-nw275\" (UID: \"86a15b69-0911-4261-a2a4-08198bfb9142\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5fff639-nw275" Oct 30 17:09:33.158757 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:33.158728 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5fff639-nw275" Oct 30 17:09:33.465158 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:33.465134 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5fff639-nw275"] Oct 30 17:09:33.465847 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:09:33.465820 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86a15b69_0911_4261_a2a4_08198bfb9142.slice/crio-38ba320af8a7557ac33126c701a87ffe9975f21330c4adb4a0c0479947e324a9 WatchSource:0}: Error finding container 38ba320af8a7557ac33126c701a87ffe9975f21330c4adb4a0c0479947e324a9: Status 404 returned error can't find the container with id 38ba320af8a7557ac33126c701a87ffe9975f21330c4adb4a0c0479947e324a9 Oct 30 17:09:33.673206 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:33.673174 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5fff639-nw275" event={"ID":"86a15b69-0911-4261-a2a4-08198bfb9142","Type":"ContainerStarted","Data":"38ba320af8a7557ac33126c701a87ffe9975f21330c4adb4a0c0479947e324a9"} Oct 30 17:09:34.681653 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:34.681575 4493 generic.go:356] "Generic (PLEG): container finished" podID="86a15b69-0911-4261-a2a4-08198bfb9142" containerID="88ad625b311ade19a7c792ee8678afb4e1d4bd51fe74d56bfbc748b143695695" exitCode=0 Oct 30 17:09:34.682076 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:34.681667 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5fff639-nw275" event={"ID":"86a15b69-0911-4261-a2a4-08198bfb9142","Type":"ContainerDied","Data":"88ad625b311ade19a7c792ee8678afb4e1d4bd51fe74d56bfbc748b143695695"} Oct 30 17:09:35.862314 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:35.862290 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5fff639-nw275" Oct 30 17:09:36.002307 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:36.002220 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86a15b69-0911-4261-a2a4-08198bfb9142-kube-api-access-7gkdg\" (UniqueName: \"kubernetes.io/projected/86a15b69-0911-4261-a2a4-08198bfb9142-kube-api-access-7gkdg\") pod \"86a15b69-0911-4261-a2a4-08198bfb9142\" (UID: \"86a15b69-0911-4261-a2a4-08198bfb9142\") " Oct 30 17:09:36.002475 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:36.002375 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86a15b69-0911-4261-a2a4-08198bfb9142-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86a15b69-0911-4261-a2a4-08198bfb9142-maas-api-xks-ca\") pod \"86a15b69-0911-4261-a2a4-08198bfb9142\" (UID: \"86a15b69-0911-4261-a2a4-08198bfb9142\") " Oct 30 17:09:36.002475 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:36.002445 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86a15b69-0911-4261-a2a4-08198bfb9142-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86a15b69-0911-4261-a2a4-08198bfb9142-maas-api-openshift-ca\") pod \"86a15b69-0911-4261-a2a4-08198bfb9142\" (UID: \"86a15b69-0911-4261-a2a4-08198bfb9142\") " Oct 30 17:09:36.002749 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:36.002722 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86a15b69-0911-4261-a2a4-08198bfb9142-maas-api-openshift-ca" pod "86a15b69-0911-4261-a2a4-08198bfb9142" (UID: "86a15b69-0911-4261-a2a4-08198bfb9142"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:09:36.002941 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:36.002926 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86a15b69-0911-4261-a2a4-08198bfb9142-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:09:36.004539 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:36.004510 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86a15b69-0911-4261-a2a4-08198bfb9142-kube-api-access-7gkdg" pod "86a15b69-0911-4261-a2a4-08198bfb9142" (UID: "86a15b69-0911-4261-a2a4-08198bfb9142"). InnerVolumeSpecName "kube-api-access-7gkdg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:09:36.004777 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:36.004754 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86a15b69-0911-4261-a2a4-08198bfb9142-maas-api-xks-ca" pod "86a15b69-0911-4261-a2a4-08198bfb9142" (UID: "86a15b69-0911-4261-a2a4-08198bfb9142"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:09:36.104266 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:36.104230 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gkdg\" (UniqueName: \"kubernetes.io/projected/86a15b69-0911-4261-a2a4-08198bfb9142-kube-api-access-7gkdg\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:09:36.104266 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:36.104260 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86a15b69-0911-4261-a2a4-08198bfb9142-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:09:36.698376 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:36.698348 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5fff639-nw275" Oct 30 17:09:36.698548 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:36.698379 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5fff639-nw275" event={"ID":"86a15b69-0911-4261-a2a4-08198bfb9142","Type":"ContainerDied","Data":"38ba320af8a7557ac33126c701a87ffe9975f21330c4adb4a0c0479947e324a9"} Oct 30 17:09:36.698548 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:36.698429 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="38ba320af8a7557ac33126c701a87ffe9975f21330c4adb4a0c0479947e324a9" Oct 30 17:09:44.694559 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:44.694520 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3820b22-w45fd"] Oct 30 17:09:44.695357 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:44.695341 4493 cpu_manager.go:414] "containerMap: removing container" podUID="86a15b69-0911-4261-a2a4-08198bfb9142" containerName="revoke-keys" Oct 30 17:09:44.695404 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:44.695360 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86a15b69-0911-4261-a2a4-08198bfb9142" containerName="revoke-keys" Oct 30 17:09:44.695556 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:44.695545 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="86a15b69-0911-4261-a2a4-08198bfb9142" containerName="revoke-keys" Oct 30 17:09:44.699871 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:44.699853 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3820b22-w45fd" Oct 30 17:09:44.721727 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:44.721700 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3820b22-w45fd"] Oct 30 17:09:44.732218 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:44.727109 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:09:44.732218 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:44.727169 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 17:09:44.732218 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:44.729761 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:09:44.732218 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:44.729876 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:09:44.733315 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:44.733290 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvkvx\" (UniqueName: \"kubernetes.io/projected/44f0bfa4-b78a-4cad-99e5-53bfc3979a2a-kube-api-access-wvkvx\") pod \"maas-api-revoke-sub-f3820b22-w45fd\" (UID: \"44f0bfa4-b78a-4cad-99e5-53bfc3979a2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3820b22-w45fd" Oct 30 17:09:44.733502 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:44.733473 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44f0bfa4-b78a-4cad-99e5-53bfc3979a2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3820b22-w45fd\" (UID: \"44f0bfa4-b78a-4cad-99e5-53bfc3979a2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3820b22-w45fd" Oct 30 17:09:44.733594 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:44.733577 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44f0bfa4-b78a-4cad-99e5-53bfc3979a2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3820b22-w45fd\" (UID: \"44f0bfa4-b78a-4cad-99e5-53bfc3979a2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3820b22-w45fd" Oct 30 17:09:44.835443 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:44.835403 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvkvx\" (UniqueName: \"kubernetes.io/projected/44f0bfa4-b78a-4cad-99e5-53bfc3979a2a-kube-api-access-wvkvx\") pod \"maas-api-revoke-sub-f3820b22-w45fd\" (UID: \"44f0bfa4-b78a-4cad-99e5-53bfc3979a2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3820b22-w45fd" Oct 30 17:09:44.835609 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:44.835535 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44f0bfa4-b78a-4cad-99e5-53bfc3979a2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3820b22-w45fd\" (UID: \"44f0bfa4-b78a-4cad-99e5-53bfc3979a2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3820b22-w45fd" Oct 30 17:09:44.835609 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:44.835602 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44f0bfa4-b78a-4cad-99e5-53bfc3979a2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3820b22-w45fd\" (UID: \"44f0bfa4-b78a-4cad-99e5-53bfc3979a2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3820b22-w45fd" Oct 30 17:09:44.836204 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:44.836185 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44f0bfa4-b78a-4cad-99e5-53bfc3979a2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3820b22-w45fd\" (UID: \"44f0bfa4-b78a-4cad-99e5-53bfc3979a2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3820b22-w45fd" Oct 30 17:09:44.838216 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:44.838186 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44f0bfa4-b78a-4cad-99e5-53bfc3979a2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3820b22-w45fd\" (UID: \"44f0bfa4-b78a-4cad-99e5-53bfc3979a2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3820b22-w45fd" Oct 30 17:09:44.860538 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:44.860506 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvkvx\" (UniqueName: \"kubernetes.io/projected/44f0bfa4-b78a-4cad-99e5-53bfc3979a2a-kube-api-access-wvkvx\") pod \"maas-api-revoke-sub-f3820b22-w45fd\" (UID: \"44f0bfa4-b78a-4cad-99e5-53bfc3979a2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3820b22-w45fd" Oct 30 17:09:45.017197 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:45.017128 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3820b22-w45fd" Oct 30 17:09:45.158129 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:45.158086 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7fd7cnbdl"] Oct 30 17:09:45.163616 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:45.163591 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7fd7cnbdl" Oct 30 17:09:45.182771 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:45.182747 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2589b5\"/\"payload-processing-external-model-e2e--8e3f1132-dockercfg-gjbn7\"" Oct 30 17:09:45.183093 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:45.183061 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2589b5\"/\"openshift-service-ca.crt\"" Oct 30 17:09:45.183218 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:45.183192 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2589b5\"/\"payload-processing-external-model-plugins-e2e-extmod-a-2589b5\"" Oct 30 17:09:45.194795 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:45.194767 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2589b5\"/\"kube-root-ca.crt\"" Oct 30 17:09:45.194930 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:45.194866 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2589b5\"/\"routing-overlay\"" Oct 30 17:09:45.199367 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:45.199336 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7fd7cnbdl"] Oct 30 17:09:45.240870 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:45.240844 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9f936193-e52e-4dc7-a3c0-e1541c8cea11-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-7fd7cnbdl\" (UID: \"9f936193-e52e-4dc7-a3c0-e1541c8cea11\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7fd7cnbdl" Oct 30 17:09:45.241006 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:45.240958 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9f936193-e52e-4dc7-a3c0-e1541c8cea11-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-7fd7cnbdl\" (UID: \"9f936193-e52e-4dc7-a3c0-e1541c8cea11\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7fd7cnbdl" Oct 30 17:09:45.241006 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:45.240987 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f936193-e52e-4dc7-a3c0-e1541c8cea11-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-7fd7cnbdl\" (UID: \"9f936193-e52e-4dc7-a3c0-e1541c8cea11\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7fd7cnbdl" Oct 30 17:09:45.241139 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:45.241041 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwnvw\" (UniqueName: \"kubernetes.io/projected/9f936193-e52e-4dc7-a3c0-e1541c8cea11-kube-api-access-bwnvw\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-7fd7cnbdl\" (UID: \"9f936193-e52e-4dc7-a3c0-e1541c8cea11\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7fd7cnbdl" Oct 30 17:09:45.281367 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:45.281312 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2589b5\"/\"e2e-cfl-9d7fa9-provider-cred\"" Oct 30 17:09:45.322844 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:09:45.322770 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44f0bfa4_b78a_4cad_99e5_53bfc3979a2a.slice/crio-c78099c1f345a39d5a917c39e8d387f9aa6e3c27296f1bacc7769cd80c441997 WatchSource:0}: Error finding container c78099c1f345a39d5a917c39e8d387f9aa6e3c27296f1bacc7769cd80c441997: Status 404 returned error can't find the container with id c78099c1f345a39d5a917c39e8d387f9aa6e3c27296f1bacc7769cd80c441997 Oct 30 17:09:45.324507 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:45.324487 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3820b22-w45fd"] Oct 30 17:09:45.341993 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:45.341962 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9f936193-e52e-4dc7-a3c0-e1541c8cea11-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-7fd7cnbdl\" (UID: \"9f936193-e52e-4dc7-a3c0-e1541c8cea11\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7fd7cnbdl" Oct 30 17:09:45.342200 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:45.342022 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f936193-e52e-4dc7-a3c0-e1541c8cea11-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-7fd7cnbdl\" (UID: \"9f936193-e52e-4dc7-a3c0-e1541c8cea11\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7fd7cnbdl" Oct 30 17:09:45.342200 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:45.342096 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwnvw\" (UniqueName: \"kubernetes.io/projected/9f936193-e52e-4dc7-a3c0-e1541c8cea11-kube-api-access-bwnvw\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-7fd7cnbdl\" (UID: \"9f936193-e52e-4dc7-a3c0-e1541c8cea11\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7fd7cnbdl" Oct 30 17:09:45.342323 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:45.342246 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9f936193-e52e-4dc7-a3c0-e1541c8cea11-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-7fd7cnbdl\" (UID: \"9f936193-e52e-4dc7-a3c0-e1541c8cea11\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7fd7cnbdl" Oct 30 17:09:45.342592 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:45.342572 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9f936193-e52e-4dc7-a3c0-e1541c8cea11-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-7fd7cnbdl\" (UID: \"9f936193-e52e-4dc7-a3c0-e1541c8cea11\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7fd7cnbdl" Oct 30 17:09:45.342925 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:45.342904 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f936193-e52e-4dc7-a3c0-e1541c8cea11-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-7fd7cnbdl\" (UID: \"9f936193-e52e-4dc7-a3c0-e1541c8cea11\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7fd7cnbdl" Oct 30 17:09:45.344976 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:45.344957 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9f936193-e52e-4dc7-a3c0-e1541c8cea11-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-7fd7cnbdl\" (UID: \"9f936193-e52e-4dc7-a3c0-e1541c8cea11\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7fd7cnbdl" Oct 30 17:09:45.366025 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:45.366003 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwnvw\" (UniqueName: \"kubernetes.io/projected/9f936193-e52e-4dc7-a3c0-e1541c8cea11-kube-api-access-bwnvw\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-7fd7cnbdl\" (UID: \"9f936193-e52e-4dc7-a3c0-e1541c8cea11\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7fd7cnbdl" Oct 30 17:09:45.483086 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:45.483057 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7fd7cnbdl" Oct 30 17:09:45.791050 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:45.791016 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3820b22-w45fd" event={"ID":"44f0bfa4-b78a-4cad-99e5-53bfc3979a2a","Type":"ContainerStarted","Data":"9c80f19d37c35de1aed8434920a182db260688710af160d4ddc982514a538c32"} Oct 30 17:09:45.791507 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:45.791069 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3820b22-w45fd" event={"ID":"44f0bfa4-b78a-4cad-99e5-53bfc3979a2a","Type":"ContainerStarted","Data":"c78099c1f345a39d5a917c39e8d387f9aa6e3c27296f1bacc7769cd80c441997"} Oct 30 17:09:45.836932 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:45.836870 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3820b22-w45fd" podStartSLOduration=1.836851258 podStartE2EDuration="1.836851258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:09:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:09:45.831285321 +0000 UTC m=+2813.058460421" watchObservedRunningTime="2026-10-30 17:09:45.836851258 +0000 UTC m=+2813.064026359" Oct 30 17:09:45.875143 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:09:45.875116 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f936193_e52e_4dc7_a3c0_e1541c8cea11.slice/crio-d3800fa2d1da69e2be6137b0a7fd1e32ae33420fa24c56b6936cae7ffe22b16a WatchSource:0}: Error finding container d3800fa2d1da69e2be6137b0a7fd1e32ae33420fa24c56b6936cae7ffe22b16a: Status 404 returned error can't find the container with id d3800fa2d1da69e2be6137b0a7fd1e32ae33420fa24c56b6936cae7ffe22b16a Oct 30 17:09:45.876704 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:45.876676 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7fd7cnbdl"] Oct 30 17:09:46.800814 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:46.800757 4493 generic.go:356] "Generic (PLEG): container finished" podID="44f0bfa4-b78a-4cad-99e5-53bfc3979a2a" containerID="9c80f19d37c35de1aed8434920a182db260688710af160d4ddc982514a538c32" exitCode=0 Oct 30 17:09:46.801288 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:46.800832 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3820b22-w45fd" event={"ID":"44f0bfa4-b78a-4cad-99e5-53bfc3979a2a","Type":"ContainerDied","Data":"9c80f19d37c35de1aed8434920a182db260688710af160d4ddc982514a538c32"} Oct 30 17:09:46.802770 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:46.802737 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7fd7cnbdl" event={"ID":"9f936193-e52e-4dc7-a3c0-e1541c8cea11","Type":"ContainerStarted","Data":"bc284db96f542258e09f15fdd888648f6ccd7e6aa3d4669e3bce3ae078f21ab5"} Oct 30 17:09:46.802925 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:46.802814 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7fd7cnbdl" event={"ID":"9f936193-e52e-4dc7-a3c0-e1541c8cea11","Type":"ContainerStarted","Data":"d3800fa2d1da69e2be6137b0a7fd1e32ae33420fa24c56b6936cae7ffe22b16a"} Oct 30 17:09:46.802925 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:46.802850 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7fd7cnbdl" Oct 30 17:09:46.883659 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:46.883596 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7fd7cnbdl" podStartSLOduration=1.883577667 podStartE2EDuration="1.883577667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:09:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:09:46.882409225 +0000 UTC m=+2814.109584326" watchObservedRunningTime="2026-10-30 17:09:46.883577667 +0000 UTC m=+2814.110752768" Oct 30 17:09:48.007167 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:48.007140 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3820b22-w45fd" Oct 30 17:09:48.091167 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:48.091068 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44f0bfa4-b78a-4cad-99e5-53bfc3979a2a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44f0bfa4-b78a-4cad-99e5-53bfc3979a2a-maas-api-xks-ca\") pod \"44f0bfa4-b78a-4cad-99e5-53bfc3979a2a\" (UID: \"44f0bfa4-b78a-4cad-99e5-53bfc3979a2a\") " Oct 30 17:09:48.091167 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:48.091129 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44f0bfa4-b78a-4cad-99e5-53bfc3979a2a-kube-api-access-wvkvx\" (UniqueName: \"kubernetes.io/projected/44f0bfa4-b78a-4cad-99e5-53bfc3979a2a-kube-api-access-wvkvx\") pod \"44f0bfa4-b78a-4cad-99e5-53bfc3979a2a\" (UID: \"44f0bfa4-b78a-4cad-99e5-53bfc3979a2a\") " Oct 30 17:09:48.091386 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:48.091217 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44f0bfa4-b78a-4cad-99e5-53bfc3979a2a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44f0bfa4-b78a-4cad-99e5-53bfc3979a2a-maas-api-openshift-ca\") pod \"44f0bfa4-b78a-4cad-99e5-53bfc3979a2a\" (UID: \"44f0bfa4-b78a-4cad-99e5-53bfc3979a2a\") " Oct 30 17:09:48.091680 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:48.091645 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44f0bfa4-b78a-4cad-99e5-53bfc3979a2a-maas-api-openshift-ca" pod "44f0bfa4-b78a-4cad-99e5-53bfc3979a2a" (UID: "44f0bfa4-b78a-4cad-99e5-53bfc3979a2a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:09:48.092050 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:48.091927 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44f0bfa4-b78a-4cad-99e5-53bfc3979a2a-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:09:48.093472 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:48.093449 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44f0bfa4-b78a-4cad-99e5-53bfc3979a2a-maas-api-xks-ca" pod "44f0bfa4-b78a-4cad-99e5-53bfc3979a2a" (UID: "44f0bfa4-b78a-4cad-99e5-53bfc3979a2a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:09:48.093981 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:48.093953 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44f0bfa4-b78a-4cad-99e5-53bfc3979a2a-kube-api-access-wvkvx" pod "44f0bfa4-b78a-4cad-99e5-53bfc3979a2a" (UID: "44f0bfa4-b78a-4cad-99e5-53bfc3979a2a"). InnerVolumeSpecName "kube-api-access-wvkvx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:09:48.192831 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:48.192765 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44f0bfa4-b78a-4cad-99e5-53bfc3979a2a-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:09:48.192831 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:48.192829 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvkvx\" (UniqueName: \"kubernetes.io/projected/44f0bfa4-b78a-4cad-99e5-53bfc3979a2a-kube-api-access-wvkvx\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:09:48.819360 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:48.819328 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3820b22-w45fd" Oct 30 17:09:48.819360 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:48.819356 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3820b22-w45fd" event={"ID":"44f0bfa4-b78a-4cad-99e5-53bfc3979a2a","Type":"ContainerDied","Data":"c78099c1f345a39d5a917c39e8d387f9aa6e3c27296f1bacc7769cd80c441997"} Oct 30 17:09:48.819658 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:48.819398 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c78099c1f345a39d5a917c39e8d387f9aa6e3c27296f1bacc7769cd80c441997" Oct 30 17:09:55.962449 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:55.962414 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7fd7cnbdl"] Oct 30 17:09:55.965608 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:09:55.965584 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7fd7cnbdl" Oct 30 17:10:00.452136 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:10:00.451950 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 17:10:00.970808 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:00.970409 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7fd7cnbdl" podUID="9f936193-e52e-4dc7-a3c0-e1541c8cea11" containerName="payload-processing" containerID="cri-o://bc284db96f542258e09f15fdd888648f6ccd7e6aa3d4669e3bce3ae078f21ab5" gracePeriod=25 Oct 30 17:10:01.380444 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:01.380412 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7fd7cnbdl" Oct 30 17:10:01.487849 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:01.487808 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f936193-e52e-4dc7-a3c0-e1541c8cea11-kube-api-access-bwnvw\" (UniqueName: \"kubernetes.io/projected/9f936193-e52e-4dc7-a3c0-e1541c8cea11-kube-api-access-bwnvw\") pod \"9f936193-e52e-4dc7-a3c0-e1541c8cea11\" (UID: \"9f936193-e52e-4dc7-a3c0-e1541c8cea11\") " Oct 30 17:10:01.488377 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:01.488054 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f936193-e52e-4dc7-a3c0-e1541c8cea11-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f936193-e52e-4dc7-a3c0-e1541c8cea11-plugins-config-volume\") pod \"9f936193-e52e-4dc7-a3c0-e1541c8cea11\" (UID: \"9f936193-e52e-4dc7-a3c0-e1541c8cea11\") " Oct 30 17:10:01.488377 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:01.488122 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f936193-e52e-4dc7-a3c0-e1541c8cea11-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9f936193-e52e-4dc7-a3c0-e1541c8cea11-routing-overlay-volume\") pod \"9f936193-e52e-4dc7-a3c0-e1541c8cea11\" (UID: \"9f936193-e52e-4dc7-a3c0-e1541c8cea11\") " Oct 30 17:10:01.488377 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:01.488184 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f936193-e52e-4dc7-a3c0-e1541c8cea11-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9f936193-e52e-4dc7-a3c0-e1541c8cea11-provider-credentials\") pod \"9f936193-e52e-4dc7-a3c0-e1541c8cea11\" (UID: \"9f936193-e52e-4dc7-a3c0-e1541c8cea11\") " Oct 30 17:10:01.489112 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:01.489082 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f936193-e52e-4dc7-a3c0-e1541c8cea11-plugins-config-volume" pod "9f936193-e52e-4dc7-a3c0-e1541c8cea11" (UID: "9f936193-e52e-4dc7-a3c0-e1541c8cea11"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:10:01.489244 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:01.489116 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f936193-e52e-4dc7-a3c0-e1541c8cea11-routing-overlay-volume" pod "9f936193-e52e-4dc7-a3c0-e1541c8cea11" (UID: "9f936193-e52e-4dc7-a3c0-e1541c8cea11"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:10:01.490637 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:01.490610 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f936193-e52e-4dc7-a3c0-e1541c8cea11-kube-api-access-bwnvw" pod "9f936193-e52e-4dc7-a3c0-e1541c8cea11" (UID: "9f936193-e52e-4dc7-a3c0-e1541c8cea11"). InnerVolumeSpecName "kube-api-access-bwnvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:10:01.491624 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:01.491599 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f936193-e52e-4dc7-a3c0-e1541c8cea11-provider-credentials" pod "9f936193-e52e-4dc7-a3c0-e1541c8cea11" (UID: "9f936193-e52e-4dc7-a3c0-e1541c8cea11"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:10:01.589952 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:01.589846 4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f936193-e52e-4dc7-a3c0-e1541c8cea11-plugins-config-volume\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:10:01.589952 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:01.589882 4493 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9f936193-e52e-4dc7-a3c0-e1541c8cea11-routing-overlay-volume\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:10:01.589952 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:01.589904 4493 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9f936193-e52e-4dc7-a3c0-e1541c8cea11-provider-credentials\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:10:01.589952 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:01.589917 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwnvw\" (UniqueName: \"kubernetes.io/projected/9f936193-e52e-4dc7-a3c0-e1541c8cea11-kube-api-access-bwnvw\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:10:02.006510 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:02.006476 4493 generic.go:356] "Generic (PLEG): container finished" podID="9f936193-e52e-4dc7-a3c0-e1541c8cea11" containerID="bc284db96f542258e09f15fdd888648f6ccd7e6aa3d4669e3bce3ae078f21ab5" exitCode=0 Oct 30 17:10:02.006698 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:02.006542 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7fd7cnbdl" Oct 30 17:10:02.006698 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:02.006564 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7fd7cnbdl" event={"ID":"9f936193-e52e-4dc7-a3c0-e1541c8cea11","Type":"ContainerDied","Data":"bc284db96f542258e09f15fdd888648f6ccd7e6aa3d4669e3bce3ae078f21ab5"} Oct 30 17:10:02.006698 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:02.006620 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7fd7cnbdl" event={"ID":"9f936193-e52e-4dc7-a3c0-e1541c8cea11","Type":"ContainerDied","Data":"d3800fa2d1da69e2be6137b0a7fd1e32ae33420fa24c56b6936cae7ffe22b16a"} Oct 30 17:10:02.006698 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:02.006658 4493 scope.go:122] "RemoveContainer" containerID="bc284db96f542258e09f15fdd888648f6ccd7e6aa3d4669e3bce3ae078f21ab5" Oct 30 17:10:02.018823 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:02.018767 4493 scope.go:122] "RemoveContainer" containerID="bc284db96f542258e09f15fdd888648f6ccd7e6aa3d4669e3bce3ae078f21ab5" Oct 30 17:10:02.019209 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:10:02.019181 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc284db96f542258e09f15fdd888648f6ccd7e6aa3d4669e3bce3ae078f21ab5\": container with ID starting with bc284db96f542258e09f15fdd888648f6ccd7e6aa3d4669e3bce3ae078f21ab5 not found: ID does not exist" containerID="bc284db96f542258e09f15fdd888648f6ccd7e6aa3d4669e3bce3ae078f21ab5" Oct 30 17:10:02.019291 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:02.019220 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc284db96f542258e09f15fdd888648f6ccd7e6aa3d4669e3bce3ae078f21ab5"} err="failed to get container status \"bc284db96f542258e09f15fdd888648f6ccd7e6aa3d4669e3bce3ae078f21ab5\": rpc error: code = NotFound desc = could not find container \"bc284db96f542258e09f15fdd888648f6ccd7e6aa3d4669e3bce3ae078f21ab5\": container with ID starting with bc284db96f542258e09f15fdd888648f6ccd7e6aa3d4669e3bce3ae078f21ab5 not found: ID does not exist" Oct 30 17:10:02.061280 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:02.061238 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7fd7cnbdl"] Oct 30 17:10:02.070440 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:02.070406 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7fd7cnbdl"] Oct 30 17:10:02.937180 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:02.937139 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f936193-e52e-4dc7-a3c0-e1541c8cea11" path="/var/lib/kubelet/pods/9f936193-e52e-4dc7-a3c0-e1541c8cea11/volumes" Oct 30 17:10:03.915719 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:03.915688 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70ea3790-kw8mh"] Oct 30 17:10:03.916566 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:03.916544 4493 cpu_manager.go:414] "containerMap: removing container" podUID="44f0bfa4-b78a-4cad-99e5-53bfc3979a2a" containerName="revoke-keys" Oct 30 17:10:03.916689 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:03.916673 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44f0bfa4-b78a-4cad-99e5-53bfc3979a2a" containerName="revoke-keys" Oct 30 17:10:03.916796 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:03.916728 4493 cpu_manager.go:414] "containerMap: removing container" podUID="9f936193-e52e-4dc7-a3c0-e1541c8cea11" containerName="payload-processing" Oct 30 17:10:03.916796 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:03.916738 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f936193-e52e-4dc7-a3c0-e1541c8cea11" containerName="payload-processing" Oct 30 17:10:03.916986 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:03.916971 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="44f0bfa4-b78a-4cad-99e5-53bfc3979a2a" containerName="revoke-keys" Oct 30 17:10:03.917089 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:03.917074 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f936193-e52e-4dc7-a3c0-e1541c8cea11" containerName="payload-processing" Oct 30 17:10:03.924248 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:03.924182 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70ea3790-kw8mh" Oct 30 17:10:03.931096 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:03.931069 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70ea3790-kw8mh"] Oct 30 17:10:03.978827 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:03.978756 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 17:10:03.979317 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:03.978890 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:10:03.979317 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:03.978898 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:10:03.979317 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:03.978967 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:10:04.023515 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:04.023483 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a41054e2-8a2b-4ccf-a336-1fc8aa76f378-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70ea3790-kw8mh\" (UID: \"a41054e2-8a2b-4ccf-a336-1fc8aa76f378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70ea3790-kw8mh" Oct 30 17:10:04.023759 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:04.023738 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-522ql\" (UniqueName: \"kubernetes.io/projected/a41054e2-8a2b-4ccf-a336-1fc8aa76f378-kube-api-access-522ql\") pod \"maas-api-revoke-sub-70ea3790-kw8mh\" (UID: \"a41054e2-8a2b-4ccf-a336-1fc8aa76f378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70ea3790-kw8mh" Oct 30 17:10:04.023874 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:04.023850 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a41054e2-8a2b-4ccf-a336-1fc8aa76f378-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70ea3790-kw8mh\" (UID: \"a41054e2-8a2b-4ccf-a336-1fc8aa76f378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70ea3790-kw8mh" Oct 30 17:10:04.125796 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:04.125746 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-522ql\" (UniqueName: \"kubernetes.io/projected/a41054e2-8a2b-4ccf-a336-1fc8aa76f378-kube-api-access-522ql\") pod \"maas-api-revoke-sub-70ea3790-kw8mh\" (UID: \"a41054e2-8a2b-4ccf-a336-1fc8aa76f378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70ea3790-kw8mh" Oct 30 17:10:04.125993 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:04.125835 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a41054e2-8a2b-4ccf-a336-1fc8aa76f378-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70ea3790-kw8mh\" (UID: \"a41054e2-8a2b-4ccf-a336-1fc8aa76f378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70ea3790-kw8mh" Oct 30 17:10:04.125993 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:04.125899 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a41054e2-8a2b-4ccf-a336-1fc8aa76f378-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70ea3790-kw8mh\" (UID: \"a41054e2-8a2b-4ccf-a336-1fc8aa76f378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70ea3790-kw8mh" Oct 30 17:10:04.126610 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:04.126584 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a41054e2-8a2b-4ccf-a336-1fc8aa76f378-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70ea3790-kw8mh\" (UID: \"a41054e2-8a2b-4ccf-a336-1fc8aa76f378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70ea3790-kw8mh" Oct 30 17:10:04.128461 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:04.128439 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a41054e2-8a2b-4ccf-a336-1fc8aa76f378-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70ea3790-kw8mh\" (UID: \"a41054e2-8a2b-4ccf-a336-1fc8aa76f378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70ea3790-kw8mh" Oct 30 17:10:04.179299 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:04.179210 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-522ql\" (UniqueName: \"kubernetes.io/projected/a41054e2-8a2b-4ccf-a336-1fc8aa76f378-kube-api-access-522ql\") pod \"maas-api-revoke-sub-70ea3790-kw8mh\" (UID: \"a41054e2-8a2b-4ccf-a336-1fc8aa76f378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70ea3790-kw8mh" Oct 30 17:10:04.273835 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:04.273774 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70ea3790-kw8mh" Oct 30 17:10:04.281698 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:04.281672 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f32ebb4-f4cwf"] Oct 30 17:10:04.285857 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:04.285837 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f32ebb4-f4cwf" Oct 30 17:10:04.309446 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:04.308186 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f32ebb4-f4cwf"] Oct 30 17:10:04.432036 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:04.431957 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9f9cccb-f1e1-43e6-861b-eca7d198b1de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f32ebb4-f4cwf\" (UID: \"c9f9cccb-f1e1-43e6-861b-eca7d198b1de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f32ebb4-f4cwf" Oct 30 17:10:04.432036 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:04.432017 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9f9cccb-f1e1-43e6-861b-eca7d198b1de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f32ebb4-f4cwf\" (UID: \"c9f9cccb-f1e1-43e6-861b-eca7d198b1de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f32ebb4-f4cwf" Oct 30 17:10:04.432036 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:04.432037 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl7g5\" (UniqueName: \"kubernetes.io/projected/c9f9cccb-f1e1-43e6-861b-eca7d198b1de-kube-api-access-xl7g5\") pod \"maas-api-revoke-sub-9f32ebb4-f4cwf\" (UID: \"c9f9cccb-f1e1-43e6-861b-eca7d198b1de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f32ebb4-f4cwf" Oct 30 17:10:04.533041 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:04.533011 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9f9cccb-f1e1-43e6-861b-eca7d198b1de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f32ebb4-f4cwf\" (UID: \"c9f9cccb-f1e1-43e6-861b-eca7d198b1de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f32ebb4-f4cwf" Oct 30 17:10:04.533219 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:04.533071 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9f9cccb-f1e1-43e6-861b-eca7d198b1de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f32ebb4-f4cwf\" (UID: \"c9f9cccb-f1e1-43e6-861b-eca7d198b1de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f32ebb4-f4cwf" Oct 30 17:10:04.533219 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:04.533091 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl7g5\" (UniqueName: \"kubernetes.io/projected/c9f9cccb-f1e1-43e6-861b-eca7d198b1de-kube-api-access-xl7g5\") pod \"maas-api-revoke-sub-9f32ebb4-f4cwf\" (UID: \"c9f9cccb-f1e1-43e6-861b-eca7d198b1de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f32ebb4-f4cwf" Oct 30 17:10:04.533944 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:04.533858 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9f9cccb-f1e1-43e6-861b-eca7d198b1de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f32ebb4-f4cwf\" (UID: \"c9f9cccb-f1e1-43e6-861b-eca7d198b1de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f32ebb4-f4cwf" Oct 30 17:10:04.535768 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:04.535746 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9f9cccb-f1e1-43e6-861b-eca7d198b1de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f32ebb4-f4cwf\" (UID: \"c9f9cccb-f1e1-43e6-861b-eca7d198b1de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f32ebb4-f4cwf" Oct 30 17:10:04.557048 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:04.557024 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl7g5\" (UniqueName: \"kubernetes.io/projected/c9f9cccb-f1e1-43e6-861b-eca7d198b1de-kube-api-access-xl7g5\") pod \"maas-api-revoke-sub-9f32ebb4-f4cwf\" (UID: \"c9f9cccb-f1e1-43e6-861b-eca7d198b1de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f32ebb4-f4cwf" Oct 30 17:10:04.600665 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:04.600638 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70ea3790-kw8mh"] Oct 30 17:10:04.600882 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:10:04.600861 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda41054e2_8a2b_4ccf_a336_1fc8aa76f378.slice/crio-1f055d3d2c5cab2d2d00b75fb6dfe740d3b1e288add88694a90ca2f825f47701 WatchSource:0}: Error finding container 1f055d3d2c5cab2d2d00b75fb6dfe740d3b1e288add88694a90ca2f825f47701: Status 404 returned error can't find the container with id 1f055d3d2c5cab2d2d00b75fb6dfe740d3b1e288add88694a90ca2f825f47701 Oct 30 17:10:04.638626 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:04.638604 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f32ebb4-f4cwf" Oct 30 17:10:04.877275 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:04.877245 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fda72a4f-j869l"] Oct 30 17:10:04.882123 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:04.882107 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda72a4f-j869l" Oct 30 17:10:04.903187 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:04.903160 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fda72a4f-j869l"] Oct 30 17:10:04.953656 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:04.953626 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f32ebb4-f4cwf"] Oct 30 17:10:04.954837 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:10:04.954779 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9f9cccb_f1e1_43e6_861b_eca7d198b1de.slice/crio-434dc4e9fdf8feb2188becb75e43dcabf0e7c612ed16ee2d8f1c77360357cda4 WatchSource:0}: Error finding container 434dc4e9fdf8feb2188becb75e43dcabf0e7c612ed16ee2d8f1c77360357cda4: Status 404 returned error can't find the container with id 434dc4e9fdf8feb2188becb75e43dcabf0e7c612ed16ee2d8f1c77360357cda4 Oct 30 17:10:05.034486 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:05.034455 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70ea3790-kw8mh" event={"ID":"a41054e2-8a2b-4ccf-a336-1fc8aa76f378","Type":"ContainerStarted","Data":"6da98b64478f96b4f1ff533819db0c1919ec5a1b1be6999abb2fe4c9e43e591f"} Oct 30 17:10:05.034873 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:05.034513 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70ea3790-kw8mh" event={"ID":"a41054e2-8a2b-4ccf-a336-1fc8aa76f378","Type":"ContainerStarted","Data":"1f055d3d2c5cab2d2d00b75fb6dfe740d3b1e288add88694a90ca2f825f47701"} Oct 30 17:10:05.035995 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:05.035970 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f32ebb4-f4cwf" event={"ID":"c9f9cccb-f1e1-43e6-861b-eca7d198b1de","Type":"ContainerStarted","Data":"434dc4e9fdf8feb2188becb75e43dcabf0e7c612ed16ee2d8f1c77360357cda4"} Oct 30 17:10:05.040655 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:05.040632 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b46b93f9-fb79-4f47-b25d-549cd661e9a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fda72a4f-j869l\" (UID: \"b46b93f9-fb79-4f47-b25d-549cd661e9a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda72a4f-j869l" Oct 30 17:10:05.040745 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:05.040709 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b46b93f9-fb79-4f47-b25d-549cd661e9a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fda72a4f-j869l\" (UID: \"b46b93f9-fb79-4f47-b25d-549cd661e9a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda72a4f-j869l" Oct 30 17:10:05.040824 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:05.040767 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cc7p9\" (UniqueName: \"kubernetes.io/projected/b46b93f9-fb79-4f47-b25d-549cd661e9a5-kube-api-access-cc7p9\") pod \"maas-api-revoke-sub-fda72a4f-j869l\" (UID: \"b46b93f9-fb79-4f47-b25d-549cd661e9a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda72a4f-j869l" Oct 30 17:10:05.076224 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:05.076181 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70ea3790-kw8mh" podStartSLOduration=2.076168561 podStartE2EDuration="2.076168561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:10:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:10:05.073774222 +0000 UTC m=+2832.300949324" watchObservedRunningTime="2026-10-30 17:10:05.076168561 +0000 UTC m=+2832.303343662" Oct 30 17:10:05.142696 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:05.142668 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b46b93f9-fb79-4f47-b25d-549cd661e9a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fda72a4f-j869l\" (UID: \"b46b93f9-fb79-4f47-b25d-549cd661e9a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda72a4f-j869l" Oct 30 17:10:05.142850 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:05.142715 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cc7p9\" (UniqueName: \"kubernetes.io/projected/b46b93f9-fb79-4f47-b25d-549cd661e9a5-kube-api-access-cc7p9\") pod \"maas-api-revoke-sub-fda72a4f-j869l\" (UID: \"b46b93f9-fb79-4f47-b25d-549cd661e9a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda72a4f-j869l" Oct 30 17:10:05.142959 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:05.142940 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b46b93f9-fb79-4f47-b25d-549cd661e9a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fda72a4f-j869l\" (UID: \"b46b93f9-fb79-4f47-b25d-549cd661e9a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda72a4f-j869l" Oct 30 17:10:05.143586 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:05.143533 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b46b93f9-fb79-4f47-b25d-549cd661e9a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fda72a4f-j869l\" (UID: \"b46b93f9-fb79-4f47-b25d-549cd661e9a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda72a4f-j869l" Oct 30 17:10:05.145248 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:05.145226 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b46b93f9-fb79-4f47-b25d-549cd661e9a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fda72a4f-j869l\" (UID: \"b46b93f9-fb79-4f47-b25d-549cd661e9a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda72a4f-j869l" Oct 30 17:10:05.165550 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:05.165528 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc7p9\" (UniqueName: \"kubernetes.io/projected/b46b93f9-fb79-4f47-b25d-549cd661e9a5-kube-api-access-cc7p9\") pod \"maas-api-revoke-sub-fda72a4f-j869l\" (UID: \"b46b93f9-fb79-4f47-b25d-549cd661e9a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda72a4f-j869l" Oct 30 17:10:05.194918 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:05.194894 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda72a4f-j869l" Oct 30 17:10:05.505718 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:10:05.505687 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb46b93f9_fb79_4f47_b25d_549cd661e9a5.slice/crio-705b491837709f8baa666e58104a4b00b2983026d699628fb96c66a85996a051 WatchSource:0}: Error finding container 705b491837709f8baa666e58104a4b00b2983026d699628fb96c66a85996a051: Status 404 returned error can't find the container with id 705b491837709f8baa666e58104a4b00b2983026d699628fb96c66a85996a051 Oct 30 17:10:05.508157 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:05.508126 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fda72a4f-j869l"] Oct 30 17:10:06.045031 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:06.045003 4493 generic.go:356] "Generic (PLEG): container finished" podID="a41054e2-8a2b-4ccf-a336-1fc8aa76f378" containerID="6da98b64478f96b4f1ff533819db0c1919ec5a1b1be6999abb2fe4c9e43e591f" exitCode=0 Oct 30 17:10:06.045414 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:06.045088 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70ea3790-kw8mh" event={"ID":"a41054e2-8a2b-4ccf-a336-1fc8aa76f378","Type":"ContainerDied","Data":"6da98b64478f96b4f1ff533819db0c1919ec5a1b1be6999abb2fe4c9e43e591f"} Oct 30 17:10:06.047007 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:06.046970 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda72a4f-j869l" event={"ID":"b46b93f9-fb79-4f47-b25d-549cd661e9a5","Type":"ContainerStarted","Data":"34b85aaf2ead0ef148d9cd134fb8803ef86053542a7c6fcdbc0d14ddfaf19c16"} Oct 30 17:10:06.047126 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:06.047031 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda72a4f-j869l" event={"ID":"b46b93f9-fb79-4f47-b25d-549cd661e9a5","Type":"ContainerStarted","Data":"705b491837709f8baa666e58104a4b00b2983026d699628fb96c66a85996a051"} Oct 30 17:10:06.060537 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:06.060509 4493 generic.go:356] "Generic (PLEG): container finished" podID="c9f9cccb-f1e1-43e6-861b-eca7d198b1de" containerID="21ef935358c22b5398eabf61afce23fbca38693f3aa0aee23c627cda24996036" exitCode=0 Oct 30 17:10:06.060724 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:06.060707 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f32ebb4-f4cwf" event={"ID":"c9f9cccb-f1e1-43e6-861b-eca7d198b1de","Type":"ContainerDied","Data":"21ef935358c22b5398eabf61afce23fbca38693f3aa0aee23c627cda24996036"} Oct 30 17:10:06.171126 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:06.171032 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda72a4f-j869l" podStartSLOduration=2.171013645 podStartE2EDuration="2.171013645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:10:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:10:06.169338816 +0000 UTC m=+2833.396513916" watchObservedRunningTime="2026-10-30 17:10:06.171013645 +0000 UTC m=+2833.398188745" Oct 30 17:10:07.071130 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:07.071094 4493 generic.go:356] "Generic (PLEG): container finished" podID="b46b93f9-fb79-4f47-b25d-549cd661e9a5" containerID="34b85aaf2ead0ef148d9cd134fb8803ef86053542a7c6fcdbc0d14ddfaf19c16" exitCode=0 Oct 30 17:10:07.071627 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:07.071182 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda72a4f-j869l" event={"ID":"b46b93f9-fb79-4f47-b25d-549cd661e9a5","Type":"ContainerDied","Data":"34b85aaf2ead0ef148d9cd134fb8803ef86053542a7c6fcdbc0d14ddfaf19c16"} Oct 30 17:10:07.292245 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:07.292216 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70ea3790-kw8mh" Oct 30 17:10:07.329756 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:07.329701 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f32ebb4-f4cwf" Oct 30 17:10:07.380241 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:07.380204 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a41054e2-8a2b-4ccf-a336-1fc8aa76f378-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a41054e2-8a2b-4ccf-a336-1fc8aa76f378-maas-api-xks-ca\") pod \"a41054e2-8a2b-4ccf-a336-1fc8aa76f378\" (UID: \"a41054e2-8a2b-4ccf-a336-1fc8aa76f378\") " Oct 30 17:10:07.380482 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:07.380330 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a41054e2-8a2b-4ccf-a336-1fc8aa76f378-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a41054e2-8a2b-4ccf-a336-1fc8aa76f378-maas-api-openshift-ca\") pod \"a41054e2-8a2b-4ccf-a336-1fc8aa76f378\" (UID: \"a41054e2-8a2b-4ccf-a336-1fc8aa76f378\") " Oct 30 17:10:07.380482 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:07.380364 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a41054e2-8a2b-4ccf-a336-1fc8aa76f378-kube-api-access-522ql\" (UniqueName: \"kubernetes.io/projected/a41054e2-8a2b-4ccf-a336-1fc8aa76f378-kube-api-access-522ql\") pod \"a41054e2-8a2b-4ccf-a336-1fc8aa76f378\" (UID: \"a41054e2-8a2b-4ccf-a336-1fc8aa76f378\") " Oct 30 17:10:07.381530 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:07.381503 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a41054e2-8a2b-4ccf-a336-1fc8aa76f378-maas-api-openshift-ca" pod "a41054e2-8a2b-4ccf-a336-1fc8aa76f378" (UID: "a41054e2-8a2b-4ccf-a336-1fc8aa76f378"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:10:07.387868 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:07.385075 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a41054e2-8a2b-4ccf-a336-1fc8aa76f378-maas-api-xks-ca" pod "a41054e2-8a2b-4ccf-a336-1fc8aa76f378" (UID: "a41054e2-8a2b-4ccf-a336-1fc8aa76f378"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:10:07.395939 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:07.395915 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a41054e2-8a2b-4ccf-a336-1fc8aa76f378-kube-api-access-522ql" pod "a41054e2-8a2b-4ccf-a336-1fc8aa76f378" (UID: "a41054e2-8a2b-4ccf-a336-1fc8aa76f378"). InnerVolumeSpecName "kube-api-access-522ql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:10:07.482052 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:07.482023 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9f9cccb-f1e1-43e6-861b-eca7d198b1de-kube-api-access-xl7g5\" (UniqueName: \"kubernetes.io/projected/c9f9cccb-f1e1-43e6-861b-eca7d198b1de-kube-api-access-xl7g5\") pod \"c9f9cccb-f1e1-43e6-861b-eca7d198b1de\" (UID: \"c9f9cccb-f1e1-43e6-861b-eca7d198b1de\") " Oct 30 17:10:07.482184 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:07.482060 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9f9cccb-f1e1-43e6-861b-eca7d198b1de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9f9cccb-f1e1-43e6-861b-eca7d198b1de-maas-api-xks-ca\") pod \"c9f9cccb-f1e1-43e6-861b-eca7d198b1de\" (UID: \"c9f9cccb-f1e1-43e6-861b-eca7d198b1de\") " Oct 30 17:10:07.482184 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:07.482175 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9f9cccb-f1e1-43e6-861b-eca7d198b1de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9f9cccb-f1e1-43e6-861b-eca7d198b1de-maas-api-openshift-ca\") pod \"c9f9cccb-f1e1-43e6-861b-eca7d198b1de\" (UID: \"c9f9cccb-f1e1-43e6-861b-eca7d198b1de\") " Oct 30 17:10:07.482492 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:07.482466 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9f9cccb-f1e1-43e6-861b-eca7d198b1de-maas-api-openshift-ca" pod "c9f9cccb-f1e1-43e6-861b-eca7d198b1de" (UID: "c9f9cccb-f1e1-43e6-861b-eca7d198b1de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:10:07.482754 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:07.482737 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a41054e2-8a2b-4ccf-a336-1fc8aa76f378-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:10:07.482851 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:07.482760 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-522ql\" (UniqueName: \"kubernetes.io/projected/a41054e2-8a2b-4ccf-a336-1fc8aa76f378-kube-api-access-522ql\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:10:07.482851 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:07.482774 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9f9cccb-f1e1-43e6-861b-eca7d198b1de-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:10:07.482851 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:07.482806 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a41054e2-8a2b-4ccf-a336-1fc8aa76f378-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:10:07.484303 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:07.484282 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9f9cccb-f1e1-43e6-861b-eca7d198b1de-maas-api-xks-ca" pod "c9f9cccb-f1e1-43e6-861b-eca7d198b1de" (UID: "c9f9cccb-f1e1-43e6-861b-eca7d198b1de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:10:07.484395 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:07.484377 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9f9cccb-f1e1-43e6-861b-eca7d198b1de-kube-api-access-xl7g5" pod "c9f9cccb-f1e1-43e6-861b-eca7d198b1de" (UID: "c9f9cccb-f1e1-43e6-861b-eca7d198b1de"). InnerVolumeSpecName "kube-api-access-xl7g5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:10:07.584340 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:07.584263 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xl7g5\" (UniqueName: \"kubernetes.io/projected/c9f9cccb-f1e1-43e6-861b-eca7d198b1de-kube-api-access-xl7g5\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:10:07.584340 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:07.584294 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9f9cccb-f1e1-43e6-861b-eca7d198b1de-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:10:08.083612 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:08.083564 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f32ebb4-f4cwf" Oct 30 17:10:08.084120 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:08.083633 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f32ebb4-f4cwf" event={"ID":"c9f9cccb-f1e1-43e6-861b-eca7d198b1de","Type":"ContainerDied","Data":"434dc4e9fdf8feb2188becb75e43dcabf0e7c612ed16ee2d8f1c77360357cda4"} Oct 30 17:10:08.084120 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:08.083683 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="434dc4e9fdf8feb2188becb75e43dcabf0e7c612ed16ee2d8f1c77360357cda4" Oct 30 17:10:08.095998 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:08.093461 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70ea3790-kw8mh" Oct 30 17:10:08.100491 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:08.097978 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70ea3790-kw8mh" event={"ID":"a41054e2-8a2b-4ccf-a336-1fc8aa76f378","Type":"ContainerDied","Data":"1f055d3d2c5cab2d2d00b75fb6dfe740d3b1e288add88694a90ca2f825f47701"} Oct 30 17:10:08.100491 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:08.098023 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f055d3d2c5cab2d2d00b75fb6dfe740d3b1e288add88694a90ca2f825f47701" Oct 30 17:10:08.270823 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:08.270797 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda72a4f-j869l" Oct 30 17:10:08.400430 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:08.400397 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b46b93f9-fb79-4f47-b25d-549cd661e9a5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b46b93f9-fb79-4f47-b25d-549cd661e9a5-maas-api-xks-ca\") pod \"b46b93f9-fb79-4f47-b25d-549cd661e9a5\" (UID: \"b46b93f9-fb79-4f47-b25d-549cd661e9a5\") " Oct 30 17:10:08.400603 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:08.400453 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b46b93f9-fb79-4f47-b25d-549cd661e9a5-kube-api-access-cc7p9\" (UniqueName: \"kubernetes.io/projected/b46b93f9-fb79-4f47-b25d-549cd661e9a5-kube-api-access-cc7p9\") pod \"b46b93f9-fb79-4f47-b25d-549cd661e9a5\" (UID: \"b46b93f9-fb79-4f47-b25d-549cd661e9a5\") " Oct 30 17:10:08.401049 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:08.400735 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b46b93f9-fb79-4f47-b25d-549cd661e9a5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b46b93f9-fb79-4f47-b25d-549cd661e9a5-maas-api-openshift-ca\") pod \"b46b93f9-fb79-4f47-b25d-549cd661e9a5\" (UID: \"b46b93f9-fb79-4f47-b25d-549cd661e9a5\") " Oct 30 17:10:08.401269 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:08.401128 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b46b93f9-fb79-4f47-b25d-549cd661e9a5-maas-api-openshift-ca" pod "b46b93f9-fb79-4f47-b25d-549cd661e9a5" (UID: "b46b93f9-fb79-4f47-b25d-549cd661e9a5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:10:08.402848 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:08.402824 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b46b93f9-fb79-4f47-b25d-549cd661e9a5-maas-api-xks-ca" pod "b46b93f9-fb79-4f47-b25d-549cd661e9a5" (UID: "b46b93f9-fb79-4f47-b25d-549cd661e9a5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:10:08.402956 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:08.402825 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b46b93f9-fb79-4f47-b25d-549cd661e9a5-kube-api-access-cc7p9" pod "b46b93f9-fb79-4f47-b25d-549cd661e9a5" (UID: "b46b93f9-fb79-4f47-b25d-549cd661e9a5"). InnerVolumeSpecName "kube-api-access-cc7p9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:10:08.503035 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:08.503010 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b46b93f9-fb79-4f47-b25d-549cd661e9a5-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:10:08.503035 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:08.503033 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b46b93f9-fb79-4f47-b25d-549cd661e9a5-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:10:08.503202 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:08.503042 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cc7p9\" (UniqueName: \"kubernetes.io/projected/b46b93f9-fb79-4f47-b25d-549cd661e9a5-kube-api-access-cc7p9\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:10:08.897715 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:08.897687 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70d329be-27kfd"] Oct 30 17:10:08.898273 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:08.898258 4493 cpu_manager.go:414] "containerMap: removing container" podUID="c9f9cccb-f1e1-43e6-861b-eca7d198b1de" containerName="revoke-keys" Oct 30 17:10:08.898325 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:08.898274 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9f9cccb-f1e1-43e6-861b-eca7d198b1de" containerName="revoke-keys" Oct 30 17:10:08.898325 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:08.898288 4493 cpu_manager.go:414] "containerMap: removing container" podUID="b46b93f9-fb79-4f47-b25d-549cd661e9a5" containerName="revoke-keys" Oct 30 17:10:08.898325 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:08.898293 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b46b93f9-fb79-4f47-b25d-549cd661e9a5" containerName="revoke-keys" Oct 30 17:10:08.898325 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:08.898301 4493 cpu_manager.go:414] "containerMap: removing container" podUID="a41054e2-8a2b-4ccf-a336-1fc8aa76f378" containerName="revoke-keys" Oct 30 17:10:08.898325 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:08.898306 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a41054e2-8a2b-4ccf-a336-1fc8aa76f378" containerName="revoke-keys" Oct 30 17:10:08.898487 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:08.898405 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9f9cccb-f1e1-43e6-861b-eca7d198b1de" containerName="revoke-keys" Oct 30 17:10:08.898487 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:08.898414 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="b46b93f9-fb79-4f47-b25d-549cd661e9a5" containerName="revoke-keys" Oct 30 17:10:08.898487 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:08.898421 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="a41054e2-8a2b-4ccf-a336-1fc8aa76f378" containerName="revoke-keys" Oct 30 17:10:08.902009 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:08.901994 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70d329be-27kfd" Oct 30 17:10:08.907761 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:08.907729 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cead726f-c70c-4c0a-acb0-c3381f720710-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70d329be-27kfd\" (UID: \"cead726f-c70c-4c0a-acb0-c3381f720710\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70d329be-27kfd" Oct 30 17:10:08.907946 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:08.907814 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cead726f-c70c-4c0a-acb0-c3381f720710-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70d329be-27kfd\" (UID: \"cead726f-c70c-4c0a-acb0-c3381f720710\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70d329be-27kfd" Oct 30 17:10:08.908033 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:08.907985 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6twzk\" (UniqueName: \"kubernetes.io/projected/cead726f-c70c-4c0a-acb0-c3381f720710-kube-api-access-6twzk\") pod \"maas-api-revoke-sub-70d329be-27kfd\" (UID: \"cead726f-c70c-4c0a-acb0-c3381f720710\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70d329be-27kfd" Oct 30 17:10:08.943589 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:08.943542 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70d329be-27kfd"] Oct 30 17:10:09.009320 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:09.009290 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cead726f-c70c-4c0a-acb0-c3381f720710-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70d329be-27kfd\" (UID: \"cead726f-c70c-4c0a-acb0-c3381f720710\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70d329be-27kfd" Oct 30 17:10:09.009487 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:09.009344 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cead726f-c70c-4c0a-acb0-c3381f720710-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70d329be-27kfd\" (UID: \"cead726f-c70c-4c0a-acb0-c3381f720710\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70d329be-27kfd" Oct 30 17:10:09.009487 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:09.009438 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6twzk\" (UniqueName: \"kubernetes.io/projected/cead726f-c70c-4c0a-acb0-c3381f720710-kube-api-access-6twzk\") pod \"maas-api-revoke-sub-70d329be-27kfd\" (UID: \"cead726f-c70c-4c0a-acb0-c3381f720710\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70d329be-27kfd" Oct 30 17:10:09.010087 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:09.010064 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cead726f-c70c-4c0a-acb0-c3381f720710-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70d329be-27kfd\" (UID: \"cead726f-c70c-4c0a-acb0-c3381f720710\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70d329be-27kfd" Oct 30 17:10:09.011919 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:09.011895 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cead726f-c70c-4c0a-acb0-c3381f720710-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70d329be-27kfd\" (UID: \"cead726f-c70c-4c0a-acb0-c3381f720710\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70d329be-27kfd" Oct 30 17:10:09.032837 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:09.032814 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6twzk\" (UniqueName: \"kubernetes.io/projected/cead726f-c70c-4c0a-acb0-c3381f720710-kube-api-access-6twzk\") pod \"maas-api-revoke-sub-70d329be-27kfd\" (UID: \"cead726f-c70c-4c0a-acb0-c3381f720710\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70d329be-27kfd" Oct 30 17:10:09.102154 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:09.102126 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda72a4f-j869l" event={"ID":"b46b93f9-fb79-4f47-b25d-549cd661e9a5","Type":"ContainerDied","Data":"705b491837709f8baa666e58104a4b00b2983026d699628fb96c66a85996a051"} Oct 30 17:10:09.102497 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:09.102172 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="705b491837709f8baa666e58104a4b00b2983026d699628fb96c66a85996a051" Oct 30 17:10:09.102497 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:09.102184 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda72a4f-j869l" Oct 30 17:10:09.219606 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:09.219530 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70d329be-27kfd" Oct 30 17:10:09.528770 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:09.528736 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70d329be-27kfd"] Oct 30 17:10:09.529133 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:10:09.529087 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcead726f_c70c_4c0a_acb0_c3381f720710.slice/crio-2a16ae64e709a8374c6044ed8ecdd948fc6f5b01f3261477ed96672ba3ae6fef WatchSource:0}: Error finding container 2a16ae64e709a8374c6044ed8ecdd948fc6f5b01f3261477ed96672ba3ae6fef: Status 404 returned error can't find the container with id 2a16ae64e709a8374c6044ed8ecdd948fc6f5b01f3261477ed96672ba3ae6fef Oct 30 17:10:10.113307 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:10.113268 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70d329be-27kfd" event={"ID":"cead726f-c70c-4c0a-acb0-c3381f720710","Type":"ContainerStarted","Data":"eaa2c70a4937e82a0fe5b38d21f84f8875dd56c24b73759954205343d6e6664c"} Oct 30 17:10:10.113756 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:10.113337 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70d329be-27kfd" event={"ID":"cead726f-c70c-4c0a-acb0-c3381f720710","Type":"ContainerStarted","Data":"2a16ae64e709a8374c6044ed8ecdd948fc6f5b01f3261477ed96672ba3ae6fef"} Oct 30 17:10:10.156554 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:10.156502 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70d329be-27kfd" podStartSLOduration=2.156487302 podStartE2EDuration="2.156487302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:10:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:10:10.154113629 +0000 UTC m=+2837.381288730" watchObservedRunningTime="2026-10-30 17:10:10.156487302 +0000 UTC m=+2837.383662398" Oct 30 17:10:11.128441 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:11.128404 4493 generic.go:356] "Generic (PLEG): container finished" podID="cead726f-c70c-4c0a-acb0-c3381f720710" containerID="eaa2c70a4937e82a0fe5b38d21f84f8875dd56c24b73759954205343d6e6664c" exitCode=0 Oct 30 17:10:11.128941 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:11.128463 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70d329be-27kfd" event={"ID":"cead726f-c70c-4c0a-acb0-c3381f720710","Type":"ContainerDied","Data":"eaa2c70a4937e82a0fe5b38d21f84f8875dd56c24b73759954205343d6e6664c"} Oct 30 17:10:12.296248 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:12.296223 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70d329be-27kfd" Oct 30 17:10:12.357302 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:12.357268 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cead726f-c70c-4c0a-acb0-c3381f720710-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cead726f-c70c-4c0a-acb0-c3381f720710-maas-api-xks-ca\") pod \"cead726f-c70c-4c0a-acb0-c3381f720710\" (UID: \"cead726f-c70c-4c0a-acb0-c3381f720710\") " Oct 30 17:10:12.357466 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:12.357397 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cead726f-c70c-4c0a-acb0-c3381f720710-kube-api-access-6twzk\" (UniqueName: \"kubernetes.io/projected/cead726f-c70c-4c0a-acb0-c3381f720710-kube-api-access-6twzk\") pod \"cead726f-c70c-4c0a-acb0-c3381f720710\" (UID: \"cead726f-c70c-4c0a-acb0-c3381f720710\") " Oct 30 17:10:12.357530 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:12.357483 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cead726f-c70c-4c0a-acb0-c3381f720710-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cead726f-c70c-4c0a-acb0-c3381f720710-maas-api-openshift-ca\") pod \"cead726f-c70c-4c0a-acb0-c3381f720710\" (UID: \"cead726f-c70c-4c0a-acb0-c3381f720710\") " Oct 30 17:10:12.357931 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:12.357901 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cead726f-c70c-4c0a-acb0-c3381f720710-maas-api-openshift-ca" pod "cead726f-c70c-4c0a-acb0-c3381f720710" (UID: "cead726f-c70c-4c0a-acb0-c3381f720710"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:10:12.359662 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:12.359629 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cead726f-c70c-4c0a-acb0-c3381f720710-maas-api-xks-ca" pod "cead726f-c70c-4c0a-acb0-c3381f720710" (UID: "cead726f-c70c-4c0a-acb0-c3381f720710"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:10:12.359823 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:12.359679 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cead726f-c70c-4c0a-acb0-c3381f720710-kube-api-access-6twzk" pod "cead726f-c70c-4c0a-acb0-c3381f720710" (UID: "cead726f-c70c-4c0a-acb0-c3381f720710"). InnerVolumeSpecName "kube-api-access-6twzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:10:12.459020 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:12.458986 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cead726f-c70c-4c0a-acb0-c3381f720710-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:10:12.459020 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:12.459018 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6twzk\" (UniqueName: \"kubernetes.io/projected/cead726f-c70c-4c0a-acb0-c3381f720710-kube-api-access-6twzk\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:10:12.459229 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:12.459031 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cead726f-c70c-4c0a-acb0-c3381f720710-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:10:13.149599 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:13.149566 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70d329be-27kfd" Oct 30 17:10:13.149599 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:13.149578 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70d329be-27kfd" event={"ID":"cead726f-c70c-4c0a-acb0-c3381f720710","Type":"ContainerDied","Data":"2a16ae64e709a8374c6044ed8ecdd948fc6f5b01f3261477ed96672ba3ae6fef"} Oct 30 17:10:13.149839 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:13.149621 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a16ae64e709a8374c6044ed8ecdd948fc6f5b01f3261477ed96672ba3ae6fef" Oct 30 17:10:14.554956 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:14.554920 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c03254f-9rlgl"] Oct 30 17:10:14.555508 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:14.555469 4493 cpu_manager.go:414] "containerMap: removing container" podUID="cead726f-c70c-4c0a-acb0-c3381f720710" containerName="revoke-keys" Oct 30 17:10:14.555550 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:14.555511 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cead726f-c70c-4c0a-acb0-c3381f720710" containerName="revoke-keys" Oct 30 17:10:14.555637 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:14.555628 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="cead726f-c70c-4c0a-acb0-c3381f720710" containerName="revoke-keys" Oct 30 17:10:14.559006 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:14.558989 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c03254f-9rlgl" Oct 30 17:10:14.584391 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:14.584360 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c03254f-9rlgl"] Oct 30 17:10:14.588106 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:14.588078 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:10:14.588220 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:14.588165 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:10:14.588917 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:14.588894 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:10:14.589044 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:14.588995 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 17:10:14.606287 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:14.606256 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwgnq\" (UniqueName: \"kubernetes.io/projected/8abd1354-46f4-40cc-b5cc-c86321cfb197-kube-api-access-pwgnq\") pod \"maas-api-revoke-sub-5c03254f-9rlgl\" (UID: \"8abd1354-46f4-40cc-b5cc-c86321cfb197\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c03254f-9rlgl" Oct 30 17:10:14.606408 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:14.606308 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8abd1354-46f4-40cc-b5cc-c86321cfb197-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c03254f-9rlgl\" (UID: \"8abd1354-46f4-40cc-b5cc-c86321cfb197\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c03254f-9rlgl" Oct 30 17:10:14.606511 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:14.606487 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8abd1354-46f4-40cc-b5cc-c86321cfb197-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c03254f-9rlgl\" (UID: \"8abd1354-46f4-40cc-b5cc-c86321cfb197\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c03254f-9rlgl" Oct 30 17:10:14.708182 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:14.708131 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8abd1354-46f4-40cc-b5cc-c86321cfb197-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c03254f-9rlgl\" (UID: \"8abd1354-46f4-40cc-b5cc-c86321cfb197\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c03254f-9rlgl" Oct 30 17:10:14.708367 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:14.708281 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8abd1354-46f4-40cc-b5cc-c86321cfb197-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c03254f-9rlgl\" (UID: \"8abd1354-46f4-40cc-b5cc-c86321cfb197\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c03254f-9rlgl" Oct 30 17:10:14.708503 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:14.708442 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwgnq\" (UniqueName: \"kubernetes.io/projected/8abd1354-46f4-40cc-b5cc-c86321cfb197-kube-api-access-pwgnq\") pod \"maas-api-revoke-sub-5c03254f-9rlgl\" (UID: \"8abd1354-46f4-40cc-b5cc-c86321cfb197\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c03254f-9rlgl" Oct 30 17:10:14.710401 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:14.710362 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8abd1354-46f4-40cc-b5cc-c86321cfb197-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c03254f-9rlgl\" (UID: \"8abd1354-46f4-40cc-b5cc-c86321cfb197\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c03254f-9rlgl" Oct 30 17:10:14.724187 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:14.724156 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8abd1354-46f4-40cc-b5cc-c86321cfb197-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c03254f-9rlgl\" (UID: \"8abd1354-46f4-40cc-b5cc-c86321cfb197\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c03254f-9rlgl" Oct 30 17:10:14.732440 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:14.732391 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwgnq\" (UniqueName: \"kubernetes.io/projected/8abd1354-46f4-40cc-b5cc-c86321cfb197-kube-api-access-pwgnq\") pod \"maas-api-revoke-sub-5c03254f-9rlgl\" (UID: \"8abd1354-46f4-40cc-b5cc-c86321cfb197\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c03254f-9rlgl" Oct 30 17:10:14.872970 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:14.872891 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c03254f-9rlgl" Oct 30 17:10:15.335811 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:15.335579 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c03254f-9rlgl"] Oct 30 17:10:15.349654 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:10:15.349621 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8abd1354_46f4_40cc_b5cc_c86321cfb197.slice/crio-3e0957b509065a655037ef5d449e9c45162d3322297b6978b41cf996f95e1aa5 WatchSource:0}: Error finding container 3e0957b509065a655037ef5d449e9c45162d3322297b6978b41cf996f95e1aa5: Status 404 returned error can't find the container with id 3e0957b509065a655037ef5d449e9c45162d3322297b6978b41cf996f95e1aa5 Oct 30 17:10:15.794871 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:15.794840 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82d4a543-x4slv"] Oct 30 17:10:15.799081 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:15.799063 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d4a543-x4slv" Oct 30 17:10:15.822276 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:15.822250 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82d4a543-x4slv"] Oct 30 17:10:15.847390 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:15.847358 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4075789-9024-4cb3-bf1b-fd540609361f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82d4a543-x4slv\" (UID: \"f4075789-9024-4cb3-bf1b-fd540609361f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d4a543-x4slv" Oct 30 17:10:15.847540 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:15.847417 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4075789-9024-4cb3-bf1b-fd540609361f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82d4a543-x4slv\" (UID: \"f4075789-9024-4cb3-bf1b-fd540609361f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d4a543-x4slv" Oct 30 17:10:15.847606 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:15.847585 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5n7s\" (UniqueName: \"kubernetes.io/projected/f4075789-9024-4cb3-bf1b-fd540609361f-kube-api-access-c5n7s\") pod \"maas-api-revoke-sub-82d4a543-x4slv\" (UID: \"f4075789-9024-4cb3-bf1b-fd540609361f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d4a543-x4slv" Oct 30 17:10:15.948433 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:15.948394 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4075789-9024-4cb3-bf1b-fd540609361f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82d4a543-x4slv\" (UID: \"f4075789-9024-4cb3-bf1b-fd540609361f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d4a543-x4slv" Oct 30 17:10:15.948639 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:15.948464 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4075789-9024-4cb3-bf1b-fd540609361f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82d4a543-x4slv\" (UID: \"f4075789-9024-4cb3-bf1b-fd540609361f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d4a543-x4slv" Oct 30 17:10:15.948639 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:15.948599 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5n7s\" (UniqueName: \"kubernetes.io/projected/f4075789-9024-4cb3-bf1b-fd540609361f-kube-api-access-c5n7s\") pod \"maas-api-revoke-sub-82d4a543-x4slv\" (UID: \"f4075789-9024-4cb3-bf1b-fd540609361f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d4a543-x4slv" Oct 30 17:10:15.949214 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:15.949191 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4075789-9024-4cb3-bf1b-fd540609361f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82d4a543-x4slv\" (UID: \"f4075789-9024-4cb3-bf1b-fd540609361f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d4a543-x4slv" Oct 30 17:10:15.950998 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:15.950972 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4075789-9024-4cb3-bf1b-fd540609361f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82d4a543-x4slv\" (UID: \"f4075789-9024-4cb3-bf1b-fd540609361f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d4a543-x4slv" Oct 30 17:10:15.971736 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:15.971715 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5n7s\" (UniqueName: \"kubernetes.io/projected/f4075789-9024-4cb3-bf1b-fd540609361f-kube-api-access-c5n7s\") pod \"maas-api-revoke-sub-82d4a543-x4slv\" (UID: \"f4075789-9024-4cb3-bf1b-fd540609361f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d4a543-x4slv" Oct 30 17:10:16.114826 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:16.113429 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d4a543-x4slv" Oct 30 17:10:16.178302 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:16.178263 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c03254f-9rlgl" event={"ID":"8abd1354-46f4-40cc-b5cc-c86321cfb197","Type":"ContainerStarted","Data":"2c984d23330fc1b8a018652cba6542db6099cb0a4b8a62216991aa2a3752d601"} Oct 30 17:10:16.178507 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:16.178327 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c03254f-9rlgl" event={"ID":"8abd1354-46f4-40cc-b5cc-c86321cfb197","Type":"ContainerStarted","Data":"3e0957b509065a655037ef5d449e9c45162d3322297b6978b41cf996f95e1aa5"} Oct 30 17:10:16.223369 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:16.223263 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c03254f-9rlgl" podStartSLOduration=2.223247943 podStartE2EDuration="2.223247943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:10:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:10:16.218913894 +0000 UTC m=+2843.446088996" watchObservedRunningTime="2026-10-30 17:10:16.223247943 +0000 UTC m=+2843.450423042" Oct 30 17:10:16.434438 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:16.434409 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82d4a543-x4slv"] Oct 30 17:10:16.435482 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:10:16.435453 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4075789_9024_4cb3_bf1b_fd540609361f.slice/crio-554638de505415ceddc56860592c0e9a7a8c419a583ae78696c196d1e78f0deb WatchSource:0}: Error finding container 554638de505415ceddc56860592c0e9a7a8c419a583ae78696c196d1e78f0deb: Status 404 returned error can't find the container with id 554638de505415ceddc56860592c0e9a7a8c419a583ae78696c196d1e78f0deb Oct 30 17:10:17.245809 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:17.231143 4493 generic.go:356] "Generic (PLEG): container finished" podID="8abd1354-46f4-40cc-b5cc-c86321cfb197" containerID="2c984d23330fc1b8a018652cba6542db6099cb0a4b8a62216991aa2a3752d601" exitCode=0 Oct 30 17:10:17.245809 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:17.231289 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c03254f-9rlgl" event={"ID":"8abd1354-46f4-40cc-b5cc-c86321cfb197","Type":"ContainerDied","Data":"2c984d23330fc1b8a018652cba6542db6099cb0a4b8a62216991aa2a3752d601"} Oct 30 17:10:17.245809 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:17.238085 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d4a543-x4slv" event={"ID":"f4075789-9024-4cb3-bf1b-fd540609361f","Type":"ContainerStarted","Data":"4ed64a9ddd5eb0bcd614c03f8cdda1a916a9e691006b83283206ad641dd7dfad"} Oct 30 17:10:17.245809 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:17.238145 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d4a543-x4slv" event={"ID":"f4075789-9024-4cb3-bf1b-fd540609361f","Type":"ContainerStarted","Data":"554638de505415ceddc56860592c0e9a7a8c419a583ae78696c196d1e78f0deb"} Oct 30 17:10:17.349808 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:17.349664 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d4a543-x4slv" podStartSLOduration=2.349644371 podStartE2EDuration="2.349644371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:10:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:10:17.348257236 +0000 UTC m=+2844.575432339" watchObservedRunningTime="2026-10-30 17:10:17.349644371 +0000 UTC m=+2844.576819472" Oct 30 17:10:18.250691 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.250652 4493 generic.go:356] "Generic (PLEG): container finished" podID="f4075789-9024-4cb3-bf1b-fd540609361f" containerID="4ed64a9ddd5eb0bcd614c03f8cdda1a916a9e691006b83283206ad641dd7dfad" exitCode=0 Oct 30 17:10:18.251196 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.250731 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d4a543-x4slv" event={"ID":"f4075789-9024-4cb3-bf1b-fd540609361f","Type":"ContainerDied","Data":"4ed64a9ddd5eb0bcd614c03f8cdda1a916a9e691006b83283206ad641dd7dfad"} Oct 30 17:10:18.290947 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.288962 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fl6dw"] Oct 30 17:10:18.294561 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.294533 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q2t9l"] Oct 30 17:10:18.294704 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.294685 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fl6dw" Oct 30 17:10:18.300535 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.299571 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q2t9l" Oct 30 17:10:18.319132 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.318445 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-xtm6x\"" Oct 30 17:10:18.350024 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.330734 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fl6dw"] Oct 30 17:10:18.350964 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.350918 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q2t9l"] Oct 30 17:10:18.393042 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.392612 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dczl\" (UniqueName: \"kubernetes.io/projected/98c9aa43-9e50-4612-9965-4a20f9458deb-kube-api-access-6dczl\") pod \"maas-default-gateway-openshift-default-7854949c9-q2t9l\" (UID: \"98c9aa43-9e50-4612-9965-4a20f9458deb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q2t9l" Oct 30 17:10:18.393369 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.393340 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98c9aa43-9e50-4612-9965-4a20f9458deb-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-q2t9l\" (UID: \"98c9aa43-9e50-4612-9965-4a20f9458deb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q2t9l" Oct 30 17:10:18.393526 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.393510 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15e78f14-6c3c-4bfa-93ea-bf58c888b372-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-fl6dw\" (UID: \"15e78f14-6c3c-4bfa-93ea-bf58c888b372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fl6dw" Oct 30 17:10:18.393660 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.393646 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15e78f14-6c3c-4bfa-93ea-bf58c888b372-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-fl6dw\" (UID: \"15e78f14-6c3c-4bfa-93ea-bf58c888b372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fl6dw" Oct 30 17:10:18.393827 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.393813 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98c9aa43-9e50-4612-9965-4a20f9458deb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-q2t9l\" (UID: \"98c9aa43-9e50-4612-9965-4a20f9458deb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q2t9l" Oct 30 17:10:18.393952 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.393939 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98c9aa43-9e50-4612-9965-4a20f9458deb-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-q2t9l\" (UID: \"98c9aa43-9e50-4612-9965-4a20f9458deb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q2t9l" Oct 30 17:10:18.394086 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.394071 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15e78f14-6c3c-4bfa-93ea-bf58c888b372-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-fl6dw\" (UID: \"15e78f14-6c3c-4bfa-93ea-bf58c888b372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fl6dw" Oct 30 17:10:18.394248 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.394233 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98c9aa43-9e50-4612-9965-4a20f9458deb-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-q2t9l\" (UID: \"98c9aa43-9e50-4612-9965-4a20f9458deb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q2t9l" Oct 30 17:10:18.394381 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.394366 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98c9aa43-9e50-4612-9965-4a20f9458deb-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-q2t9l\" (UID: \"98c9aa43-9e50-4612-9965-4a20f9458deb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q2t9l" Oct 30 17:10:18.394567 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.394549 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98c9aa43-9e50-4612-9965-4a20f9458deb-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-q2t9l\" (UID: \"98c9aa43-9e50-4612-9965-4a20f9458deb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q2t9l" Oct 30 17:10:18.394735 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.394715 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15e78f14-6c3c-4bfa-93ea-bf58c888b372-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-fl6dw\" (UID: \"15e78f14-6c3c-4bfa-93ea-bf58c888b372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fl6dw" Oct 30 17:10:18.394875 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.394860 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15e78f14-6c3c-4bfa-93ea-bf58c888b372-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-fl6dw\" (UID: \"15e78f14-6c3c-4bfa-93ea-bf58c888b372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fl6dw" Oct 30 17:10:18.394999 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.394986 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcqbj\" (UniqueName: \"kubernetes.io/projected/15e78f14-6c3c-4bfa-93ea-bf58c888b372-kube-api-access-vcqbj\") pod \"maas-default-gateway-openshift-default-7854949c9-fl6dw\" (UID: \"15e78f14-6c3c-4bfa-93ea-bf58c888b372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fl6dw" Oct 30 17:10:18.395164 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.395149 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98c9aa43-9e50-4612-9965-4a20f9458deb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-q2t9l\" (UID: \"98c9aa43-9e50-4612-9965-4a20f9458deb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q2t9l" Oct 30 17:10:18.395291 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.395276 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15e78f14-6c3c-4bfa-93ea-bf58c888b372-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-fl6dw\" (UID: \"15e78f14-6c3c-4bfa-93ea-bf58c888b372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fl6dw" Oct 30 17:10:18.395395 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.395381 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15e78f14-6c3c-4bfa-93ea-bf58c888b372-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-fl6dw\" (UID: \"15e78f14-6c3c-4bfa-93ea-bf58c888b372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fl6dw" Oct 30 17:10:18.395614 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.395596 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98c9aa43-9e50-4612-9965-4a20f9458deb-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-q2t9l\" (UID: \"98c9aa43-9e50-4612-9965-4a20f9458deb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q2t9l" Oct 30 17:10:18.395851 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.395828 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15e78f14-6c3c-4bfa-93ea-bf58c888b372-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-fl6dw\" (UID: \"15e78f14-6c3c-4bfa-93ea-bf58c888b372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fl6dw" Oct 30 17:10:18.462523 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.462481 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 17:10:18.472628 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.472602 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c03254f-9rlgl" Oct 30 17:10:18.497277 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.497250 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8abd1354-46f4-40cc-b5cc-c86321cfb197-kube-api-access-pwgnq\" (UniqueName: \"kubernetes.io/projected/8abd1354-46f4-40cc-b5cc-c86321cfb197-kube-api-access-pwgnq\") pod \"8abd1354-46f4-40cc-b5cc-c86321cfb197\" (UID: \"8abd1354-46f4-40cc-b5cc-c86321cfb197\") " Oct 30 17:10:18.497440 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.497321 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8abd1354-46f4-40cc-b5cc-c86321cfb197-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8abd1354-46f4-40cc-b5cc-c86321cfb197-maas-api-openshift-ca\") pod \"8abd1354-46f4-40cc-b5cc-c86321cfb197\" (UID: \"8abd1354-46f4-40cc-b5cc-c86321cfb197\") " Oct 30 17:10:18.497526 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.497496 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8abd1354-46f4-40cc-b5cc-c86321cfb197-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8abd1354-46f4-40cc-b5cc-c86321cfb197-maas-api-xks-ca\") pod \"8abd1354-46f4-40cc-b5cc-c86321cfb197\" (UID: \"8abd1354-46f4-40cc-b5cc-c86321cfb197\") " Oct 30 17:10:18.497734 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.497691 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8abd1354-46f4-40cc-b5cc-c86321cfb197-maas-api-openshift-ca" pod "8abd1354-46f4-40cc-b5cc-c86321cfb197" (UID: "8abd1354-46f4-40cc-b5cc-c86321cfb197"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:10:18.497954 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.497932 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98c9aa43-9e50-4612-9965-4a20f9458deb-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-q2t9l\" (UID: \"98c9aa43-9e50-4612-9965-4a20f9458deb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q2t9l" Oct 30 17:10:18.498039 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.497998 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15e78f14-6c3c-4bfa-93ea-bf58c888b372-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-fl6dw\" (UID: \"15e78f14-6c3c-4bfa-93ea-bf58c888b372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fl6dw" Oct 30 17:10:18.498039 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.498025 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15e78f14-6c3c-4bfa-93ea-bf58c888b372-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-fl6dw\" (UID: \"15e78f14-6c3c-4bfa-93ea-bf58c888b372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fl6dw" Oct 30 17:10:18.498129 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.498042 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcqbj\" (UniqueName: \"kubernetes.io/projected/15e78f14-6c3c-4bfa-93ea-bf58c888b372-kube-api-access-vcqbj\") pod \"maas-default-gateway-openshift-default-7854949c9-fl6dw\" (UID: \"15e78f14-6c3c-4bfa-93ea-bf58c888b372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fl6dw" Oct 30 17:10:18.498129 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.498074 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98c9aa43-9e50-4612-9965-4a20f9458deb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-q2t9l\" (UID: \"98c9aa43-9e50-4612-9965-4a20f9458deb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q2t9l" Oct 30 17:10:18.498129 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.498096 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15e78f14-6c3c-4bfa-93ea-bf58c888b372-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-fl6dw\" (UID: \"15e78f14-6c3c-4bfa-93ea-bf58c888b372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fl6dw" Oct 30 17:10:18.498129 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.498111 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15e78f14-6c3c-4bfa-93ea-bf58c888b372-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-fl6dw\" (UID: \"15e78f14-6c3c-4bfa-93ea-bf58c888b372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fl6dw" Oct 30 17:10:18.498315 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.498167 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98c9aa43-9e50-4612-9965-4a20f9458deb-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-q2t9l\" (UID: \"98c9aa43-9e50-4612-9965-4a20f9458deb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q2t9l" Oct 30 17:10:18.498315 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.498211 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15e78f14-6c3c-4bfa-93ea-bf58c888b372-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-fl6dw\" (UID: \"15e78f14-6c3c-4bfa-93ea-bf58c888b372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fl6dw" Oct 30 17:10:18.498315 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.498248 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dczl\" (UniqueName: \"kubernetes.io/projected/98c9aa43-9e50-4612-9965-4a20f9458deb-kube-api-access-6dczl\") pod \"maas-default-gateway-openshift-default-7854949c9-q2t9l\" (UID: \"98c9aa43-9e50-4612-9965-4a20f9458deb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q2t9l" Oct 30 17:10:18.498315 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.498274 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98c9aa43-9e50-4612-9965-4a20f9458deb-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-q2t9l\" (UID: \"98c9aa43-9e50-4612-9965-4a20f9458deb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q2t9l" Oct 30 17:10:18.498315 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.498307 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15e78f14-6c3c-4bfa-93ea-bf58c888b372-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-fl6dw\" (UID: \"15e78f14-6c3c-4bfa-93ea-bf58c888b372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fl6dw" Oct 30 17:10:18.498575 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.498335 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15e78f14-6c3c-4bfa-93ea-bf58c888b372-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-fl6dw\" (UID: \"15e78f14-6c3c-4bfa-93ea-bf58c888b372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fl6dw" Oct 30 17:10:18.498575 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.498381 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98c9aa43-9e50-4612-9965-4a20f9458deb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-q2t9l\" (UID: \"98c9aa43-9e50-4612-9965-4a20f9458deb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q2t9l" Oct 30 17:10:18.498575 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.498411 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98c9aa43-9e50-4612-9965-4a20f9458deb-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-q2t9l\" (UID: \"98c9aa43-9e50-4612-9965-4a20f9458deb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q2t9l" Oct 30 17:10:18.498575 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.498429 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15e78f14-6c3c-4bfa-93ea-bf58c888b372-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-fl6dw\" (UID: \"15e78f14-6c3c-4bfa-93ea-bf58c888b372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fl6dw" Oct 30 17:10:18.498575 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.498445 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98c9aa43-9e50-4612-9965-4a20f9458deb-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-q2t9l\" (UID: \"98c9aa43-9e50-4612-9965-4a20f9458deb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q2t9l" Oct 30 17:10:18.498575 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.498476 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98c9aa43-9e50-4612-9965-4a20f9458deb-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-q2t9l\" (UID: \"98c9aa43-9e50-4612-9965-4a20f9458deb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q2t9l" Oct 30 17:10:18.498575 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.498549 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8abd1354-46f4-40cc-b5cc-c86321cfb197-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:10:18.499026 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.498821 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98c9aa43-9e50-4612-9965-4a20f9458deb-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-q2t9l\" (UID: \"98c9aa43-9e50-4612-9965-4a20f9458deb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q2t9l" Oct 30 17:10:18.499104 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.499083 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98c9aa43-9e50-4612-9965-4a20f9458deb-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-q2t9l\" (UID: \"98c9aa43-9e50-4612-9965-4a20f9458deb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q2t9l" Oct 30 17:10:18.499351 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.499325 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15e78f14-6c3c-4bfa-93ea-bf58c888b372-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-fl6dw\" (UID: \"15e78f14-6c3c-4bfa-93ea-bf58c888b372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fl6dw" Oct 30 17:10:18.499579 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.499560 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98c9aa43-9e50-4612-9965-4a20f9458deb-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-q2t9l\" (UID: \"98c9aa43-9e50-4612-9965-4a20f9458deb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q2t9l" Oct 30 17:10:18.501737 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.500184 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15e78f14-6c3c-4bfa-93ea-bf58c888b372-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-fl6dw\" (UID: \"15e78f14-6c3c-4bfa-93ea-bf58c888b372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fl6dw" Oct 30 17:10:18.501737 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.500223 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8abd1354-46f4-40cc-b5cc-c86321cfb197-maas-api-xks-ca" pod "8abd1354-46f4-40cc-b5cc-c86321cfb197" (UID: "8abd1354-46f4-40cc-b5cc-c86321cfb197"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:10:18.501737 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.500430 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15e78f14-6c3c-4bfa-93ea-bf58c888b372-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-fl6dw\" (UID: \"15e78f14-6c3c-4bfa-93ea-bf58c888b372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fl6dw" Oct 30 17:10:18.501737 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.501009 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15e78f14-6c3c-4bfa-93ea-bf58c888b372-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-fl6dw\" (UID: \"15e78f14-6c3c-4bfa-93ea-bf58c888b372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fl6dw" Oct 30 17:10:18.501737 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.501275 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98c9aa43-9e50-4612-9965-4a20f9458deb-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-q2t9l\" (UID: \"98c9aa43-9e50-4612-9965-4a20f9458deb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q2t9l" Oct 30 17:10:18.501737 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.501338 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15e78f14-6c3c-4bfa-93ea-bf58c888b372-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-fl6dw\" (UID: \"15e78f14-6c3c-4bfa-93ea-bf58c888b372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fl6dw" Oct 30 17:10:18.502376 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.502343 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98c9aa43-9e50-4612-9965-4a20f9458deb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-q2t9l\" (UID: \"98c9aa43-9e50-4612-9965-4a20f9458deb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q2t9l" Oct 30 17:10:18.502545 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.502517 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8abd1354-46f4-40cc-b5cc-c86321cfb197-kube-api-access-pwgnq" pod "8abd1354-46f4-40cc-b5cc-c86321cfb197" (UID: "8abd1354-46f4-40cc-b5cc-c86321cfb197"). InnerVolumeSpecName "kube-api-access-pwgnq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:10:18.502730 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.502707 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15e78f14-6c3c-4bfa-93ea-bf58c888b372-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-fl6dw\" (UID: \"15e78f14-6c3c-4bfa-93ea-bf58c888b372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fl6dw" Oct 30 17:10:18.504158 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.504129 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98c9aa43-9e50-4612-9965-4a20f9458deb-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-q2t9l\" (UID: \"98c9aa43-9e50-4612-9965-4a20f9458deb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q2t9l" Oct 30 17:10:18.504331 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.504308 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98c9aa43-9e50-4612-9965-4a20f9458deb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-q2t9l\" (UID: \"98c9aa43-9e50-4612-9965-4a20f9458deb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q2t9l" Oct 30 17:10:18.504482 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.504458 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15e78f14-6c3c-4bfa-93ea-bf58c888b372-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-fl6dw\" (UID: \"15e78f14-6c3c-4bfa-93ea-bf58c888b372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fl6dw" Oct 30 17:10:18.531935 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.531282 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcqbj\" (UniqueName: \"kubernetes.io/projected/15e78f14-6c3c-4bfa-93ea-bf58c888b372-kube-api-access-vcqbj\") pod \"maas-default-gateway-openshift-default-7854949c9-fl6dw\" (UID: \"15e78f14-6c3c-4bfa-93ea-bf58c888b372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fl6dw" Oct 30 17:10:18.531935 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.531472 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98c9aa43-9e50-4612-9965-4a20f9458deb-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-q2t9l\" (UID: \"98c9aa43-9e50-4612-9965-4a20f9458deb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q2t9l" Oct 30 17:10:18.531935 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.531887 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dczl\" (UniqueName: \"kubernetes.io/projected/98c9aa43-9e50-4612-9965-4a20f9458deb-kube-api-access-6dczl\") pod \"maas-default-gateway-openshift-default-7854949c9-q2t9l\" (UID: \"98c9aa43-9e50-4612-9965-4a20f9458deb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q2t9l" Oct 30 17:10:18.534271 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.534240 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15e78f14-6c3c-4bfa-93ea-bf58c888b372-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-fl6dw\" (UID: \"15e78f14-6c3c-4bfa-93ea-bf58c888b372\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fl6dw" Oct 30 17:10:18.600363 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.600323 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8abd1354-46f4-40cc-b5cc-c86321cfb197-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:10:18.600363 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.600362 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwgnq\" (UniqueName: \"kubernetes.io/projected/8abd1354-46f4-40cc-b5cc-c86321cfb197-kube-api-access-pwgnq\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:10:18.644572 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.644534 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fl6dw" Oct 30 17:10:18.674934 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.674899 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q2t9l" Oct 30 17:10:18.977581 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.977552 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fl6dw"] Oct 30 17:10:18.978883 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:10:18.978850 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15e78f14_6c3c_4bfa_93ea_bf58c888b372.slice/crio-c1459ad7364fc96ddb40e3e8f11d277242754a5ec09dc6d4241e3b39c882ab4e WatchSource:0}: Error finding container c1459ad7364fc96ddb40e3e8f11d277242754a5ec09dc6d4241e3b39c882ab4e: Status 404 returned error can't find the container with id c1459ad7364fc96ddb40e3e8f11d277242754a5ec09dc6d4241e3b39c882ab4e Oct 30 17:10:18.987418 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.986435 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:10:18.987418 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.986625 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:10:18.987418 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:18.986680 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:10:19.031658 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:19.031629 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q2t9l"] Oct 30 17:10:19.035245 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:10:19.035209 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98c9aa43_9e50_4612_9965_4a20f9458deb.slice/crio-fe82f4bc1e48b20ebe6abee23c91700f058f0aeec06dcf4032feb0a02a6bf225 WatchSource:0}: Error finding container fe82f4bc1e48b20ebe6abee23c91700f058f0aeec06dcf4032feb0a02a6bf225: Status 404 returned error can't find the container with id fe82f4bc1e48b20ebe6abee23c91700f058f0aeec06dcf4032feb0a02a6bf225 Oct 30 17:10:19.038566 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:19.038527 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:10:19.038676 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:19.038609 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:10:19.038676 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:19.038659 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:10:19.277372 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:19.276572 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c03254f-9rlgl" Oct 30 17:10:19.277372 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:19.277293 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c03254f-9rlgl" event={"ID":"8abd1354-46f4-40cc-b5cc-c86321cfb197","Type":"ContainerDied","Data":"3e0957b509065a655037ef5d449e9c45162d3322297b6978b41cf996f95e1aa5"} Oct 30 17:10:19.277372 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:19.277341 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e0957b509065a655037ef5d449e9c45162d3322297b6978b41cf996f95e1aa5" Oct 30 17:10:19.282510 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:19.282435 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q2t9l" event={"ID":"98c9aa43-9e50-4612-9965-4a20f9458deb","Type":"ContainerStarted","Data":"989d496ae2a43920deeb5428d4c4342649a7a36d786be51b7fab7d6c28a8f2a1"} Oct 30 17:10:19.282708 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:19.282689 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q2t9l" event={"ID":"98c9aa43-9e50-4612-9965-4a20f9458deb","Type":"ContainerStarted","Data":"fe82f4bc1e48b20ebe6abee23c91700f058f0aeec06dcf4032feb0a02a6bf225"} Oct 30 17:10:19.287182 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:19.286869 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fl6dw" event={"ID":"15e78f14-6c3c-4bfa-93ea-bf58c888b372","Type":"ContainerStarted","Data":"fc739947d000efae787b77ed56d9a0cca572582d78a7b436981200dff3b53e84"} Oct 30 17:10:19.287182 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:19.286912 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fl6dw" event={"ID":"15e78f14-6c3c-4bfa-93ea-bf58c888b372","Type":"ContainerStarted","Data":"c1459ad7364fc96ddb40e3e8f11d277242754a5ec09dc6d4241e3b39c882ab4e"} Oct 30 17:10:19.435042 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:19.434072 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q2t9l" podStartSLOduration=1.434053341 podStartE2EDuration="1.434053341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:10:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:10:19.340868136 +0000 UTC m=+2846.568043236" watchObservedRunningTime="2026-10-30 17:10:19.434053341 +0000 UTC m=+2846.661228442" Oct 30 17:10:19.441809 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:19.441510 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fl6dw" podStartSLOduration=1.441490973 podStartE2EDuration="1.441490973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:10:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:10:19.399457623 +0000 UTC m=+2846.626632724" watchObservedRunningTime="2026-10-30 17:10:19.441490973 +0000 UTC m=+2846.668666073" Oct 30 17:10:19.554042 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:19.554012 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d4a543-x4slv" Oct 30 17:10:19.638610 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:19.637425 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4075789-9024-4cb3-bf1b-fd540609361f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4075789-9024-4cb3-bf1b-fd540609361f-maas-api-xks-ca\") pod \"f4075789-9024-4cb3-bf1b-fd540609361f\" (UID: \"f4075789-9024-4cb3-bf1b-fd540609361f\") " Oct 30 17:10:19.639684 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:19.639659 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4075789-9024-4cb3-bf1b-fd540609361f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4075789-9024-4cb3-bf1b-fd540609361f-maas-api-openshift-ca\") pod \"f4075789-9024-4cb3-bf1b-fd540609361f\" (UID: \"f4075789-9024-4cb3-bf1b-fd540609361f\") " Oct 30 17:10:19.652420 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:19.642972 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4075789-9024-4cb3-bf1b-fd540609361f-maas-api-xks-ca" pod "f4075789-9024-4cb3-bf1b-fd540609361f" (UID: "f4075789-9024-4cb3-bf1b-fd540609361f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:10:19.652420 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:19.645074 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4075789-9024-4cb3-bf1b-fd540609361f-maas-api-openshift-ca" pod "f4075789-9024-4cb3-bf1b-fd540609361f" (UID: "f4075789-9024-4cb3-bf1b-fd540609361f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:10:19.652420 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:19.645133 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fl6dw" Oct 30 17:10:19.653033 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:19.653012 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4075789-9024-4cb3-bf1b-fd540609361f-kube-api-access-c5n7s\" (UniqueName: \"kubernetes.io/projected/f4075789-9024-4cb3-bf1b-fd540609361f-kube-api-access-c5n7s\") pod \"f4075789-9024-4cb3-bf1b-fd540609361f\" (UID: \"f4075789-9024-4cb3-bf1b-fd540609361f\") " Oct 30 17:10:19.653904 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:19.653808 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4075789-9024-4cb3-bf1b-fd540609361f-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:10:19.653904 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:19.653835 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4075789-9024-4cb3-bf1b-fd540609361f-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:10:19.657833 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:19.657034 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4075789-9024-4cb3-bf1b-fd540609361f-kube-api-access-c5n7s" pod "f4075789-9024-4cb3-bf1b-fd540609361f" (UID: "f4075789-9024-4cb3-bf1b-fd540609361f"). InnerVolumeSpecName "kube-api-access-c5n7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:10:19.679001 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:19.678923 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q2t9l" Oct 30 17:10:19.759821 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:19.756826 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5n7s\" (UniqueName: \"kubernetes.io/projected/f4075789-9024-4cb3-bf1b-fd540609361f-kube-api-access-c5n7s\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:10:20.135897 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:20.135852 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d1a2471-lfwx6"] Oct 30 17:10:20.238284 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:20.238253 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d1a2471-lfwx6"] Oct 30 17:10:20.705970 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:20.705933 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-fl6dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" start-of-body= Oct 30 17:10:20.747810 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:20.709855 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fl6dw" podUID="15e78f14-6c3c-4bfa-93ea-bf58c888b372" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" Oct 30 17:10:20.747810 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:20.710290 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-q2t9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" start-of-body= Oct 30 17:10:20.747810 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:20.710341 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q2t9l" podUID="98c9aa43-9e50-4612-9965-4a20f9458deb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" Oct 30 17:10:20.917838 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:20.865373 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d4a543-x4slv" Oct 30 17:10:20.917838 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:20.866865 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d4a543-x4slv" event={"ID":"f4075789-9024-4cb3-bf1b-fd540609361f","Type":"ContainerDied","Data":"554638de505415ceddc56860592c0e9a7a8c419a583ae78696c196d1e78f0deb"} Oct 30 17:10:20.917838 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:20.867154 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="554638de505415ceddc56860592c0e9a7a8c419a583ae78696c196d1e78f0deb" Oct 30 17:10:21.245819 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:21.237399 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dab6fd8b-e841-4e04-889f-5d5c35a9cdba" path="/var/lib/kubelet/pods/dab6fd8b-e841-4e04-889f-5d5c35a9cdba/volumes" Oct 30 17:10:21.245819 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:21.238145 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g"] Oct 30 17:10:21.338809 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:21.338017 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" podUID="18b72648-f523-405e-b5cd-1266ff13f8f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 17:10:21.370601 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:21.370209 4493 cpu_manager.go:414] "containerMap: removing container" podUID="8abd1354-46f4-40cc-b5cc-c86321cfb197" containerName="revoke-keys" Oct 30 17:10:21.370601 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:21.370234 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8abd1354-46f4-40cc-b5cc-c86321cfb197" containerName="revoke-keys" Oct 30 17:10:21.370601 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:21.370291 4493 cpu_manager.go:414] "containerMap: removing container" podUID="f4075789-9024-4cb3-bf1b-fd540609361f" containerName="revoke-keys" Oct 30 17:10:21.370601 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:21.370299 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4075789-9024-4cb3-bf1b-fd540609361f" containerName="revoke-keys" Oct 30 17:10:21.370601 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:21.370486 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4075789-9024-4cb3-bf1b-fd540609361f" containerName="revoke-keys" Oct 30 17:10:21.370601 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:21.370507 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="8abd1354-46f4-40cc-b5cc-c86321cfb197" containerName="revoke-keys" Oct 30 17:10:21.376839 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:21.375307 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g"] Oct 30 17:10:21.376839 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:21.375424 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g" Oct 30 17:10:21.394458 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:21.394421 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-f94fb5-openshift-default-dockercfg-9vtqh\"" Oct 30 17:10:21.489015 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:10:21.488972 4493 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4075789_9024_4cb3_bf1b_fd540609361f.slice\": RecentStats: unable to find data in memory cache]" Oct 30 17:10:21.525649 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:21.525549 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a4af0cd-c986-4b31-b123-545551673476-istiod-ca-cert\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g\" (UID: \"5a4af0cd-c986-4b31-b123-545551673476\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g" Oct 30 17:10:21.525965 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:21.525944 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a4af0cd-c986-4b31-b123-545551673476-istio-data\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g\" (UID: \"5a4af0cd-c986-4b31-b123-545551673476\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g" Oct 30 17:10:21.526120 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:21.526103 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5txdt\" (UniqueName: \"kubernetes.io/projected/5a4af0cd-c986-4b31-b123-545551673476-kube-api-access-5txdt\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g\" (UID: \"5a4af0cd-c986-4b31-b123-545551673476\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g" Oct 30 17:10:21.526253 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:21.526240 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a4af0cd-c986-4b31-b123-545551673476-istio-token\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g\" (UID: \"5a4af0cd-c986-4b31-b123-545551673476\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g" Oct 30 17:10:21.526369 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:21.526356 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a4af0cd-c986-4b31-b123-545551673476-workload-certs\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g\" (UID: \"5a4af0cd-c986-4b31-b123-545551673476\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g" Oct 30 17:10:21.526492 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:21.526480 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a4af0cd-c986-4b31-b123-545551673476-istio-podinfo\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g\" (UID: \"5a4af0cd-c986-4b31-b123-545551673476\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g" Oct 30 17:10:21.526663 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:21.526648 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a4af0cd-c986-4b31-b123-545551673476-istio-envoy\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g\" (UID: \"5a4af0cd-c986-4b31-b123-545551673476\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g" Oct 30 17:10:21.526857 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:21.526843 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a4af0cd-c986-4b31-b123-545551673476-workload-socket\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g\" (UID: \"5a4af0cd-c986-4b31-b123-545551673476\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g" Oct 30 17:10:21.526990 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:21.526976 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a4af0cd-c986-4b31-b123-545551673476-credential-socket\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g\" (UID: \"5a4af0cd-c986-4b31-b123-545551673476\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g" Oct 30 17:10:21.631811 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:21.631589 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a4af0cd-c986-4b31-b123-545551673476-istio-podinfo\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g\" (UID: \"5a4af0cd-c986-4b31-b123-545551673476\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g" Oct 30 17:10:21.631811 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:21.631687 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a4af0cd-c986-4b31-b123-545551673476-istio-envoy\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g\" (UID: \"5a4af0cd-c986-4b31-b123-545551673476\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g" Oct 30 17:10:21.632080 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:21.631843 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a4af0cd-c986-4b31-b123-545551673476-workload-socket\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g\" (UID: \"5a4af0cd-c986-4b31-b123-545551673476\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g" Oct 30 17:10:21.632080 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:21.631898 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a4af0cd-c986-4b31-b123-545551673476-credential-socket\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g\" (UID: \"5a4af0cd-c986-4b31-b123-545551673476\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g" Oct 30 17:10:21.632080 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:21.631965 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a4af0cd-c986-4b31-b123-545551673476-istiod-ca-cert\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g\" (UID: \"5a4af0cd-c986-4b31-b123-545551673476\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g" Oct 30 17:10:21.632080 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:21.632020 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a4af0cd-c986-4b31-b123-545551673476-istio-data\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g\" (UID: \"5a4af0cd-c986-4b31-b123-545551673476\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g" Oct 30 17:10:21.632080 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:21.632068 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5txdt\" (UniqueName: \"kubernetes.io/projected/5a4af0cd-c986-4b31-b123-545551673476-kube-api-access-5txdt\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g\" (UID: \"5a4af0cd-c986-4b31-b123-545551673476\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g" Oct 30 17:10:21.632381 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:21.632124 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a4af0cd-c986-4b31-b123-545551673476-istio-token\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g\" (UID: \"5a4af0cd-c986-4b31-b123-545551673476\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g" Oct 30 17:10:21.632381 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:21.632168 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a4af0cd-c986-4b31-b123-545551673476-workload-certs\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g\" (UID: \"5a4af0cd-c986-4b31-b123-545551673476\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g" Oct 30 17:10:21.654224 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:21.632645 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a4af0cd-c986-4b31-b123-545551673476-workload-certs\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g\" (UID: \"5a4af0cd-c986-4b31-b123-545551673476\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g" Oct 30 17:10:21.654224 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:21.633720 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a4af0cd-c986-4b31-b123-545551673476-istiod-ca-cert\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g\" (UID: \"5a4af0cd-c986-4b31-b123-545551673476\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g" Oct 30 17:10:21.654224 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:21.645376 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a4af0cd-c986-4b31-b123-545551673476-istio-data\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g\" (UID: \"5a4af0cd-c986-4b31-b123-545551673476\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g" Oct 30 17:10:21.654224 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:21.645899 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a4af0cd-c986-4b31-b123-545551673476-workload-socket\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g\" (UID: \"5a4af0cd-c986-4b31-b123-545551673476\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g" Oct 30 17:10:21.654224 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:21.646057 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a4af0cd-c986-4b31-b123-545551673476-credential-socket\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g\" (UID: \"5a4af0cd-c986-4b31-b123-545551673476\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g" Oct 30 17:10:21.654224 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:21.647883 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-fl6dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" start-of-body= Oct 30 17:10:21.654224 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:21.647926 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fl6dw" podUID="15e78f14-6c3c-4bfa-93ea-bf58c888b372" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" Oct 30 17:10:21.667809 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:21.666146 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a4af0cd-c986-4b31-b123-545551673476-istio-envoy\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g\" (UID: \"5a4af0cd-c986-4b31-b123-545551673476\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g" Oct 30 17:10:21.670285 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:21.670257 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a4af0cd-c986-4b31-b123-545551673476-istio-token\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g\" (UID: \"5a4af0cd-c986-4b31-b123-545551673476\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g" Oct 30 17:10:21.672067 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:21.672037 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a4af0cd-c986-4b31-b123-545551673476-istio-podinfo\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g\" (UID: \"5a4af0cd-c986-4b31-b123-545551673476\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g" Oct 30 17:10:21.672871 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:21.672852 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5txdt\" (UniqueName: \"kubernetes.io/projected/5a4af0cd-c986-4b31-b123-545551673476-kube-api-access-5txdt\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g\" (UID: \"5a4af0cd-c986-4b31-b123-545551673476\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g" Oct 30 17:10:21.675830 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:21.675809 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-q2t9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" start-of-body= Oct 30 17:10:21.675918 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:21.675852 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q2t9l" podUID="98c9aa43-9e50-4612-9965-4a20f9458deb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" Oct 30 17:10:21.723284 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:21.723013 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g" Oct 30 17:10:22.173044 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:22.173012 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g"] Oct 30 17:10:22.175577 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:10:22.175547 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a4af0cd_c986_4b31_b123_545551673476.slice/crio-b3b680da4a45bbdef8078b6a59fb01bf9b6652605befa87fc661941b5c5b553f WatchSource:0}: Error finding container b3b680da4a45bbdef8078b6a59fb01bf9b6652605befa87fc661941b5c5b553f: Status 404 returned error can't find the container with id b3b680da4a45bbdef8078b6a59fb01bf9b6652605befa87fc661941b5c5b553f Oct 30 17:10:22.179880 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:22.179836 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:10:22.179987 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:22.179934 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:10:22.179987 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:22.179980 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:10:22.645219 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:22.645185 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-fl6dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" start-of-body= Oct 30 17:10:22.645501 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:22.645467 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fl6dw" podUID="15e78f14-6c3c-4bfa-93ea-bf58c888b372" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" Oct 30 17:10:22.676104 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:22.676069 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-q2t9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" start-of-body= Oct 30 17:10:22.676365 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:22.676337 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q2t9l" podUID="98c9aa43-9e50-4612-9965-4a20f9458deb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" Oct 30 17:10:22.979464 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:22.979359 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g" event={"ID":"5a4af0cd-c986-4b31-b123-545551673476","Type":"ContainerStarted","Data":"aa7da326256c6a80b6c838b35b061c7ac6712e8ff7a185f4408c6392edcaed72"} Oct 30 17:10:22.979464 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:22.979434 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g" event={"ID":"5a4af0cd-c986-4b31-b123-545551673476","Type":"ContainerStarted","Data":"b3b680da4a45bbdef8078b6a59fb01bf9b6652605befa87fc661941b5c5b553f"} Oct 30 17:10:22.998149 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:22.996288 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g" podStartSLOduration=2.996267114 podStartE2EDuration="2.996267114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:10:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:10:22.992602719 +0000 UTC m=+2850.219777821" watchObservedRunningTime="2026-10-30 17:10:22.996267114 +0000 UTC m=+2850.223442215" Oct 30 17:10:23.653809 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:23.653351 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-fl6dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" start-of-body= Oct 30 17:10:23.653809 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:23.653416 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fl6dw" podUID="15e78f14-6c3c-4bfa-93ea-bf58c888b372" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" Oct 30 17:10:23.675666 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:23.675631 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-q2t9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" start-of-body= Oct 30 17:10:23.675986 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:23.675956 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q2t9l" podUID="98c9aa43-9e50-4612-9965-4a20f9458deb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" Oct 30 17:10:23.760811 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:23.732900 4493 patch_prober.go:29] interesting pod/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" start-of-body= Oct 30 17:10:23.760811 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:23.732991 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g" podUID="5a4af0cd-c986-4b31-b123-545551673476" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" Oct 30 17:10:23.796918 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:23.768846 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g" Oct 30 17:10:24.681366 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:24.680546 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-fl6dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" start-of-body= Oct 30 17:10:24.681366 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:24.680623 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fl6dw" podUID="15e78f14-6c3c-4bfa-93ea-bf58c888b372" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" Oct 30 17:10:24.681366 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:24.681269 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-q2t9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" start-of-body= Oct 30 17:10:24.681366 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:24.681311 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q2t9l" podUID="98c9aa43-9e50-4612-9965-4a20f9458deb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" Oct 30 17:10:24.736319 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:24.736164 4493 patch_prober.go:29] interesting pod/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" start-of-body= Oct 30 17:10:24.736319 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:24.736232 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g" podUID="5a4af0cd-c986-4b31-b123-545551673476" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" Oct 30 17:10:25.310203 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:25.305918 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" podUID="80b9bd9a-30d6-45ff-b318-26fcdc608e5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:10:25.495815 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:25.494904 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" podUID="0eadc5d4-98c3-4bf2-b3c2-f73fe249182f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:10:25.738255 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:25.738215 4493 patch_prober.go:29] interesting pod/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" start-of-body= Oct 30 17:10:25.738853 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:25.738819 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g" podUID="5a4af0cd-c986-4b31-b123-545551673476" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" Oct 30 17:10:25.827934 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:25.827896 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-q2t9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" start-of-body= Oct 30 17:10:25.828144 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:25.827969 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q2t9l" podUID="98c9aa43-9e50-4612-9965-4a20f9458deb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" Oct 30 17:10:25.828546 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:25.828519 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-fl6dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" start-of-body= Oct 30 17:10:25.828657 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:25.828561 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fl6dw" podUID="15e78f14-6c3c-4bfa-93ea-bf58c888b372" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" Oct 30 17:10:26.170875 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:26.123678 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" podUID="e76172b1-659d-45bd-91c6-1a0ce4062335" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:10:26.171350 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:26.129851 4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" podUID="e76172b1-659d-45bd-91c6-1a0ce4062335" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:10:26.191984 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:26.191568 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" podUID="e90f43d3-f5eb-45ab-af1e-340d0ce18966" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 17:10:26.298901 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:26.298859 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" podUID="18b72648-f523-405e-b5cd-1266ff13f8f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 17:10:26.657820 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:26.657275 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-fl6dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" start-of-body= Oct 30 17:10:26.657820 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:26.657334 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fl6dw" podUID="15e78f14-6c3c-4bfa-93ea-bf58c888b372" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" Oct 30 17:10:26.678770 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:26.678736 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-q2t9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" start-of-body= Oct 30 17:10:26.679129 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:26.679097 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q2t9l" podUID="98c9aa43-9e50-4612-9965-4a20f9458deb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" Oct 30 17:10:26.729470 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:26.729424 4493 patch_prober.go:29] interesting pod/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" start-of-body= Oct 30 17:10:26.729774 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:26.729739 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g" podUID="5a4af0cd-c986-4b31-b123-545551673476" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" Oct 30 17:10:27.647242 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:27.647202 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-fl6dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" start-of-body= Oct 30 17:10:27.647662 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:27.647269 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fl6dw" podUID="15e78f14-6c3c-4bfa-93ea-bf58c888b372" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" Oct 30 17:10:27.676032 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:27.675991 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-q2t9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" start-of-body= Oct 30 17:10:27.676212 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:27.676063 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q2t9l" podUID="98c9aa43-9e50-4612-9965-4a20f9458deb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" Oct 30 17:10:27.743810 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:27.736413 4493 patch_prober.go:29] interesting pod/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" start-of-body= Oct 30 17:10:27.743810 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:27.736472 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g" podUID="5a4af0cd-c986-4b31-b123-545551673476" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" Oct 30 17:10:28.654303 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:28.654263 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-fl6dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" start-of-body= Oct 30 17:10:28.655411 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:28.654955 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fl6dw" podUID="15e78f14-6c3c-4bfa-93ea-bf58c888b372" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" Oct 30 17:10:28.676216 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:28.676185 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-q2t9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" start-of-body= Oct 30 17:10:28.676331 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:28.676235 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q2t9l" podUID="98c9aa43-9e50-4612-9965-4a20f9458deb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" Oct 30 17:10:28.737810 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:28.731937 4493 patch_prober.go:29] interesting pod/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" start-of-body= Oct 30 17:10:28.737810 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:28.732010 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g" podUID="5a4af0cd-c986-4b31-b123-545551673476" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" Oct 30 17:10:29.686812 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:29.686308 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-q2t9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" start-of-body= Oct 30 17:10:29.686812 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:29.686369 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q2t9l" podUID="98c9aa43-9e50-4612-9965-4a20f9458deb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" Oct 30 17:10:29.739815 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:29.739435 4493 patch_prober.go:29] interesting pod/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" start-of-body= Oct 30 17:10:29.739815 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:29.739491 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g" podUID="5a4af0cd-c986-4b31-b123-545551673476" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" Oct 30 17:10:30.368238 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:10:30.368162 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 17:10:30.694008 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:30.693962 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-fl6dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:10:30.694643 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:30.694612 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fl6dw" podUID="15e78f14-6c3c-4bfa-93ea-bf58c888b372" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:10:30.740514 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:30.740306 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g" Oct 30 17:10:30.741832 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:30.741808 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g" Oct 30 17:10:30.742979 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:30.742918 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g" Oct 30 17:10:31.158988 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:31.158943 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fl6dw" Oct 30 17:10:31.160820 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:31.160644 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fl6dw" Oct 30 17:10:31.162571 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:31.162547 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fl6dw" Oct 30 17:10:31.419516 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:31.419427 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q2t9l" Oct 30 17:10:31.419944 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:31.419924 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q2t9l" Oct 30 17:10:31.420885 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:31.420868 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q2t9l" Oct 30 17:10:33.344091 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:33.344060 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wh8m9"] Oct 30 17:10:33.349410 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:33.349382 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wh8m9" Oct 30 17:10:33.356198 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:33.356162 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a84d4da5-c4fd-4099-b994-4f49214532e5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-wh8m9\" (UID: \"a84d4da5-c4fd-4099-b994-4f49214532e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wh8m9" Oct 30 17:10:33.356355 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:33.356233 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4w9r\" (UniqueName: \"kubernetes.io/projected/a84d4da5-c4fd-4099-b994-4f49214532e5-kube-api-access-m4w9r\") pod \"maas-default-gateway-openshift-default-7854949c9-wh8m9\" (UID: \"a84d4da5-c4fd-4099-b994-4f49214532e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wh8m9" Oct 30 17:10:33.356355 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:33.356312 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a84d4da5-c4fd-4099-b994-4f49214532e5-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-wh8m9\" (UID: \"a84d4da5-c4fd-4099-b994-4f49214532e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wh8m9" Oct 30 17:10:33.356493 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:33.356367 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a84d4da5-c4fd-4099-b994-4f49214532e5-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-wh8m9\" (UID: \"a84d4da5-c4fd-4099-b994-4f49214532e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wh8m9" Oct 30 17:10:33.356558 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:33.356516 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a84d4da5-c4fd-4099-b994-4f49214532e5-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-wh8m9\" (UID: \"a84d4da5-c4fd-4099-b994-4f49214532e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wh8m9" Oct 30 17:10:33.356618 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:33.356562 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a84d4da5-c4fd-4099-b994-4f49214532e5-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-wh8m9\" (UID: \"a84d4da5-c4fd-4099-b994-4f49214532e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wh8m9" Oct 30 17:10:33.356680 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:33.356622 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a84d4da5-c4fd-4099-b994-4f49214532e5-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-wh8m9\" (UID: \"a84d4da5-c4fd-4099-b994-4f49214532e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wh8m9" Oct 30 17:10:33.356847 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:33.356819 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a84d4da5-c4fd-4099-b994-4f49214532e5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-wh8m9\" (UID: \"a84d4da5-c4fd-4099-b994-4f49214532e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wh8m9" Oct 30 17:10:33.357114 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:33.356885 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a84d4da5-c4fd-4099-b994-4f49214532e5-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-wh8m9\" (UID: \"a84d4da5-c4fd-4099-b994-4f49214532e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wh8m9" Oct 30 17:10:33.375670 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:33.375641 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wh8m9"] Oct 30 17:10:33.457634 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:33.457596 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a84d4da5-c4fd-4099-b994-4f49214532e5-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-wh8m9\" (UID: \"a84d4da5-c4fd-4099-b994-4f49214532e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wh8m9" Oct 30 17:10:33.457860 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:33.457641 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a84d4da5-c4fd-4099-b994-4f49214532e5-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-wh8m9\" (UID: \"a84d4da5-c4fd-4099-b994-4f49214532e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wh8m9" Oct 30 17:10:33.457860 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:33.457715 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a84d4da5-c4fd-4099-b994-4f49214532e5-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-wh8m9\" (UID: \"a84d4da5-c4fd-4099-b994-4f49214532e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wh8m9" Oct 30 17:10:33.457860 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:33.457765 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a84d4da5-c4fd-4099-b994-4f49214532e5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-wh8m9\" (UID: \"a84d4da5-c4fd-4099-b994-4f49214532e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wh8m9" Oct 30 17:10:33.457860 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:33.457836 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a84d4da5-c4fd-4099-b994-4f49214532e5-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-wh8m9\" (UID: \"a84d4da5-c4fd-4099-b994-4f49214532e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wh8m9" Oct 30 17:10:33.458142 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:33.457921 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a84d4da5-c4fd-4099-b994-4f49214532e5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-wh8m9\" (UID: \"a84d4da5-c4fd-4099-b994-4f49214532e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wh8m9" Oct 30 17:10:33.458142 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:33.457980 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4w9r\" (UniqueName: \"kubernetes.io/projected/a84d4da5-c4fd-4099-b994-4f49214532e5-kube-api-access-m4w9r\") pod \"maas-default-gateway-openshift-default-7854949c9-wh8m9\" (UID: \"a84d4da5-c4fd-4099-b994-4f49214532e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wh8m9" Oct 30 17:10:33.458142 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:33.458012 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a84d4da5-c4fd-4099-b994-4f49214532e5-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-wh8m9\" (UID: \"a84d4da5-c4fd-4099-b994-4f49214532e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wh8m9" Oct 30 17:10:33.458142 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:33.458068 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a84d4da5-c4fd-4099-b994-4f49214532e5-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-wh8m9\" (UID: \"a84d4da5-c4fd-4099-b994-4f49214532e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wh8m9" Oct 30 17:10:33.458142 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:33.458073 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a84d4da5-c4fd-4099-b994-4f49214532e5-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-wh8m9\" (UID: \"a84d4da5-c4fd-4099-b994-4f49214532e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wh8m9" Oct 30 17:10:33.458437 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:33.458402 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a84d4da5-c4fd-4099-b994-4f49214532e5-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-wh8m9\" (UID: \"a84d4da5-c4fd-4099-b994-4f49214532e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wh8m9" Oct 30 17:10:33.458681 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:33.458660 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a84d4da5-c4fd-4099-b994-4f49214532e5-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-wh8m9\" (UID: \"a84d4da5-c4fd-4099-b994-4f49214532e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wh8m9" Oct 30 17:10:33.458769 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:33.458682 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a84d4da5-c4fd-4099-b994-4f49214532e5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-wh8m9\" (UID: \"a84d4da5-c4fd-4099-b994-4f49214532e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wh8m9" Oct 30 17:10:33.458769 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:33.458718 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a84d4da5-c4fd-4099-b994-4f49214532e5-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-wh8m9\" (UID: \"a84d4da5-c4fd-4099-b994-4f49214532e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wh8m9" Oct 30 17:10:33.460405 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:33.460376 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a84d4da5-c4fd-4099-b994-4f49214532e5-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-wh8m9\" (UID: \"a84d4da5-c4fd-4099-b994-4f49214532e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wh8m9" Oct 30 17:10:33.461030 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:33.461012 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a84d4da5-c4fd-4099-b994-4f49214532e5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-wh8m9\" (UID: \"a84d4da5-c4fd-4099-b994-4f49214532e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wh8m9" Oct 30 17:10:33.480920 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:33.480889 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a84d4da5-c4fd-4099-b994-4f49214532e5-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-wh8m9\" (UID: \"a84d4da5-c4fd-4099-b994-4f49214532e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wh8m9" Oct 30 17:10:33.481099 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:33.481077 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4w9r\" (UniqueName: \"kubernetes.io/projected/a84d4da5-c4fd-4099-b994-4f49214532e5-kube-api-access-m4w9r\") pod \"maas-default-gateway-openshift-default-7854949c9-wh8m9\" (UID: \"a84d4da5-c4fd-4099-b994-4f49214532e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wh8m9" Oct 30 17:10:33.668528 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:33.668489 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wh8m9" Oct 30 17:10:34.000516 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:34.000489 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wh8m9"] Oct 30 17:10:34.001097 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:10:34.001070 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda84d4da5_c4fd_4099_b994_4f49214532e5.slice/crio-294c2846775003aa4f61e9c4580f5d41e7dd6ba5cd7ede5603a2a869db7f3262 WatchSource:0}: Error finding container 294c2846775003aa4f61e9c4580f5d41e7dd6ba5cd7ede5603a2a869db7f3262: Status 404 returned error can't find the container with id 294c2846775003aa4f61e9c4580f5d41e7dd6ba5cd7ede5603a2a869db7f3262 Oct 30 17:10:34.004397 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:34.004364 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:10:34.004477 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:34.004428 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:10:34.004477 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:34.004457 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:10:34.053368 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:34.053340 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6301a11a-w4mbz"] Oct 30 17:10:34.059291 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:34.059076 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6301a11a-w4mbz"] Oct 30 17:10:34.558119 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:34.558086 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wh8m9" event={"ID":"a84d4da5-c4fd-4099-b994-4f49214532e5","Type":"ContainerStarted","Data":"d1b71f8998a095adf8e8b79217733bb316b32e8263f238d3ddfe18445654ced7"} Oct 30 17:10:34.558546 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:34.558150 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wh8m9" event={"ID":"a84d4da5-c4fd-4099-b994-4f49214532e5","Type":"ContainerStarted","Data":"294c2846775003aa4f61e9c4580f5d41e7dd6ba5cd7ede5603a2a869db7f3262"} Oct 30 17:10:34.605969 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:34.605912 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wh8m9" podStartSLOduration=1.605883177 podStartE2EDuration="1.605883177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:10:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:10:34.602548326 +0000 UTC m=+2861.829723427" watchObservedRunningTime="2026-10-30 17:10:34.605883177 +0000 UTC m=+2861.833058276" Oct 30 17:10:34.669511 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:34.669476 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wh8m9" Oct 30 17:10:34.944435 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:34.944385 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="664a4424-09d2-463a-a9ce-be604190999f" path="/var/lib/kubelet/pods/664a4424-09d2-463a-a9ce-be604190999f/volumes" Oct 30 17:10:35.670665 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:35.670631 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-wh8m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body= Oct 30 17:10:35.671100 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:35.670682 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wh8m9" podUID="a84d4da5-c4fd-4099-b994-4f49214532e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" Oct 30 17:10:36.669328 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:36.669284 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-wh8m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body= Oct 30 17:10:36.669557 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:36.669343 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wh8m9" podUID="a84d4da5-c4fd-4099-b994-4f49214532e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" Oct 30 17:10:37.700397 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:37.700359 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-wh8m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body= Oct 30 17:10:37.701007 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:37.700970 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wh8m9" podUID="a84d4da5-c4fd-4099-b994-4f49214532e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" Oct 30 17:10:38.672295 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:38.672260 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-wh8m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body= Oct 30 17:10:38.672595 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:38.672566 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wh8m9" podUID="a84d4da5-c4fd-4099-b994-4f49214532e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" Oct 30 17:10:39.185142 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:39.185102 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" podUID="14805045-a727-41ba-8bd0-100d41f9f27d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 17:10:39.187082 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:39.187042 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" podUID="e90f43d3-f5eb-45ab-af1e-340d0ce18966" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:10:39.200179 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:39.200144 4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" podUID="80b9bd9a-30d6-45ff-b318-26fcdc608e5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:10:39.297548 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:39.297506 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" podUID="80b9bd9a-30d6-45ff-b318-26fcdc608e5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 17:10:39.448977 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:39.448876 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" podUID="0eadc5d4-98c3-4bf2-b3c2-f73fe249182f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 30 17:10:39.671515 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:39.671481 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-wh8m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body= Oct 30 17:10:39.671795 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:39.671746 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wh8m9" podUID="a84d4da5-c4fd-4099-b994-4f49214532e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" Oct 30 17:10:40.270807 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:40.270474 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" podUID="18b72648-f523-405e-b5cd-1266ff13f8f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 30 17:10:40.284812 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:40.281915 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" podUID="e43738f3-d55f-4c0a-a364-bf2ac76605c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:10:40.401625 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:40.401578 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" podUID="f3e1c8ff-fcc3-4a1f-9730-c617d57b5193" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded" Oct 30 17:10:40.673098 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:40.670256 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-wh8m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body= Oct 30 17:10:40.673098 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:40.670330 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wh8m9" podUID="a84d4da5-c4fd-4099-b994-4f49214532e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" Oct 30 17:10:41.162864 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:41.129301 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" podUID="e76172b1-659d-45bd-91c6-1a0ce4062335" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 17:10:41.235201 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:41.137948 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" podUID="b5abe703-fee0-4185-aa22-88d1db06392b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:10:41.260130 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:41.204902 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" podUID="14805045-a727-41ba-8bd0-100d41f9f27d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:10:41.373810 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:41.372991 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" podUID="80b9bd9a-30d6-45ff-b318-26fcdc608e5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 17:10:41.703141 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:41.703059 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-wh8m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body= Oct 30 17:10:41.703141 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:41.703131 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wh8m9" podUID="a84d4da5-c4fd-4099-b994-4f49214532e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" Oct 30 17:10:42.677812 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:42.669256 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-wh8m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body= Oct 30 17:10:42.677812 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:42.669328 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wh8m9" podUID="a84d4da5-c4fd-4099-b994-4f49214532e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" Oct 30 17:10:43.669500 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:43.669457 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-wh8m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body= Oct 30 17:10:43.669677 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:43.669523 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wh8m9" podUID="a84d4da5-c4fd-4099-b994-4f49214532e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" Oct 30 17:10:44.668898 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:44.668860 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-wh8m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body= Oct 30 17:10:44.669393 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:44.668923 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wh8m9" podUID="a84d4da5-c4fd-4099-b994-4f49214532e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" Oct 30 17:10:45.112613 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:45.112519 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99120b73-m4sg2"] Oct 30 17:10:45.131106 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:45.131074 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99120b73-m4sg2"] Oct 30 17:10:46.710001 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:46.709961 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-wh8m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:10:46.710660 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:46.710617 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wh8m9" podUID="a84d4da5-c4fd-4099-b994-4f49214532e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:10:46.934618 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:46.934584 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac57c348-9196-42ad-9873-37c879019e21" path="/var/lib/kubelet/pods/ac57c348-9196-42ad-9873-37c879019e21/volumes" Oct 30 17:10:47.711060 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:47.711024 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-wh8m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:10:47.711586 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:47.711562 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wh8m9" podUID="a84d4da5-c4fd-4099-b994-4f49214532e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:10:48.675814 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:48.674969 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wh8m9" Oct 30 17:10:48.760806 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:48.759919 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-wh8m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:10:48.760806 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:48.759978 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wh8m9" podUID="a84d4da5-c4fd-4099-b994-4f49214532e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:10:49.845706 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:49.845670 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-wh8m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:10:49.855998 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:49.855956 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wh8m9" podUID="a84d4da5-c4fd-4099-b994-4f49214532e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:10:50.897986 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:50.897945 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-wh8m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:10:50.898501 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:50.898022 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wh8m9" podUID="a84d4da5-c4fd-4099-b994-4f49214532e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:10:51.381289 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:51.381202 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-extmod-shr-user-e1267c/e2e-usage-sim-53fe84-6c9ffdddfd-t4swq"] Oct 30 17:10:51.388041 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:51.388016 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-e1267c/e2e-usage-sim-53fe84-6c9ffdddfd-t4swq" Oct 30 17:10:51.427810 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:51.425353 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-e1267c/e2e-usage-sim-53fe84-6c9ffdddfd-t4swq"] Oct 30 17:10:51.465914 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:51.465869 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-e1267c\"/\"default-dockercfg-gdcm9\"" Oct 30 17:10:51.466237 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:51.466215 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-e1267c\"/\"e2e-usage-sim-53fe84-cert\"" Oct 30 17:10:51.466525 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:51.466506 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-e1267c\"/\"kube-root-ca.crt\"" Oct 30 17:10:51.466754 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:51.466738 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-e1267c\"/\"openshift-service-ca.crt\"" Oct 30 17:10:51.486136 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:51.485690 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8g56\" (UniqueName: \"kubernetes.io/projected/0da18196-46f9-493f-aad4-bb3bb825363a-kube-api-access-m8g56\") pod \"e2e-usage-sim-53fe84-6c9ffdddfd-t4swq\" (UID: \"0da18196-46f9-493f-aad4-bb3bb825363a\") " pod="e2e-extmod-shr-user-e1267c/e2e-usage-sim-53fe84-6c9ffdddfd-t4swq" Oct 30 17:10:51.486136 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:51.485887 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/0da18196-46f9-493f-aad4-bb3bb825363a-tls\") pod \"e2e-usage-sim-53fe84-6c9ffdddfd-t4swq\" (UID: \"0da18196-46f9-493f-aad4-bb3bb825363a\") " pod="e2e-extmod-shr-user-e1267c/e2e-usage-sim-53fe84-6c9ffdddfd-t4swq" Oct 30 17:10:51.589529 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:51.589475 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8g56\" (UniqueName: \"kubernetes.io/projected/0da18196-46f9-493f-aad4-bb3bb825363a-kube-api-access-m8g56\") pod \"e2e-usage-sim-53fe84-6c9ffdddfd-t4swq\" (UID: \"0da18196-46f9-493f-aad4-bb3bb825363a\") " pod="e2e-extmod-shr-user-e1267c/e2e-usage-sim-53fe84-6c9ffdddfd-t4swq" Oct 30 17:10:51.589964 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:51.589939 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/0da18196-46f9-493f-aad4-bb3bb825363a-tls\") pod \"e2e-usage-sim-53fe84-6c9ffdddfd-t4swq\" (UID: \"0da18196-46f9-493f-aad4-bb3bb825363a\") " pod="e2e-extmod-shr-user-e1267c/e2e-usage-sim-53fe84-6c9ffdddfd-t4swq" Oct 30 17:10:51.596738 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:51.596710 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/0da18196-46f9-493f-aad4-bb3bb825363a-tls\") pod \"e2e-usage-sim-53fe84-6c9ffdddfd-t4swq\" (UID: \"0da18196-46f9-493f-aad4-bb3bb825363a\") " pod="e2e-extmod-shr-user-e1267c/e2e-usage-sim-53fe84-6c9ffdddfd-t4swq" Oct 30 17:10:51.634866 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:51.634770 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8g56\" (UniqueName: \"kubernetes.io/projected/0da18196-46f9-493f-aad4-bb3bb825363a-kube-api-access-m8g56\") pod \"e2e-usage-sim-53fe84-6c9ffdddfd-t4swq\" (UID: \"0da18196-46f9-493f-aad4-bb3bb825363a\") " pod="e2e-extmod-shr-user-e1267c/e2e-usage-sim-53fe84-6c9ffdddfd-t4swq" Oct 30 17:10:51.790118 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:51.790072 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-e1267c/e2e-usage-sim-53fe84-6c9ffdddfd-t4swq" Oct 30 17:10:51.898734 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:51.898337 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-wh8m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:10:51.898734 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:51.898413 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wh8m9" podUID="a84d4da5-c4fd-4099-b994-4f49214532e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:10:52.251353 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:52.251250 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-e1267c/e2e-usage-sim-53fe84-6c9ffdddfd-t4swq"] Oct 30 17:10:52.292993 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:10:52.292963 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0da18196_46f9_493f_aad4_bb3bb825363a.slice/crio-404d6f7dd40317435162d2159c79cd89f73bcfb3965415e64ec2b3a30a79ef3f WatchSource:0}: Error finding container 404d6f7dd40317435162d2159c79cd89f73bcfb3965415e64ec2b3a30a79ef3f: Status 404 returned error can't find the container with id 404d6f7dd40317435162d2159c79cd89f73bcfb3965415e64ec2b3a30a79ef3f Oct 30 17:10:52.598877 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:52.592976 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-e1267c/e2e-usage-sim-53fe84-6c9ffdddfd-t4swq" event={"ID":"0da18196-46f9-493f-aad4-bb3bb825363a","Type":"ContainerStarted","Data":"404d6f7dd40317435162d2159c79cd89f73bcfb3965415e64ec2b3a30a79ef3f"} Oct 30 17:10:52.940807 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:52.940315 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-wh8m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:10:52.940807 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:52.940375 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wh8m9" podUID="a84d4da5-c4fd-4099-b994-4f49214532e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:10:53.624808 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:53.619431 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-e1267c/e2e-usage-sim-53fe84-6c9ffdddfd-t4swq" event={"ID":"0da18196-46f9-493f-aad4-bb3bb825363a","Type":"ContainerStarted","Data":"05a4ecef3af0bcbcbc90f09eef6a47b8be853a6c32acab6a1447430dbf06d346"} Oct 30 17:10:53.624808 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:53.620751 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-extmod-shr-user-e1267c/e2e-usage-sim-53fe84-6c9ffdddfd-t4swq" Oct 30 17:10:53.681515 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:53.681456 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-extmod-shr-user-e1267c/e2e-usage-sim-53fe84-6c9ffdddfd-t4swq" podStartSLOduration=2.6814417600000002 podStartE2EDuration="2.68144176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:10:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:10:53.677658363 +0000 UTC m=+2880.904833465" watchObservedRunningTime="2026-10-30 17:10:53.68144176 +0000 UTC m=+2880.908616903" Oct 30 17:10:53.682427 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:53.682397 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-extmod-shr-user-e1267c/e2e-usage-sim-53fe84-6c9ffdddfd-t4swq" Oct 30 17:10:54.023210 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:54.021057 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-wh8m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:10:54.023210 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:54.021131 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wh8m9" podUID="a84d4da5-c4fd-4099-b994-4f49214532e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:10:54.138885 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:54.138859 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wh8m9" Oct 30 17:10:54.146721 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:54.146698 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wh8m9" Oct 30 17:10:59.049594 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:59.049553 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71b91dc5-48xgq"] Oct 30 17:10:59.057711 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:10:59.057685 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71b91dc5-48xgq"] Oct 30 17:11:00.351471 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:11:00.351424 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 17:11:00.936177 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:00.936143 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32ae81e0-de61-47ec-809c-a1ec55b181a6" path="/var/lib/kubelet/pods/32ae81e0-de61-47ec-809c-a1ec55b181a6/volumes" Oct 30 17:11:02.348094 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:02.348055 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg"] Oct 30 17:11:02.355805 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:02.355766 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg" Oct 30 17:11:02.391969 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:02.391890 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg"] Oct 30 17:11:02.442889 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:02.442849 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-6acc78-openshift-default-dockercfg-sdf49\"" Oct 30 17:11:02.517864 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:02.517809 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5146890-4f44-4e48-9147-dc215b8195d9-istiod-ca-cert\") pod \"e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg\" (UID: \"c5146890-4f44-4e48-9147-dc215b8195d9\") " pod="openshift-ingress/e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg" Oct 30 17:11:02.518065 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:02.517951 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdznp\" (UniqueName: \"kubernetes.io/projected/c5146890-4f44-4e48-9147-dc215b8195d9-kube-api-access-jdznp\") pod \"e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg\" (UID: \"c5146890-4f44-4e48-9147-dc215b8195d9\") " pod="openshift-ingress/e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg" Oct 30 17:11:02.518065 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:02.518001 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5146890-4f44-4e48-9147-dc215b8195d9-istio-data\") pod \"e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg\" (UID: \"c5146890-4f44-4e48-9147-dc215b8195d9\") " pod="openshift-ingress/e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg" Oct 30 17:11:02.518065 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:02.518058 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5146890-4f44-4e48-9147-dc215b8195d9-credential-socket\") pod \"e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg\" (UID: \"c5146890-4f44-4e48-9147-dc215b8195d9\") " pod="openshift-ingress/e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg" Oct 30 17:11:02.518219 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:02.518174 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5146890-4f44-4e48-9147-dc215b8195d9-istio-token\") pod \"e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg\" (UID: \"c5146890-4f44-4e48-9147-dc215b8195d9\") " pod="openshift-ingress/e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg" Oct 30 17:11:02.518276 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:02.518226 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5146890-4f44-4e48-9147-dc215b8195d9-workload-socket\") pod \"e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg\" (UID: \"c5146890-4f44-4e48-9147-dc215b8195d9\") " pod="openshift-ingress/e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg" Oct 30 17:11:02.518317 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:02.518298 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5146890-4f44-4e48-9147-dc215b8195d9-workload-certs\") pod \"e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg\" (UID: \"c5146890-4f44-4e48-9147-dc215b8195d9\") " pod="openshift-ingress/e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg" Oct 30 17:11:02.518450 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:02.518430 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5146890-4f44-4e48-9147-dc215b8195d9-istio-envoy\") pod \"e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg\" (UID: \"c5146890-4f44-4e48-9147-dc215b8195d9\") " pod="openshift-ingress/e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg" Oct 30 17:11:02.518563 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:02.518543 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5146890-4f44-4e48-9147-dc215b8195d9-istio-podinfo\") pod \"e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg\" (UID: \"c5146890-4f44-4e48-9147-dc215b8195d9\") " pod="openshift-ingress/e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg" Oct 30 17:11:02.622423 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:02.620453 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5146890-4f44-4e48-9147-dc215b8195d9-workload-socket\") pod \"e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg\" (UID: \"c5146890-4f44-4e48-9147-dc215b8195d9\") " pod="openshift-ingress/e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg" Oct 30 17:11:02.622423 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:02.620512 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5146890-4f44-4e48-9147-dc215b8195d9-workload-certs\") pod \"e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg\" (UID: \"c5146890-4f44-4e48-9147-dc215b8195d9\") " pod="openshift-ingress/e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg" Oct 30 17:11:02.622423 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:02.620585 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5146890-4f44-4e48-9147-dc215b8195d9-istio-envoy\") pod \"e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg\" (UID: \"c5146890-4f44-4e48-9147-dc215b8195d9\") " pod="openshift-ingress/e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg" Oct 30 17:11:02.622423 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:02.620661 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5146890-4f44-4e48-9147-dc215b8195d9-istio-podinfo\") pod \"e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg\" (UID: \"c5146890-4f44-4e48-9147-dc215b8195d9\") " pod="openshift-ingress/e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg" Oct 30 17:11:02.622423 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:02.620778 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5146890-4f44-4e48-9147-dc215b8195d9-istiod-ca-cert\") pod \"e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg\" (UID: \"c5146890-4f44-4e48-9147-dc215b8195d9\") " pod="openshift-ingress/e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg" Oct 30 17:11:02.622423 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:02.620847 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdznp\" (UniqueName: \"kubernetes.io/projected/c5146890-4f44-4e48-9147-dc215b8195d9-kube-api-access-jdznp\") pod \"e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg\" (UID: \"c5146890-4f44-4e48-9147-dc215b8195d9\") " pod="openshift-ingress/e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg" Oct 30 17:11:02.622423 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:02.620873 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5146890-4f44-4e48-9147-dc215b8195d9-istio-data\") pod \"e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg\" (UID: \"c5146890-4f44-4e48-9147-dc215b8195d9\") " pod="openshift-ingress/e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg" Oct 30 17:11:02.622423 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:02.620930 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5146890-4f44-4e48-9147-dc215b8195d9-credential-socket\") pod \"e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg\" (UID: \"c5146890-4f44-4e48-9147-dc215b8195d9\") " pod="openshift-ingress/e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg" Oct 30 17:11:02.622423 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:02.620989 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5146890-4f44-4e48-9147-dc215b8195d9-istio-token\") pod \"e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg\" (UID: \"c5146890-4f44-4e48-9147-dc215b8195d9\") " pod="openshift-ingress/e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg" Oct 30 17:11:02.622423 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:02.621816 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5146890-4f44-4e48-9147-dc215b8195d9-workload-socket\") pod \"e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg\" (UID: \"c5146890-4f44-4e48-9147-dc215b8195d9\") " pod="openshift-ingress/e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg" Oct 30 17:11:02.622423 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:02.622080 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5146890-4f44-4e48-9147-dc215b8195d9-workload-certs\") pod \"e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg\" (UID: \"c5146890-4f44-4e48-9147-dc215b8195d9\") " pod="openshift-ingress/e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg" Oct 30 17:11:02.624898 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:02.624403 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5146890-4f44-4e48-9147-dc215b8195d9-credential-socket\") pod \"e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg\" (UID: \"c5146890-4f44-4e48-9147-dc215b8195d9\") " pod="openshift-ingress/e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg" Oct 30 17:11:02.624898 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:02.624844 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5146890-4f44-4e48-9147-dc215b8195d9-istiod-ca-cert\") pod \"e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg\" (UID: \"c5146890-4f44-4e48-9147-dc215b8195d9\") " pod="openshift-ingress/e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg" Oct 30 17:11:02.625264 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:02.625245 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5146890-4f44-4e48-9147-dc215b8195d9-istio-data\") pod \"e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg\" (UID: \"c5146890-4f44-4e48-9147-dc215b8195d9\") " pod="openshift-ingress/e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg" Oct 30 17:11:02.629639 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:02.629265 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5146890-4f44-4e48-9147-dc215b8195d9-istio-envoy\") pod \"e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg\" (UID: \"c5146890-4f44-4e48-9147-dc215b8195d9\") " pod="openshift-ingress/e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg" Oct 30 17:11:02.631139 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:02.631111 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5146890-4f44-4e48-9147-dc215b8195d9-istio-podinfo\") pod \"e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg\" (UID: \"c5146890-4f44-4e48-9147-dc215b8195d9\") " pod="openshift-ingress/e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg" Oct 30 17:11:02.690837 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:02.687053 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdznp\" (UniqueName: \"kubernetes.io/projected/c5146890-4f44-4e48-9147-dc215b8195d9-kube-api-access-jdznp\") pod \"e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg\" (UID: \"c5146890-4f44-4e48-9147-dc215b8195d9\") " pod="openshift-ingress/e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg" Oct 30 17:11:02.698818 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:02.698380 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5146890-4f44-4e48-9147-dc215b8195d9-istio-token\") pod \"e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg\" (UID: \"c5146890-4f44-4e48-9147-dc215b8195d9\") " pod="openshift-ingress/e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg" Oct 30 17:11:02.983964 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:02.983928 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg" Oct 30 17:11:03.040963 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:03.040922 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b87b52b0-kqjxx"] Oct 30 17:11:03.048146 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:03.048105 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b87b52b0-kqjxx"] Oct 30 17:11:03.340471 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:03.340442 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg"] Oct 30 17:11:03.341106 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:11:03.341063 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5146890_4f44_4e48_9147_dc215b8195d9.slice/crio-19a1bbcda04783c7159eb2a3bea75cd2602c8d274034d3507fc04f0d760ebe0b WatchSource:0}: Error finding container 19a1bbcda04783c7159eb2a3bea75cd2602c8d274034d3507fc04f0d760ebe0b: Status 404 returned error can't find the container with id 19a1bbcda04783c7159eb2a3bea75cd2602c8d274034d3507fc04f0d760ebe0b Oct 30 17:11:03.344412 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:03.344378 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:11:03.344531 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:03.344461 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:11:03.344631 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:03.344529 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:11:03.436613 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:03.436569 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-xjbtt"] Oct 30 17:11:03.441960 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:03.441941 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-xjbtt" Oct 30 17:11:03.471824 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:03.470662 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9860f52-b69d-4de3-8999-eff8e90620dc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-xjbtt\" (UID: \"b9860f52-b69d-4de3-8999-eff8e90620dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-xjbtt" Oct 30 17:11:03.471824 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:03.470715 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9860f52-b69d-4de3-8999-eff8e90620dc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-xjbtt\" (UID: \"b9860f52-b69d-4de3-8999-eff8e90620dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-xjbtt" Oct 30 17:11:03.471824 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:03.470819 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9860f52-b69d-4de3-8999-eff8e90620dc-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-xjbtt\" (UID: \"b9860f52-b69d-4de3-8999-eff8e90620dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-xjbtt" Oct 30 17:11:03.471824 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:03.470863 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9860f52-b69d-4de3-8999-eff8e90620dc-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-xjbtt\" (UID: \"b9860f52-b69d-4de3-8999-eff8e90620dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-xjbtt" Oct 30 17:11:03.471824 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:03.470924 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9860f52-b69d-4de3-8999-eff8e90620dc-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-xjbtt\" (UID: \"b9860f52-b69d-4de3-8999-eff8e90620dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-xjbtt" Oct 30 17:11:03.471824 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:03.470949 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9860f52-b69d-4de3-8999-eff8e90620dc-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-xjbtt\" (UID: \"b9860f52-b69d-4de3-8999-eff8e90620dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-xjbtt" Oct 30 17:11:03.471824 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:03.471057 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg2nq\" (UniqueName: \"kubernetes.io/projected/b9860f52-b69d-4de3-8999-eff8e90620dc-kube-api-access-pg2nq\") pod \"maas-default-gateway-openshift-default-7854949c9-xjbtt\" (UID: \"b9860f52-b69d-4de3-8999-eff8e90620dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-xjbtt" Oct 30 17:11:03.471824 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:03.471086 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9860f52-b69d-4de3-8999-eff8e90620dc-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-xjbtt\" (UID: \"b9860f52-b69d-4de3-8999-eff8e90620dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-xjbtt" Oct 30 17:11:03.471824 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:03.471115 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9860f52-b69d-4de3-8999-eff8e90620dc-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-xjbtt\" (UID: \"b9860f52-b69d-4de3-8999-eff8e90620dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-xjbtt" Oct 30 17:11:03.477811 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:03.477503 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-xjbtt"] Oct 30 17:11:03.587265 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:03.586857 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9860f52-b69d-4de3-8999-eff8e90620dc-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-xjbtt\" (UID: \"b9860f52-b69d-4de3-8999-eff8e90620dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-xjbtt" Oct 30 17:11:03.587265 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:03.586894 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9860f52-b69d-4de3-8999-eff8e90620dc-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-xjbtt\" (UID: \"b9860f52-b69d-4de3-8999-eff8e90620dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-xjbtt" Oct 30 17:11:03.587265 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:03.586983 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg2nq\" (UniqueName: \"kubernetes.io/projected/b9860f52-b69d-4de3-8999-eff8e90620dc-kube-api-access-pg2nq\") pod \"maas-default-gateway-openshift-default-7854949c9-xjbtt\" (UID: \"b9860f52-b69d-4de3-8999-eff8e90620dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-xjbtt" Oct 30 17:11:03.587265 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:03.587001 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9860f52-b69d-4de3-8999-eff8e90620dc-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-xjbtt\" (UID: \"b9860f52-b69d-4de3-8999-eff8e90620dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-xjbtt" Oct 30 17:11:03.587265 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:03.587020 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9860f52-b69d-4de3-8999-eff8e90620dc-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-xjbtt\" (UID: \"b9860f52-b69d-4de3-8999-eff8e90620dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-xjbtt" Oct 30 17:11:03.587265 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:03.587058 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9860f52-b69d-4de3-8999-eff8e90620dc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-xjbtt\" (UID: \"b9860f52-b69d-4de3-8999-eff8e90620dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-xjbtt" Oct 30 17:11:03.587265 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:03.587086 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9860f52-b69d-4de3-8999-eff8e90620dc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-xjbtt\" (UID: \"b9860f52-b69d-4de3-8999-eff8e90620dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-xjbtt" Oct 30 17:11:03.587265 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:03.587156 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9860f52-b69d-4de3-8999-eff8e90620dc-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-xjbtt\" (UID: \"b9860f52-b69d-4de3-8999-eff8e90620dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-xjbtt" Oct 30 17:11:03.587265 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:03.587187 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9860f52-b69d-4de3-8999-eff8e90620dc-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-xjbtt\" (UID: \"b9860f52-b69d-4de3-8999-eff8e90620dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-xjbtt" Oct 30 17:11:03.587974 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:03.587535 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9860f52-b69d-4de3-8999-eff8e90620dc-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-xjbtt\" (UID: \"b9860f52-b69d-4de3-8999-eff8e90620dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-xjbtt" Oct 30 17:11:03.587974 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:03.587693 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9860f52-b69d-4de3-8999-eff8e90620dc-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-xjbtt\" (UID: \"b9860f52-b69d-4de3-8999-eff8e90620dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-xjbtt" Oct 30 17:11:03.588518 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:03.588462 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9860f52-b69d-4de3-8999-eff8e90620dc-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-xjbtt\" (UID: \"b9860f52-b69d-4de3-8999-eff8e90620dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-xjbtt" Oct 30 17:11:03.588518 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:03.588504 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9860f52-b69d-4de3-8999-eff8e90620dc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-xjbtt\" (UID: \"b9860f52-b69d-4de3-8999-eff8e90620dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-xjbtt" Oct 30 17:11:03.588759 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:03.588733 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9860f52-b69d-4de3-8999-eff8e90620dc-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-xjbtt\" (UID: \"b9860f52-b69d-4de3-8999-eff8e90620dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-xjbtt" Oct 30 17:11:03.591466 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:03.591384 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9860f52-b69d-4de3-8999-eff8e90620dc-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-xjbtt\" (UID: \"b9860f52-b69d-4de3-8999-eff8e90620dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-xjbtt" Oct 30 17:11:03.591466 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:03.591411 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9860f52-b69d-4de3-8999-eff8e90620dc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-xjbtt\" (UID: \"b9860f52-b69d-4de3-8999-eff8e90620dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-xjbtt" Oct 30 17:11:03.625270 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:03.625236 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9860f52-b69d-4de3-8999-eff8e90620dc-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-xjbtt\" (UID: \"b9860f52-b69d-4de3-8999-eff8e90620dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-xjbtt" Oct 30 17:11:03.626873 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:03.626848 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg2nq\" (UniqueName: \"kubernetes.io/projected/b9860f52-b69d-4de3-8999-eff8e90620dc-kube-api-access-pg2nq\") pod \"maas-default-gateway-openshift-default-7854949c9-xjbtt\" (UID: \"b9860f52-b69d-4de3-8999-eff8e90620dc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-xjbtt" Oct 30 17:11:03.764608 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:03.764568 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg" event={"ID":"c5146890-4f44-4e48-9147-dc215b8195d9","Type":"ContainerStarted","Data":"f3d935c69434a834596a8b60e289ddca19d0a6132e8ffc572e33f8952aa1ae38"} Oct 30 17:11:03.764819 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:03.764625 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg" event={"ID":"c5146890-4f44-4e48-9147-dc215b8195d9","Type":"ContainerStarted","Data":"19a1bbcda04783c7159eb2a3bea75cd2602c8d274034d3507fc04f0d760ebe0b"} Oct 30 17:11:03.790103 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:03.790075 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-xjbtt" Oct 30 17:11:03.815503 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:03.814203 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg" podStartSLOduration=1.81418501 podStartE2EDuration="1.81418501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:11:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:11:03.808821366 +0000 UTC m=+2891.035996490" watchObservedRunningTime="2026-10-30 17:11:03.81418501 +0000 UTC m=+2891.041360112" Oct 30 17:11:03.988815 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:03.987041 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg" Oct 30 17:11:04.137654 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:04.137613 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-xjbtt"] Oct 30 17:11:04.146846 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:04.145653 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:11:04.146846 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:04.145737 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:11:04.146846 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:04.145821 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:11:04.775434 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:04.775361 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-xjbtt" event={"ID":"b9860f52-b69d-4de3-8999-eff8e90620dc","Type":"ContainerStarted","Data":"301d8812d0adec42dad331d678a4dec9af00b159c618daeff1f7973d18d24914"} Oct 30 17:11:04.776022 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:04.775496 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-xjbtt" event={"ID":"b9860f52-b69d-4de3-8999-eff8e90620dc","Type":"ContainerStarted","Data":"5159e35db94133641e5f8db4482f36ceea6c6966a1d175784d4bbc9513b20833"} Oct 30 17:11:04.793426 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:04.790752 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-xjbtt" Oct 30 17:11:04.838461 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:04.838396 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-xjbtt" podStartSLOduration=1.8383777540000001 podStartE2EDuration="1.838377754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:11:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:11:04.835153802 +0000 UTC m=+2892.062328917" watchObservedRunningTime="2026-10-30 17:11:04.838377754 +0000 UTC m=+2892.065552854" Oct 30 17:11:04.939280 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:04.939250 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8d4ac15-01f8-42c9-83aa-dbe69955b0d8" path="/var/lib/kubelet/pods/b8d4ac15-01f8-42c9-83aa-dbe69955b0d8/volumes" Oct 30 17:11:04.991702 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:04.991671 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg" Oct 30 17:11:05.485560 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:05.483537 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl"] Oct 30 17:11:05.492887 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:05.490141 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl" Oct 30 17:11:05.505872 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:05.505845 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-3106d6-openshift-default-dockercfg-hpqnn\"" Oct 30 17:11:05.546810 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:05.546525 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl"] Oct 30 17:11:05.555832 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:05.553620 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35f0aae9-952b-4239-ae42-f9dfb3dd94be-istio-token\") pod \"e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl\" (UID: \"35f0aae9-952b-4239-ae42-f9dfb3dd94be\") " pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl" Oct 30 17:11:05.555832 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:05.553693 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35f0aae9-952b-4239-ae42-f9dfb3dd94be-istio-data\") pod \"e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl\" (UID: \"35f0aae9-952b-4239-ae42-f9dfb3dd94be\") " pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl" Oct 30 17:11:05.555832 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:05.553734 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35f0aae9-952b-4239-ae42-f9dfb3dd94be-istiod-ca-cert\") pod \"e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl\" (UID: \"35f0aae9-952b-4239-ae42-f9dfb3dd94be\") " pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl" Oct 30 17:11:05.555832 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:05.553823 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f0aae9-952b-4239-ae42-f9dfb3dd94be-workload-socket\") pod \"e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl\" (UID: \"35f0aae9-952b-4239-ae42-f9dfb3dd94be\") " pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl" Oct 30 17:11:05.555832 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:05.553924 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f0aae9-952b-4239-ae42-f9dfb3dd94be-credential-socket\") pod \"e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl\" (UID: \"35f0aae9-952b-4239-ae42-f9dfb3dd94be\") " pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl" Oct 30 17:11:05.555832 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:05.554035 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpztg\" (UniqueName: \"kubernetes.io/projected/35f0aae9-952b-4239-ae42-f9dfb3dd94be-kube-api-access-rpztg\") pod \"e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl\" (UID: \"35f0aae9-952b-4239-ae42-f9dfb3dd94be\") " pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl" Oct 30 17:11:05.555832 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:05.554074 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35f0aae9-952b-4239-ae42-f9dfb3dd94be-workload-certs\") pod \"e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl\" (UID: \"35f0aae9-952b-4239-ae42-f9dfb3dd94be\") " pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl" Oct 30 17:11:05.555832 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:05.554206 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35f0aae9-952b-4239-ae42-f9dfb3dd94be-istio-envoy\") pod \"e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl\" (UID: \"35f0aae9-952b-4239-ae42-f9dfb3dd94be\") " pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl" Oct 30 17:11:05.555832 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:05.554251 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35f0aae9-952b-4239-ae42-f9dfb3dd94be-istio-podinfo\") pod \"e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl\" (UID: \"35f0aae9-952b-4239-ae42-f9dfb3dd94be\") " pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl" Oct 30 17:11:05.585211 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:05.585177 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb"] Oct 30 17:11:05.591287 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:05.591262 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb" Oct 30 17:11:05.635808 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:05.632996 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-cb6251-openshift-default-dockercfg-5cbjg\"" Oct 30 17:11:05.650880 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:05.650843 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb"] Oct 30 17:11:05.656109 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:05.656068 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35f0aae9-952b-4239-ae42-f9dfb3dd94be-istio-data\") pod \"e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl\" (UID: \"35f0aae9-952b-4239-ae42-f9dfb3dd94be\") " pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl" Oct 30 17:11:05.656251 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:05.656160 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35f0aae9-952b-4239-ae42-f9dfb3dd94be-istiod-ca-cert\") pod \"e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl\" (UID: \"35f0aae9-952b-4239-ae42-f9dfb3dd94be\") " pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl" Oct 30 17:11:05.656251 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:05.656195 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84f45fe0-793d-4f0d-98d5-f78b9a287c5a-istio-data\") pod \"e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb\" (UID: \"84f45fe0-793d-4f0d-98d5-f78b9a287c5a\") " pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb" Oct 30 17:11:05.656374 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:05.656270 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84f45fe0-793d-4f0d-98d5-f78b9a287c5a-istio-token\") pod \"e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb\" (UID: \"84f45fe0-793d-4f0d-98d5-f78b9a287c5a\") " pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb" Oct 30 17:11:05.656374 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:05.656335 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f0aae9-952b-4239-ae42-f9dfb3dd94be-workload-socket\") pod \"e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl\" (UID: \"35f0aae9-952b-4239-ae42-f9dfb3dd94be\") " pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl" Oct 30 17:11:05.656491 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:05.656398 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84f45fe0-793d-4f0d-98d5-f78b9a287c5a-istiod-ca-cert\") pod \"e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb\" (UID: \"84f45fe0-793d-4f0d-98d5-f78b9a287c5a\") " pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb" Oct 30 17:11:05.656491 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:05.656427 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f0aae9-952b-4239-ae42-f9dfb3dd94be-credential-socket\") pod \"e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl\" (UID: \"35f0aae9-952b-4239-ae42-f9dfb3dd94be\") " pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl" Oct 30 17:11:05.656491 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:05.656467 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84f45fe0-793d-4f0d-98d5-f78b9a287c5a-istio-podinfo\") pod \"e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb\" (UID: \"84f45fe0-793d-4f0d-98d5-f78b9a287c5a\") " pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb" Oct 30 17:11:05.656491 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:05.656484 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84f45fe0-793d-4f0d-98d5-f78b9a287c5a-workload-certs\") pod \"e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb\" (UID: \"84f45fe0-793d-4f0d-98d5-f78b9a287c5a\") " pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb" Oct 30 17:11:05.656723 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:05.656512 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84f45fe0-793d-4f0d-98d5-f78b9a287c5a-istio-envoy\") pod \"e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb\" (UID: \"84f45fe0-793d-4f0d-98d5-f78b9a287c5a\") " pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb" Oct 30 17:11:05.656723 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:05.656542 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84f45fe0-793d-4f0d-98d5-f78b9a287c5a-workload-socket\") pod \"e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb\" (UID: \"84f45fe0-793d-4f0d-98d5-f78b9a287c5a\") " pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb" Oct 30 17:11:05.656723 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:05.656575 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpztg\" (UniqueName: \"kubernetes.io/projected/35f0aae9-952b-4239-ae42-f9dfb3dd94be-kube-api-access-rpztg\") pod \"e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl\" (UID: \"35f0aae9-952b-4239-ae42-f9dfb3dd94be\") " pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl" Oct 30 17:11:05.656723 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:05.656599 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35f0aae9-952b-4239-ae42-f9dfb3dd94be-workload-certs\") pod \"e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl\" (UID: \"35f0aae9-952b-4239-ae42-f9dfb3dd94be\") " pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl" Oct 30 17:11:05.656723 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:05.656686 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35f0aae9-952b-4239-ae42-f9dfb3dd94be-istio-envoy\") pod \"e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl\" (UID: \"35f0aae9-952b-4239-ae42-f9dfb3dd94be\") " pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl" Oct 30 17:11:05.656723 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:05.656716 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54pw5\" (UniqueName: \"kubernetes.io/projected/84f45fe0-793d-4f0d-98d5-f78b9a287c5a-kube-api-access-54pw5\") pod \"e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb\" (UID: \"84f45fe0-793d-4f0d-98d5-f78b9a287c5a\") " pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb" Oct 30 17:11:05.657074 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:05.656737 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35f0aae9-952b-4239-ae42-f9dfb3dd94be-istio-podinfo\") pod \"e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl\" (UID: \"35f0aae9-952b-4239-ae42-f9dfb3dd94be\") " pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl" Oct 30 17:11:05.657074 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:05.656743 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f0aae9-952b-4239-ae42-f9dfb3dd94be-credential-socket\") pod \"e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl\" (UID: \"35f0aae9-952b-4239-ae42-f9dfb3dd94be\") " pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl" Oct 30 17:11:05.657074 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:05.656754 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84f45fe0-793d-4f0d-98d5-f78b9a287c5a-credential-socket\") pod \"e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb\" (UID: \"84f45fe0-793d-4f0d-98d5-f78b9a287c5a\") " pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb" Oct 30 17:11:05.657074 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:05.656809 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35f0aae9-952b-4239-ae42-f9dfb3dd94be-istio-token\") pod \"e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl\" (UID: \"35f0aae9-952b-4239-ae42-f9dfb3dd94be\") " pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl" Oct 30 17:11:05.657074 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:05.656511 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35f0aae9-952b-4239-ae42-f9dfb3dd94be-istio-data\") pod \"e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl\" (UID: \"35f0aae9-952b-4239-ae42-f9dfb3dd94be\") " pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl" Oct 30 17:11:05.657349 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:05.657143 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f0aae9-952b-4239-ae42-f9dfb3dd94be-workload-socket\") pod \"e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl\" (UID: \"35f0aae9-952b-4239-ae42-f9dfb3dd94be\") " pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl" Oct 30 17:11:05.657410 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:05.657362 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35f0aae9-952b-4239-ae42-f9dfb3dd94be-workload-certs\") pod \"e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl\" (UID: \"35f0aae9-952b-4239-ae42-f9dfb3dd94be\") " pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl" Oct 30 17:11:05.658355 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:05.658330 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35f0aae9-952b-4239-ae42-f9dfb3dd94be-istiod-ca-cert\") pod \"e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl\" (UID: \"35f0aae9-952b-4239-ae42-f9dfb3dd94be\") " pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl" Oct 30 17:11:05.660677 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:05.660404 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35f0aae9-952b-4239-ae42-f9dfb3dd94be-istio-podinfo\") pod \"e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl\" (UID: \"35f0aae9-952b-4239-ae42-f9dfb3dd94be\") " pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl" Oct 30 17:11:05.660893 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:05.660869 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35f0aae9-952b-4239-ae42-f9dfb3dd94be-istio-envoy\") pod \"e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl\" (UID: \"35f0aae9-952b-4239-ae42-f9dfb3dd94be\") " pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl" Oct 30 17:11:05.680697 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:05.680616 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35f0aae9-952b-4239-ae42-f9dfb3dd94be-istio-token\") pod \"e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl\" (UID: \"35f0aae9-952b-4239-ae42-f9dfb3dd94be\") " pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl" Oct 30 17:11:05.680697 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:05.680634 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpztg\" (UniqueName: \"kubernetes.io/projected/35f0aae9-952b-4239-ae42-f9dfb3dd94be-kube-api-access-rpztg\") pod \"e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl\" (UID: \"35f0aae9-952b-4239-ae42-f9dfb3dd94be\") " pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl" Oct 30 17:11:05.758863 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:05.758737 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84f45fe0-793d-4f0d-98d5-f78b9a287c5a-istio-data\") pod \"e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb\" (UID: \"84f45fe0-793d-4f0d-98d5-f78b9a287c5a\") " pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb" Oct 30 17:11:05.758863 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:05.758846 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84f45fe0-793d-4f0d-98d5-f78b9a287c5a-istio-token\") pod \"e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb\" (UID: \"84f45fe0-793d-4f0d-98d5-f78b9a287c5a\") " pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb" Oct 30 17:11:05.759105 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:05.758927 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84f45fe0-793d-4f0d-98d5-f78b9a287c5a-istiod-ca-cert\") pod \"e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb\" (UID: \"84f45fe0-793d-4f0d-98d5-f78b9a287c5a\") " pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb" Oct 30 17:11:05.759105 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:05.758985 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84f45fe0-793d-4f0d-98d5-f78b9a287c5a-istio-podinfo\") pod \"e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb\" (UID: \"84f45fe0-793d-4f0d-98d5-f78b9a287c5a\") " pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb" Oct 30 17:11:05.759105 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:05.759009 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84f45fe0-793d-4f0d-98d5-f78b9a287c5a-workload-certs\") pod \"e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb\" (UID: \"84f45fe0-793d-4f0d-98d5-f78b9a287c5a\") " pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb" Oct 30 17:11:05.759105 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:05.759047 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84f45fe0-793d-4f0d-98d5-f78b9a287c5a-istio-envoy\") pod \"e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb\" (UID: \"84f45fe0-793d-4f0d-98d5-f78b9a287c5a\") " pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb" Oct 30 17:11:05.759105 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:05.759095 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84f45fe0-793d-4f0d-98d5-f78b9a287c5a-workload-socket\") pod \"e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb\" (UID: \"84f45fe0-793d-4f0d-98d5-f78b9a287c5a\") " pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb" Oct 30 17:11:05.759366 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:05.759263 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84f45fe0-793d-4f0d-98d5-f78b9a287c5a-istio-data\") pod \"e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb\" (UID: \"84f45fe0-793d-4f0d-98d5-f78b9a287c5a\") " pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb" Oct 30 17:11:05.759366 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:05.759281 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54pw5\" (UniqueName: \"kubernetes.io/projected/84f45fe0-793d-4f0d-98d5-f78b9a287c5a-kube-api-access-54pw5\") pod \"e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb\" (UID: \"84f45fe0-793d-4f0d-98d5-f78b9a287c5a\") " pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb" Oct 30 17:11:05.759366 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:05.759341 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84f45fe0-793d-4f0d-98d5-f78b9a287c5a-credential-socket\") pod \"e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb\" (UID: \"84f45fe0-793d-4f0d-98d5-f78b9a287c5a\") " pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb" Oct 30 17:11:05.759521 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:05.759478 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84f45fe0-793d-4f0d-98d5-f78b9a287c5a-workload-certs\") pod \"e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb\" (UID: \"84f45fe0-793d-4f0d-98d5-f78b9a287c5a\") " pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb" Oct 30 17:11:05.759988 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:05.759677 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84f45fe0-793d-4f0d-98d5-f78b9a287c5a-credential-socket\") pod \"e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb\" (UID: \"84f45fe0-793d-4f0d-98d5-f78b9a287c5a\") " pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb" Oct 30 17:11:05.760135 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:05.760013 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84f45fe0-793d-4f0d-98d5-f78b9a287c5a-workload-socket\") pod \"e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb\" (UID: \"84f45fe0-793d-4f0d-98d5-f78b9a287c5a\") " pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb" Oct 30 17:11:05.760326 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:05.760304 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84f45fe0-793d-4f0d-98d5-f78b9a287c5a-istiod-ca-cert\") pod \"e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb\" (UID: \"84f45fe0-793d-4f0d-98d5-f78b9a287c5a\") " pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb" Oct 30 17:11:05.761962 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:05.761941 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84f45fe0-793d-4f0d-98d5-f78b9a287c5a-istio-envoy\") pod \"e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb\" (UID: \"84f45fe0-793d-4f0d-98d5-f78b9a287c5a\") " pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb" Oct 30 17:11:05.762688 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:05.762671 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84f45fe0-793d-4f0d-98d5-f78b9a287c5a-istio-podinfo\") pod \"e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb\" (UID: \"84f45fe0-793d-4f0d-98d5-f78b9a287c5a\") " pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb" Oct 30 17:11:05.783617 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:05.783268 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54pw5\" (UniqueName: \"kubernetes.io/projected/84f45fe0-793d-4f0d-98d5-f78b9a287c5a-kube-api-access-54pw5\") pod \"e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb\" (UID: \"84f45fe0-793d-4f0d-98d5-f78b9a287c5a\") " pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb" Oct 30 17:11:05.785201 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:05.785180 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84f45fe0-793d-4f0d-98d5-f78b9a287c5a-istio-token\") pod \"e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb\" (UID: \"84f45fe0-793d-4f0d-98d5-f78b9a287c5a\") " pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb" Oct 30 17:11:05.786686 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:05.786669 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg" Oct 30 17:11:05.787617 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:05.787599 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg" Oct 30 17:11:05.792452 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:05.792431 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-xjbtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused" start-of-body= Oct 30 17:11:05.792521 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:05.792483 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-xjbtt" podUID="b9860f52-b69d-4de3-8999-eff8e90620dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused" Oct 30 17:11:05.827332 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:05.827308 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl" Oct 30 17:11:05.911020 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:05.910979 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb" Oct 30 17:11:05.979405 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:05.979376 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8"] Oct 30 17:11:05.994872 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:05.994849 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8" Oct 30 17:11:06.000349 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:05.999919 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk"] Oct 30 17:11:06.041971 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.010820 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-deb721-openshift-default-dockercfg-bq5wb\"" Oct 30 17:11:06.041971 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.041761 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk" Oct 30 17:11:06.047526 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.047470 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8"] Oct 30 17:11:06.055721 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.055589 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk"] Oct 30 17:11:06.072686 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.071133 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56700de4-8e2f-4ede-b163-f152add8e6fa-credential-socket\") pod \"e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8\" (UID: \"56700de4-8e2f-4ede-b163-f152add8e6fa\") " pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8" Oct 30 17:11:06.072686 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.071195 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56700de4-8e2f-4ede-b163-f152add8e6fa-istio-data\") pod \"e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8\" (UID: \"56700de4-8e2f-4ede-b163-f152add8e6fa\") " pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8" Oct 30 17:11:06.072686 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.071265 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56700de4-8e2f-4ede-b163-f152add8e6fa-workload-socket\") pod \"e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8\" (UID: \"56700de4-8e2f-4ede-b163-f152add8e6fa\") " pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8" Oct 30 17:11:06.072686 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.071283 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56700de4-8e2f-4ede-b163-f152add8e6fa-workload-certs\") pod \"e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8\" (UID: \"56700de4-8e2f-4ede-b163-f152add8e6fa\") " pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8" Oct 30 17:11:06.072686 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.071301 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56700de4-8e2f-4ede-b163-f152add8e6fa-istio-envoy\") pod \"e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8\" (UID: \"56700de4-8e2f-4ede-b163-f152add8e6fa\") " pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8" Oct 30 17:11:06.072686 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.071326 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dktss\" (UniqueName: \"kubernetes.io/projected/56700de4-8e2f-4ede-b163-f152add8e6fa-kube-api-access-dktss\") pod \"e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8\" (UID: \"56700de4-8e2f-4ede-b163-f152add8e6fa\") " pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8" Oct 30 17:11:06.072686 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.071402 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56700de4-8e2f-4ede-b163-f152add8e6fa-istio-podinfo\") pod \"e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8\" (UID: \"56700de4-8e2f-4ede-b163-f152add8e6fa\") " pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8" Oct 30 17:11:06.072686 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.071426 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56700de4-8e2f-4ede-b163-f152add8e6fa-istiod-ca-cert\") pod \"e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8\" (UID: \"56700de4-8e2f-4ede-b163-f152add8e6fa\") " pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8" Oct 30 17:11:06.072686 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.071456 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56700de4-8e2f-4ede-b163-f152add8e6fa-istio-token\") pod \"e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8\" (UID: \"56700de4-8e2f-4ede-b163-f152add8e6fa\") " pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8" Oct 30 17:11:06.172971 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.172935 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56700de4-8e2f-4ede-b163-f152add8e6fa-istio-token\") pod \"e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8\" (UID: \"56700de4-8e2f-4ede-b163-f152add8e6fa\") " pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8" Oct 30 17:11:06.173131 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.172998 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26356b33-b847-4ba2-8b73-cfa37b2b14a4-workload-certs\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk\" (UID: \"26356b33-b847-4ba2-8b73-cfa37b2b14a4\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk" Oct 30 17:11:06.173131 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.173041 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26356b33-b847-4ba2-8b73-cfa37b2b14a4-istio-token\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk\" (UID: \"26356b33-b847-4ba2-8b73-cfa37b2b14a4\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk" Oct 30 17:11:06.173131 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.173102 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56700de4-8e2f-4ede-b163-f152add8e6fa-credential-socket\") pod \"e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8\" (UID: \"56700de4-8e2f-4ede-b163-f152add8e6fa\") " pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8" Oct 30 17:11:06.173331 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.173150 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpgqj\" (UniqueName: \"kubernetes.io/projected/26356b33-b847-4ba2-8b73-cfa37b2b14a4-kube-api-access-qpgqj\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk\" (UID: \"26356b33-b847-4ba2-8b73-cfa37b2b14a4\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk" Oct 30 17:11:06.173331 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.173220 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56700de4-8e2f-4ede-b163-f152add8e6fa-istio-data\") pod \"e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8\" (UID: \"56700de4-8e2f-4ede-b163-f152add8e6fa\") " pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8" Oct 30 17:11:06.173331 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.173270 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26356b33-b847-4ba2-8b73-cfa37b2b14a4-istio-podinfo\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk\" (UID: \"26356b33-b847-4ba2-8b73-cfa37b2b14a4\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk" Oct 30 17:11:06.173331 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.173295 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26356b33-b847-4ba2-8b73-cfa37b2b14a4-istio-data\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk\" (UID: \"26356b33-b847-4ba2-8b73-cfa37b2b14a4\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk" Oct 30 17:11:06.173557 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.173348 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26356b33-b847-4ba2-8b73-cfa37b2b14a4-istio-envoy\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk\" (UID: \"26356b33-b847-4ba2-8b73-cfa37b2b14a4\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk" Oct 30 17:11:06.173557 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.173416 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56700de4-8e2f-4ede-b163-f152add8e6fa-workload-socket\") pod \"e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8\" (UID: \"56700de4-8e2f-4ede-b163-f152add8e6fa\") " pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8" Oct 30 17:11:06.173557 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.173448 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56700de4-8e2f-4ede-b163-f152add8e6fa-workload-certs\") pod \"e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8\" (UID: \"56700de4-8e2f-4ede-b163-f152add8e6fa\") " pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8" Oct 30 17:11:06.173557 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.173475 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56700de4-8e2f-4ede-b163-f152add8e6fa-istio-envoy\") pod \"e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8\" (UID: \"56700de4-8e2f-4ede-b163-f152add8e6fa\") " pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8" Oct 30 17:11:06.173557 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.173522 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dktss\" (UniqueName: \"kubernetes.io/projected/56700de4-8e2f-4ede-b163-f152add8e6fa-kube-api-access-dktss\") pod \"e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8\" (UID: \"56700de4-8e2f-4ede-b163-f152add8e6fa\") " pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8" Oct 30 17:11:06.173913 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.173600 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26356b33-b847-4ba2-8b73-cfa37b2b14a4-credential-socket\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk\" (UID: \"26356b33-b847-4ba2-8b73-cfa37b2b14a4\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk" Oct 30 17:11:06.173913 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.173648 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26356b33-b847-4ba2-8b73-cfa37b2b14a4-workload-socket\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk\" (UID: \"26356b33-b847-4ba2-8b73-cfa37b2b14a4\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk" Oct 30 17:11:06.173913 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.173710 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26356b33-b847-4ba2-8b73-cfa37b2b14a4-istiod-ca-cert\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk\" (UID: \"26356b33-b847-4ba2-8b73-cfa37b2b14a4\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk" Oct 30 17:11:06.173913 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.173742 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56700de4-8e2f-4ede-b163-f152add8e6fa-istio-podinfo\") pod \"e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8\" (UID: \"56700de4-8e2f-4ede-b163-f152add8e6fa\") " pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8" Oct 30 17:11:06.173913 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.173816 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56700de4-8e2f-4ede-b163-f152add8e6fa-istiod-ca-cert\") pod \"e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8\" (UID: \"56700de4-8e2f-4ede-b163-f152add8e6fa\") " pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8" Oct 30 17:11:06.174630 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.174603 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56700de4-8e2f-4ede-b163-f152add8e6fa-istiod-ca-cert\") pod \"e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8\" (UID: \"56700de4-8e2f-4ede-b163-f152add8e6fa\") " pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8" Oct 30 17:11:06.175240 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.175213 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56700de4-8e2f-4ede-b163-f152add8e6fa-credential-socket\") pod \"e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8\" (UID: \"56700de4-8e2f-4ede-b163-f152add8e6fa\") " pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8" Oct 30 17:11:06.175489 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.175466 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56700de4-8e2f-4ede-b163-f152add8e6fa-workload-certs\") pod \"e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8\" (UID: \"56700de4-8e2f-4ede-b163-f152add8e6fa\") " pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8" Oct 30 17:11:06.175737 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.175716 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56700de4-8e2f-4ede-b163-f152add8e6fa-istio-data\") pod \"e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8\" (UID: \"56700de4-8e2f-4ede-b163-f152add8e6fa\") " pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8" Oct 30 17:11:06.176059 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.176035 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56700de4-8e2f-4ede-b163-f152add8e6fa-workload-socket\") pod \"e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8\" (UID: \"56700de4-8e2f-4ede-b163-f152add8e6fa\") " pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8" Oct 30 17:11:06.179180 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.179156 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56700de4-8e2f-4ede-b163-f152add8e6fa-istio-podinfo\") pod \"e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8\" (UID: \"56700de4-8e2f-4ede-b163-f152add8e6fa\") " pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8" Oct 30 17:11:06.179327 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.179308 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56700de4-8e2f-4ede-b163-f152add8e6fa-istio-envoy\") pod \"e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8\" (UID: \"56700de4-8e2f-4ede-b163-f152add8e6fa\") " pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8" Oct 30 17:11:06.197860 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.197836 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56700de4-8e2f-4ede-b163-f152add8e6fa-istio-token\") pod \"e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8\" (UID: \"56700de4-8e2f-4ede-b163-f152add8e6fa\") " pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8" Oct 30 17:11:06.199368 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.199340 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dktss\" (UniqueName: \"kubernetes.io/projected/56700de4-8e2f-4ede-b163-f152add8e6fa-kube-api-access-dktss\") pod \"e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8\" (UID: \"56700de4-8e2f-4ede-b163-f152add8e6fa\") " pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8" Oct 30 17:11:06.242324 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.242041 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl"] Oct 30 17:11:06.242324 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:11:06.242186 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35f0aae9_952b_4239_ae42_f9dfb3dd94be.slice/crio-06ce16c10916a495d5a2f292c27d3623f976a24e79b7a0a41384b323c2c2b543 WatchSource:0}: Error finding container 06ce16c10916a495d5a2f292c27d3623f976a24e79b7a0a41384b323c2c2b543: Status 404 returned error can't find the container with id 06ce16c10916a495d5a2f292c27d3623f976a24e79b7a0a41384b323c2c2b543 Oct 30 17:11:06.244491 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.244454 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:11:06.244617 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.244537 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:11:06.244617 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.244586 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:11:06.275900 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.275870 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26356b33-b847-4ba2-8b73-cfa37b2b14a4-istio-podinfo\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk\" (UID: \"26356b33-b847-4ba2-8b73-cfa37b2b14a4\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk" Oct 30 17:11:06.275975 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.275917 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26356b33-b847-4ba2-8b73-cfa37b2b14a4-istio-data\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk\" (UID: \"26356b33-b847-4ba2-8b73-cfa37b2b14a4\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk" Oct 30 17:11:06.275975 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.275964 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26356b33-b847-4ba2-8b73-cfa37b2b14a4-istio-envoy\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk\" (UID: \"26356b33-b847-4ba2-8b73-cfa37b2b14a4\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk" Oct 30 17:11:06.276097 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.276084 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26356b33-b847-4ba2-8b73-cfa37b2b14a4-credential-socket\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk\" (UID: \"26356b33-b847-4ba2-8b73-cfa37b2b14a4\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk" Oct 30 17:11:06.276144 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.276128 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26356b33-b847-4ba2-8b73-cfa37b2b14a4-workload-socket\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk\" (UID: \"26356b33-b847-4ba2-8b73-cfa37b2b14a4\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk" Oct 30 17:11:06.276200 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.276187 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26356b33-b847-4ba2-8b73-cfa37b2b14a4-istiod-ca-cert\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk\" (UID: \"26356b33-b847-4ba2-8b73-cfa37b2b14a4\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk" Oct 30 17:11:06.276277 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.276265 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26356b33-b847-4ba2-8b73-cfa37b2b14a4-workload-certs\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk\" (UID: \"26356b33-b847-4ba2-8b73-cfa37b2b14a4\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk" Oct 30 17:11:06.276321 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.276292 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26356b33-b847-4ba2-8b73-cfa37b2b14a4-istio-token\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk\" (UID: \"26356b33-b847-4ba2-8b73-cfa37b2b14a4\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk" Oct 30 17:11:06.276381 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.276366 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpgqj\" (UniqueName: \"kubernetes.io/projected/26356b33-b847-4ba2-8b73-cfa37b2b14a4-kube-api-access-qpgqj\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk\" (UID: \"26356b33-b847-4ba2-8b73-cfa37b2b14a4\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk" Oct 30 17:11:06.277344 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.277316 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26356b33-b847-4ba2-8b73-cfa37b2b14a4-workload-socket\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk\" (UID: \"26356b33-b847-4ba2-8b73-cfa37b2b14a4\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk" Oct 30 17:11:06.277596 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.277570 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26356b33-b847-4ba2-8b73-cfa37b2b14a4-istio-data\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk\" (UID: \"26356b33-b847-4ba2-8b73-cfa37b2b14a4\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk" Oct 30 17:11:06.278150 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.278126 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26356b33-b847-4ba2-8b73-cfa37b2b14a4-workload-certs\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk\" (UID: \"26356b33-b847-4ba2-8b73-cfa37b2b14a4\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk" Oct 30 17:11:06.278643 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.278576 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26356b33-b847-4ba2-8b73-cfa37b2b14a4-istiod-ca-cert\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk\" (UID: \"26356b33-b847-4ba2-8b73-cfa37b2b14a4\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk" Oct 30 17:11:06.278643 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.278602 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26356b33-b847-4ba2-8b73-cfa37b2b14a4-credential-socket\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk\" (UID: \"26356b33-b847-4ba2-8b73-cfa37b2b14a4\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk" Oct 30 17:11:06.279949 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.279721 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26356b33-b847-4ba2-8b73-cfa37b2b14a4-istio-envoy\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk\" (UID: \"26356b33-b847-4ba2-8b73-cfa37b2b14a4\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk" Oct 30 17:11:06.280531 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.280131 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26356b33-b847-4ba2-8b73-cfa37b2b14a4-istio-podinfo\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk\" (UID: \"26356b33-b847-4ba2-8b73-cfa37b2b14a4\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk" Oct 30 17:11:06.323770 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.323704 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpgqj\" (UniqueName: \"kubernetes.io/projected/26356b33-b847-4ba2-8b73-cfa37b2b14a4-kube-api-access-qpgqj\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk\" (UID: \"26356b33-b847-4ba2-8b73-cfa37b2b14a4\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk" Oct 30 17:11:06.327436 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.327309 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26356b33-b847-4ba2-8b73-cfa37b2b14a4-istio-token\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk\" (UID: \"26356b33-b847-4ba2-8b73-cfa37b2b14a4\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk" Oct 30 17:11:06.350275 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.350052 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8" Oct 30 17:11:06.357727 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.357700 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb"] Oct 30 17:11:06.358405 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:11:06.358369 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84f45fe0_793d_4f0d_98d5_f78b9a287c5a.slice/crio-0557abb52e80988ea44898ecdfde455af7aa05b41c9ca3badfef60f24e5cdc99 WatchSource:0}: Error finding container 0557abb52e80988ea44898ecdfde455af7aa05b41c9ca3badfef60f24e5cdc99: Status 404 returned error can't find the container with id 0557abb52e80988ea44898ecdfde455af7aa05b41c9ca3badfef60f24e5cdc99 Oct 30 17:11:06.371230 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.369262 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:11:06.371230 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.369364 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:11:06.371230 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.369410 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:11:06.371230 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.371024 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk" Oct 30 17:11:06.760279 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.760248 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8"] Oct 30 17:11:06.786002 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.774952 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:11:06.786002 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.775042 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:11:06.786002 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.775082 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:11:06.793122 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.791360 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-xjbtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused" start-of-body= Oct 30 17:11:06.793122 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.791424 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-xjbtt" podUID="b9860f52-b69d-4de3-8999-eff8e90620dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused" Oct 30 17:11:06.793122 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.793089 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk"] Oct 30 17:11:06.815851 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.815742 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8" event={"ID":"56700de4-8e2f-4ede-b163-f152add8e6fa","Type":"ContainerStarted","Data":"e7e56ee69852de55569a30b6efe3328fa8bb32916ac15e7dbdabb5a2608e1ad4"} Oct 30 17:11:06.819610 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.818387 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl" event={"ID":"35f0aae9-952b-4239-ae42-f9dfb3dd94be","Type":"ContainerStarted","Data":"e2ebfae1618e58a6b5da158c8d3200e7837c63201fe3e315a5d8090d34581620"} Oct 30 17:11:06.819610 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.818450 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl" event={"ID":"35f0aae9-952b-4239-ae42-f9dfb3dd94be","Type":"ContainerStarted","Data":"06ce16c10916a495d5a2f292c27d3623f976a24e79b7a0a41384b323c2c2b543"} Oct 30 17:11:06.824094 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.823404 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb" event={"ID":"84f45fe0-793d-4f0d-98d5-f78b9a287c5a","Type":"ContainerStarted","Data":"844da8f5d8e17772fdbe3f484dba17d3a764e5e1d2b796bc7a07dcadd2294c70"} Oct 30 17:11:06.824094 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.823459 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb" event={"ID":"84f45fe0-793d-4f0d-98d5-f78b9a287c5a","Type":"ContainerStarted","Data":"0557abb52e80988ea44898ecdfde455af7aa05b41c9ca3badfef60f24e5cdc99"} Oct 30 17:11:06.827945 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.827892 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl" Oct 30 17:11:06.886342 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:11:06.885870 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26356b33_b847_4ba2_8b73_cfa37b2b14a4.slice/crio-fda5147535928243b6065999b9b49a723c0d1923df82dfa3107fbfb7af0c36f0 WatchSource:0}: Error finding container fda5147535928243b6065999b9b49a723c0d1923df82dfa3107fbfb7af0c36f0: Status 404 returned error can't find the container with id fda5147535928243b6065999b9b49a723c0d1923df82dfa3107fbfb7af0c36f0 Oct 30 17:11:06.890840 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.890158 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:11:06.890840 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.890272 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:11:06.890840 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.890312 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:11:06.897765 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.897623 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl" podStartSLOduration=1.8976040360000002 podStartE2EDuration="1.897604036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:11:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:11:06.892623143 +0000 UTC m=+2894.119798249" watchObservedRunningTime="2026-10-30 17:11:06.897604036 +0000 UTC m=+2894.124779137" Oct 30 17:11:06.915112 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.913128 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb" Oct 30 17:11:06.974505 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:06.969681 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb" podStartSLOduration=1.969653279 podStartE2EDuration="1.969653279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:11:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:11:06.963201587 +0000 UTC m=+2894.190376737" watchObservedRunningTime="2026-10-30 17:11:06.969653279 +0000 UTC m=+2894.196828379" Oct 30 17:11:07.799880 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:07.799408 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-xjbtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused" start-of-body= Oct 30 17:11:07.799880 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:07.799477 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-xjbtt" podUID="b9860f52-b69d-4de3-8999-eff8e90620dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused" Oct 30 17:11:07.876978 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:07.876611 4493 patch_prober.go:29] interesting pod/e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" start-of-body= Oct 30 17:11:07.876978 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:07.876673 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl" podUID="35f0aae9-952b-4239-ae42-f9dfb3dd94be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" Oct 30 17:11:07.924056 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:07.923909 4493 patch_prober.go:29] interesting pod/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" start-of-body= Oct 30 17:11:07.924056 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:07.923967 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb" podUID="84f45fe0-793d-4f0d-98d5-f78b9a287c5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" Oct 30 17:11:07.942011 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:07.941940 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk" event={"ID":"26356b33-b847-4ba2-8b73-cfa37b2b14a4","Type":"ContainerStarted","Data":"46e37164664f969bb2947cf53a46c36d20379cd52502957c2a5c3ad4ac767fe4"} Oct 30 17:11:07.942206 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:07.942058 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk" event={"ID":"26356b33-b847-4ba2-8b73-cfa37b2b14a4","Type":"ContainerStarted","Data":"fda5147535928243b6065999b9b49a723c0d1923df82dfa3107fbfb7af0c36f0"} Oct 30 17:11:08.034864 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:08.034828 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8" event={"ID":"56700de4-8e2f-4ede-b163-f152add8e6fa","Type":"ContainerStarted","Data":"40aea16c219909e45a0fcac30a7f966c94b874ae718a7470de494597b5afbb69"} Oct 30 17:11:08.136867 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:08.125915 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk" podStartSLOduration=3.125888612 podStartE2EDuration="3.125888612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:11:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:11:08.060209013 +0000 UTC m=+2895.287384114" watchObservedRunningTime="2026-10-30 17:11:08.125888612 +0000 UTC m=+2895.353063714" Oct 30 17:11:08.201174 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:08.201114 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8" podStartSLOduration=3.2010942 podStartE2EDuration="3.2010942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:11:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:11:08.119213186 +0000 UTC m=+2895.346388286" watchObservedRunningTime="2026-10-30 17:11:08.2010942 +0000 UTC m=+2895.428269301" Oct 30 17:11:08.397869 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:08.393957 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk" Oct 30 17:11:08.397869 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:08.395713 4493 patch_prober.go:29] interesting pod/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" start-of-body= Oct 30 17:11:08.397869 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:08.395765 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk" podUID="26356b33-b847-4ba2-8b73-cfa37b2b14a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" Oct 30 17:11:08.397869 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:08.395954 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8" Oct 30 17:11:08.582059 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:08.582024 4493 patch_prober.go:29] interesting pod/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" start-of-body= Oct 30 17:11:08.595838 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:08.582271 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8" podUID="56700de4-8e2f-4ede-b163-f152add8e6fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" Oct 30 17:11:08.891899 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:08.801930 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-xjbtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused" start-of-body= Oct 30 17:11:08.891899 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:08.801994 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-xjbtt" podUID="b9860f52-b69d-4de3-8999-eff8e90620dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused" Oct 30 17:11:09.015835 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:08.974740 4493 patch_prober.go:29] interesting pod/e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" start-of-body= Oct 30 17:11:09.015835 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:08.974830 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl" podUID="35f0aae9-952b-4239-ae42-f9dfb3dd94be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" Oct 30 17:11:09.027098 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:09.026960 4493 patch_prober.go:29] interesting pod/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" start-of-body= Oct 30 17:11:09.027098 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:09.027026 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb" podUID="84f45fe0-793d-4f0d-98d5-f78b9a287c5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" Oct 30 17:11:09.167313 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:09.167224 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" podUID="b5abe703-fee0-4185-aa22-88d1db06392b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Oct 30 17:11:09.167778 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:09.167751 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" podUID="e90f43d3-f5eb-45ab-af1e-340d0ce18966" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Oct 30 17:11:09.168115 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:09.168093 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" podUID="14805045-a727-41ba-8bd0-100d41f9f27d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 17:11:09.217872 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:09.205098 4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" podUID="80b9bd9a-30d6-45ff-b318-26fcdc608e5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": context deadline exceeded" Oct 30 17:11:09.217872 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:09.206221 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" podUID="e43738f3-d55f-4c0a-a364-bf2ac76605c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 17:11:09.318602 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:09.271204 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" podUID="18b72648-f523-405e-b5cd-1266ff13f8f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:09.318602 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:09.309988 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" podUID="80b9bd9a-30d6-45ff-b318-26fcdc608e5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:09.419866 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:09.404999 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" podUID="08baad08-cd72-4305-b92d-c11a84ef276f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:09.419866 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:09.405338 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" podUID="f3e1c8ff-fcc3-4a1f-9730-c617d57b5193" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:09.448812 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:09.441966 4493 patch_prober.go:29] interesting pod/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" start-of-body= Oct 30 17:11:09.448812 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:09.442027 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk" podUID="26356b33-b847-4ba2-8b73-cfa37b2b14a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" Oct 30 17:11:09.448812 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:09.442279 4493 patch_prober.go:29] interesting pod/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" start-of-body= Oct 30 17:11:09.448812 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:09.442308 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8" podUID="56700de4-8e2f-4ede-b163-f152add8e6fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" Oct 30 17:11:09.570548 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:09.527024 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" podUID="0eadc5d4-98c3-4bf2-b3c2-f73fe249182f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 30 17:11:09.820073 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:09.808942 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-xjbtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused" start-of-body= Oct 30 17:11:09.820073 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:09.809021 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-xjbtt" podUID="b9860f52-b69d-4de3-8999-eff8e90620dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused" Oct 30 17:11:09.837807 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:09.837753 4493 patch_prober.go:29] interesting pod/e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" start-of-body= Oct 30 17:11:09.838079 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:09.838051 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl" podUID="35f0aae9-952b-4239-ae42-f9dfb3dd94be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" Oct 30 17:11:09.920851 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:09.920802 4493 patch_prober.go:29] interesting pod/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" start-of-body= Oct 30 17:11:09.936813 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:09.928865 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb" podUID="84f45fe0-793d-4f0d-98d5-f78b9a287c5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" Oct 30 17:11:09.960813 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:09.957929 4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" podUID="f3e1c8ff-fcc3-4a1f-9730-c617d57b5193" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:10.192811 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:10.181899 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" podUID="e76172b1-659d-45bd-91c6-1a0ce4062335" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:10.192811 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:10.182276 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" podUID="b5abe703-fee0-4185-aa22-88d1db06392b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Oct 30 17:11:10.192811 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:10.182562 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" podUID="e90f43d3-f5eb-45ab-af1e-340d0ce18966" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Oct 30 17:11:10.192811 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:10.182818 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" podUID="14805045-a727-41ba-8bd0-100d41f9f27d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 17:11:10.192811 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:10.183495 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" Oct 30 17:11:10.192811 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:10.183553 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" Oct 30 17:11:10.192811 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:10.183601 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" Oct 30 17:11:10.272948 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:10.272899 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" podUID="18b72648-f523-405e-b5cd-1266ff13f8f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:10.273459 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:10.273435 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" Oct 30 17:11:10.335241 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:10.326558 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" podUID="80b9bd9a-30d6-45ff-b318-26fcdc608e5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:10.335241 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:10.326843 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" Oct 30 17:11:10.398811 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:10.395713 4493 patch_prober.go:29] interesting pod/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" start-of-body= Oct 30 17:11:10.398811 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:10.395775 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8" podUID="56700de4-8e2f-4ede-b163-f152add8e6fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" Oct 30 17:11:10.424480 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:10.409027 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" podUID="f3e1c8ff-fcc3-4a1f-9730-c617d57b5193" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:10.424480 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:10.409376 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" podUID="08baad08-cd72-4305-b92d-c11a84ef276f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 17:11:10.424480 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:10.419945 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" Oct 30 17:11:10.424480 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:10.420017 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" Oct 30 17:11:10.424480 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:10.421004 4493 patch_prober.go:29] interesting pod/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" start-of-body= Oct 30 17:11:10.424480 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:10.421048 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk" podUID="26356b33-b847-4ba2-8b73-cfa37b2b14a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" Oct 30 17:11:10.559375 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:10.545668 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" podUID="0eadc5d4-98c3-4bf2-b3c2-f73fe249182f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:10.559375 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:10.546147 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" Oct 30 17:11:10.843860 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:10.825703 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-xjbtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused" start-of-body= Oct 30 17:11:10.843860 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:10.825774 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-xjbtt" podUID="b9860f52-b69d-4de3-8999-eff8e90620dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused" Oct 30 17:11:11.013500 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:10.982991 4493 patch_prober.go:29] interesting pod/e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" start-of-body= Oct 30 17:11:11.013500 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:10.983052 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl" podUID="35f0aae9-952b-4239-ae42-f9dfb3dd94be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" Oct 30 17:11:11.015570 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:11.015537 4493 patch_prober.go:29] interesting pod/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" start-of-body= Oct 30 17:11:11.015765 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:11.015739 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb" podUID="84f45fe0-793d-4f0d-98d5-f78b9a287c5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" Oct 30 17:11:11.234574 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:11.216982 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" podUID="e76172b1-659d-45bd-91c6-1a0ce4062335" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 17:11:11.234574 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:11.217388 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" podUID="b5abe703-fee0-4185-aa22-88d1db06392b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Oct 30 17:11:11.234574 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:11.217683 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" podUID="e90f43d3-f5eb-45ab-af1e-340d0ce18966" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Oct 30 17:11:11.234574 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:11.218534 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" podUID="14805045-a727-41ba-8bd0-100d41f9f27d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:11.236934 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:11.236888 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" podUID="e43738f3-d55f-4c0a-a364-bf2ac76605c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 17:11:11.277913 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:11.276375 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" podUID="18b72648-f523-405e-b5cd-1266ff13f8f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:11.407808 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:11.367927 4493 patch_prober.go:29] interesting pod/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" start-of-body= Oct 30 17:11:11.416893 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:11.416855 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8" podUID="56700de4-8e2f-4ede-b163-f152add8e6fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" Oct 30 17:11:11.553602 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:11.414098 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" podUID="08baad08-cd72-4305-b92d-c11a84ef276f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:11.580057 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:11.559007 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" podUID="0eadc5d4-98c3-4bf2-b3c2-f73fe249182f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:11.580057 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:11.391614 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" podUID="80b9bd9a-30d6-45ff-b318-26fcdc608e5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 17:11:11.662808 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:11.414154 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" podUID="f3e1c8ff-fcc3-4a1f-9730-c617d57b5193" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded" Oct 30 17:11:11.705844 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:11.400153 4493 patch_prober.go:29] interesting pod/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" start-of-body= Oct 30 17:11:11.709914 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:11.709842 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk" podUID="26356b33-b847-4ba2-8b73-cfa37b2b14a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" Oct 30 17:11:11.771269 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:11.771220 4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" podUID="08baad08-cd72-4305-b92d-c11a84ef276f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:11.831184 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:11.774229 4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" podUID="18b72648-f523-405e-b5cd-1266ff13f8f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": context deadline exceeded" Oct 30 17:11:11.831184 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:11.806107 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-xjbtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused" start-of-body= Oct 30 17:11:11.831184 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:11.806173 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-xjbtt" podUID="b9860f52-b69d-4de3-8999-eff8e90620dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused" Oct 30 17:11:11.831184 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:11.830004 4493 patch_prober.go:29] interesting pod/e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" start-of-body= Oct 30 17:11:11.831184 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:11.830088 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl" podUID="35f0aae9-952b-4239-ae42-f9dfb3dd94be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" Oct 30 17:11:11.943342 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:11.922307 4493 patch_prober.go:29] interesting pod/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" start-of-body= Oct 30 17:11:11.943342 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:11.922383 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb" podUID="84f45fe0-793d-4f0d-98d5-f78b9a287c5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" Oct 30 17:11:12.155869 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:11:12.155098 4493 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.211s" Oct 30 17:11:12.155869 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:12.155255 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" Oct 30 17:11:12.486868 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:12.324120 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" podUID="e43738f3-d55f-4c0a-a364-bf2ac76605c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 17:11:12.571810 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:12.334961 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" podUID="18b72648-f523-405e-b5cd-1266ff13f8f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 30 17:11:12.571810 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:12.538114 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" Oct 30 17:11:12.571810 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:12.336558 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" podUID="e76172b1-659d-45bd-91c6-1a0ce4062335" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 17:11:12.571810 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:12.336621 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" podUID="b5abe703-fee0-4185-aa22-88d1db06392b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Oct 30 17:11:12.571810 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:12.336658 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" podUID="e90f43d3-f5eb-45ab-af1e-340d0ce18966" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Oct 30 17:11:12.571810 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:12.336690 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" podUID="14805045-a727-41ba-8bd0-100d41f9f27d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 17:11:12.571810 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:12.477953 4493 patch_prober.go:29] interesting pod/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" start-of-body= Oct 30 17:11:12.870574 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:12.814289 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" podUID="08baad08-cd72-4305-b92d-c11a84ef276f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 17:11:12.870574 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:12.861087 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" podUID="0eadc5d4-98c3-4bf2-b3c2-f73fe249182f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 30 17:11:12.870574 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:12.861513 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" podUID="80b9bd9a-30d6-45ff-b318-26fcdc608e5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 17:11:12.870574 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:12.864341 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-xjbtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused" start-of-body= Oct 30 17:11:12.870574 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:12.864393 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-xjbtt" podUID="b9860f52-b69d-4de3-8999-eff8e90620dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused" Oct 30 17:11:12.875444 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:12.478019 4493 patch_prober.go:29] interesting pod/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" start-of-body= Oct 30 17:11:13.037764 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:13.032812 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8" podUID="56700de4-8e2f-4ede-b163-f152add8e6fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" Oct 30 17:11:13.037764 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:12.879211 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk" podUID="26356b33-b847-4ba2-8b73-cfa37b2b14a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" Oct 30 17:11:13.037764 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:12.879510 4493 patch_prober.go:29] interesting pod/e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" start-of-body= Oct 30 17:11:13.037764 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:13.032946 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl" podUID="35f0aae9-952b-4239-ae42-f9dfb3dd94be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" Oct 30 17:11:13.055086 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:13.042951 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" podUID="f3e1c8ff-fcc3-4a1f-9730-c617d57b5193" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded" Oct 30 17:11:13.175900 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:13.175859 4493 patch_prober.go:29] interesting pod/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" start-of-body= Oct 30 17:11:13.176516 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:13.176481 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb" podUID="84f45fe0-793d-4f0d-98d5-f78b9a287c5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" Oct 30 17:11:13.403816 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:13.399046 4493 patch_prober.go:29] interesting pod/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" start-of-body= Oct 30 17:11:13.403816 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:13.399164 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8" podUID="56700de4-8e2f-4ede-b163-f152add8e6fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" Oct 30 17:11:13.515196 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:13.509572 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" podUID="e43738f3-d55f-4c0a-a364-bf2ac76605c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:13.542201 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:13.542139 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" podUID="18b72648-f523-405e-b5cd-1266ff13f8f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:13.542545 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:13.542518 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" podUID="e76172b1-659d-45bd-91c6-1a0ce4062335" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 17:11:13.542731 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:13.542702 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" podUID="b5abe703-fee0-4185-aa22-88d1db06392b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Oct 30 17:11:13.556170 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:13.556114 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" podUID="14805045-a727-41ba-8bd0-100d41f9f27d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:13.587251 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:13.571025 4493 patch_prober.go:29] interesting pod/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" start-of-body= Oct 30 17:11:13.587519 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:13.587489 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk" podUID="26356b33-b847-4ba2-8b73-cfa37b2b14a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" Oct 30 17:11:13.587657 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:13.586962 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" Oct 30 17:11:13.809907 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:13.809625 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-xjbtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused" start-of-body= Oct 30 17:11:13.809907 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:13.809682 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-xjbtt" podUID="b9860f52-b69d-4de3-8999-eff8e90620dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused" Oct 30 17:11:13.829958 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:13.823931 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" podUID="08baad08-cd72-4305-b92d-c11a84ef276f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 17:11:13.850835 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:13.835538 4493 patch_prober.go:29] interesting pod/e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" start-of-body= Oct 30 17:11:13.850835 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:13.839867 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl" podUID="35f0aae9-952b-4239-ae42-f9dfb3dd94be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" Oct 30 17:11:13.871003 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:13.865952 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" podUID="0eadc5d4-98c3-4bf2-b3c2-f73fe249182f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 30 17:11:13.871003 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:13.866092 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" podUID="80b9bd9a-30d6-45ff-b318-26fcdc608e5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:14.066034 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:14.065950 4493 patch_prober.go:29] interesting pod/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" start-of-body= Oct 30 17:11:14.066272 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:14.066243 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb" podUID="84f45fe0-793d-4f0d-98d5-f78b9a287c5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" Oct 30 17:11:14.067937 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:14.067909 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" podUID="f3e1c8ff-fcc3-4a1f-9730-c617d57b5193" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:14.097549 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:14.097519 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" Oct 30 17:11:14.387238 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:14.387127 4493 patch_prober.go:29] interesting pod/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" start-of-body= Oct 30 17:11:14.401011 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:14.391866 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8" podUID="56700de4-8e2f-4ede-b163-f152add8e6fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" Oct 30 17:11:14.451813 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:14.419020 4493 patch_prober.go:29] interesting pod/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" start-of-body= Oct 30 17:11:14.451813 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:14.419094 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk" podUID="26356b33-b847-4ba2-8b73-cfa37b2b14a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" Oct 30 17:11:14.451813 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:14.419184 4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" podUID="14805045-a727-41ba-8bd0-100d41f9f27d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:14.537602 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:14.537514 4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" podUID="b5abe703-fee0-4185-aa22-88d1db06392b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:14.618061 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:14.618029 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" podUID="14805045-a727-41ba-8bd0-100d41f9f27d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:14.618375 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:14.618000 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" podUID="18b72648-f523-405e-b5cd-1266ff13f8f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 30 17:11:14.640973 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:14.640931 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-extmod-shr-user-e1267c/e2e-usage-sim-53fe84-6c9ffdddfd-t4swq" podUID="0da18196-46f9-493f-aad4-bb3bb825363a" containerName="sim" probeResult="failure" output="Get \"https://10.134.1.3:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:14.641262 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:14.641233 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" podUID="b5abe703-fee0-4185-aa22-88d1db06392b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Oct 30 17:11:14.642586 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:14.642558 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" podUID="e43738f3-d55f-4c0a-a364-bf2ac76605c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 17:11:14.664484 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:14.644195 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" podUID="e76172b1-659d-45bd-91c6-1a0ce4062335" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 17:11:14.831810 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:14.831612 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" podUID="08baad08-cd72-4305-b92d-c11a84ef276f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 17:11:14.833313 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:14.833287 4493 patch_prober.go:29] interesting pod/e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" start-of-body= Oct 30 17:11:14.834179 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:14.834151 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl" podUID="35f0aae9-952b-4239-ae42-f9dfb3dd94be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" Oct 30 17:11:14.867892 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:14.833760 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-xjbtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused" start-of-body= Oct 30 17:11:14.867892 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:14.842747 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-xjbtt" podUID="b9860f52-b69d-4de3-8999-eff8e90620dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.5:15021/healthz/ready\": dial tcp 10.134.1.5:15021: connect: connection refused" Oct 30 17:11:14.868349 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:14.868311 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" podUID="80b9bd9a-30d6-45ff-b318-26fcdc608e5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 17:11:14.921987 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:14.921897 4493 patch_prober.go:29] interesting pod/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" start-of-body= Oct 30 17:11:14.925812 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:14.922189 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb" podUID="84f45fe0-793d-4f0d-98d5-f78b9a287c5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" Oct 30 17:11:14.933811 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:14.926069 4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" podUID="e43738f3-d55f-4c0a-a364-bf2ac76605c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:15.098811 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:15.091036 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" podUID="f3e1c8ff-fcc3-4a1f-9730-c617d57b5193" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:15.112813 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:15.112752 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" podUID="0eadc5d4-98c3-4bf2-b3c2-f73fe249182f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:15.411709 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:15.408815 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw"] Oct 30 17:11:15.411709 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:15.410874 4493 patch_prober.go:29] interesting pod/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" start-of-body= Oct 30 17:11:15.411709 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:15.410926 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk" podUID="26356b33-b847-4ba2-8b73-cfa37b2b14a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" Oct 30 17:11:15.457154 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:15.437916 4493 patch_prober.go:29] interesting pod/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" start-of-body= Oct 30 17:11:15.457154 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:15.437985 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8" podUID="56700de4-8e2f-4ede-b163-f152add8e6fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" Oct 30 17:11:15.536208 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:15.536175 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw"] Oct 30 17:11:15.536445 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:15.536430 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5dbf38c-4wrxh"] Oct 30 17:11:15.536600 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:15.536587 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5dbf38c-4wrxh"] Oct 30 17:11:15.536812 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:15.536777 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw" Oct 30 17:11:15.578326 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:15.578291 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-2589b5-openshift-default-dockercfg-r6f89\"" Oct 30 17:11:15.623653 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:15.620881 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" podUID="e90f43d3-f5eb-45ab-af1e-340d0ce18966" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:15.623844 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:15.623630 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27945ccd-4a29-4f22-a0a2-e1f663869939-workload-socket\") pod \"e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw\" (UID: \"27945ccd-4a29-4f22-a0a2-e1f663869939\") " pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw" Oct 30 17:11:15.623844 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:15.623753 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27945ccd-4a29-4f22-a0a2-e1f663869939-istiod-ca-cert\") pod \"e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw\" (UID: \"27945ccd-4a29-4f22-a0a2-e1f663869939\") " pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw" Oct 30 17:11:15.623996 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:15.623850 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27945ccd-4a29-4f22-a0a2-e1f663869939-istio-podinfo\") pod \"e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw\" (UID: \"27945ccd-4a29-4f22-a0a2-e1f663869939\") " pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw" Oct 30 17:11:15.623996 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:15.623941 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27945ccd-4a29-4f22-a0a2-e1f663869939-istio-envoy\") pod \"e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw\" (UID: \"27945ccd-4a29-4f22-a0a2-e1f663869939\") " pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw" Oct 30 17:11:15.623996 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:15.623970 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cnjb\" (UniqueName: \"kubernetes.io/projected/27945ccd-4a29-4f22-a0a2-e1f663869939-kube-api-access-4cnjb\") pod \"e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw\" (UID: \"27945ccd-4a29-4f22-a0a2-e1f663869939\") " pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw" Oct 30 17:11:15.624159 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:15.624024 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27945ccd-4a29-4f22-a0a2-e1f663869939-istio-data\") pod \"e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw\" (UID: \"27945ccd-4a29-4f22-a0a2-e1f663869939\") " pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw" Oct 30 17:11:15.624159 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:15.624056 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27945ccd-4a29-4f22-a0a2-e1f663869939-workload-certs\") pod \"e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw\" (UID: \"27945ccd-4a29-4f22-a0a2-e1f663869939\") " pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw" Oct 30 17:11:15.624159 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:15.624118 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27945ccd-4a29-4f22-a0a2-e1f663869939-credential-socket\") pod \"e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw\" (UID: \"27945ccd-4a29-4f22-a0a2-e1f663869939\") " pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw" Oct 30 17:11:15.624159 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:15.624150 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27945ccd-4a29-4f22-a0a2-e1f663869939-istio-token\") pod \"e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw\" (UID: \"27945ccd-4a29-4f22-a0a2-e1f663869939\") " pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw" Oct 30 17:11:15.630989 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:15.624433 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" podUID="18b72648-f523-405e-b5cd-1266ff13f8f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:15.630989 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:15.624574 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" podUID="14805045-a727-41ba-8bd0-100d41f9f27d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:15.674977 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:15.641759 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" podUID="b5abe703-fee0-4185-aa22-88d1db06392b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Oct 30 17:11:15.674977 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:15.649181 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" podUID="e43738f3-d55f-4c0a-a364-bf2ac76605c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:15.674977 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:15.650093 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" podUID="e76172b1-659d-45bd-91c6-1a0ce4062335" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:15.732818 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:15.728744 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27945ccd-4a29-4f22-a0a2-e1f663869939-istio-envoy\") pod \"e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw\" (UID: \"27945ccd-4a29-4f22-a0a2-e1f663869939\") " pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw" Oct 30 17:11:15.732818 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:15.728824 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cnjb\" (UniqueName: \"kubernetes.io/projected/27945ccd-4a29-4f22-a0a2-e1f663869939-kube-api-access-4cnjb\") pod \"e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw\" (UID: \"27945ccd-4a29-4f22-a0a2-e1f663869939\") " pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw" Oct 30 17:11:15.732818 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:15.728885 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27945ccd-4a29-4f22-a0a2-e1f663869939-istio-data\") pod \"e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw\" (UID: \"27945ccd-4a29-4f22-a0a2-e1f663869939\") " pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw" Oct 30 17:11:15.732818 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:15.728921 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27945ccd-4a29-4f22-a0a2-e1f663869939-workload-certs\") pod \"e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw\" (UID: \"27945ccd-4a29-4f22-a0a2-e1f663869939\") " pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw" Oct 30 17:11:15.732818 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:15.728984 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27945ccd-4a29-4f22-a0a2-e1f663869939-credential-socket\") pod \"e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw\" (UID: \"27945ccd-4a29-4f22-a0a2-e1f663869939\") " pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw" Oct 30 17:11:15.732818 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:15.729020 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27945ccd-4a29-4f22-a0a2-e1f663869939-istio-token\") pod \"e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw\" (UID: \"27945ccd-4a29-4f22-a0a2-e1f663869939\") " pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw" Oct 30 17:11:15.732818 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:15.729119 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27945ccd-4a29-4f22-a0a2-e1f663869939-workload-socket\") pod \"e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw\" (UID: \"27945ccd-4a29-4f22-a0a2-e1f663869939\") " pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw" Oct 30 17:11:15.732818 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:15.729216 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27945ccd-4a29-4f22-a0a2-e1f663869939-istiod-ca-cert\") pod \"e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw\" (UID: \"27945ccd-4a29-4f22-a0a2-e1f663869939\") " pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw" Oct 30 17:11:15.732818 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:15.729307 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27945ccd-4a29-4f22-a0a2-e1f663869939-istio-podinfo\") pod \"e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw\" (UID: \"27945ccd-4a29-4f22-a0a2-e1f663869939\") " pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw" Oct 30 17:11:15.732818 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:15.730365 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27945ccd-4a29-4f22-a0a2-e1f663869939-workload-socket\") pod \"e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw\" (UID: \"27945ccd-4a29-4f22-a0a2-e1f663869939\") " pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw" Oct 30 17:11:15.754272 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:15.753931 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27945ccd-4a29-4f22-a0a2-e1f663869939-istio-data\") pod \"e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw\" (UID: \"27945ccd-4a29-4f22-a0a2-e1f663869939\") " pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw" Oct 30 17:11:15.754866 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:15.754613 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27945ccd-4a29-4f22-a0a2-e1f663869939-istio-envoy\") pod \"e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw\" (UID: \"27945ccd-4a29-4f22-a0a2-e1f663869939\") " pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw" Oct 30 17:11:15.761310 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:15.759428 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27945ccd-4a29-4f22-a0a2-e1f663869939-workload-certs\") pod \"e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw\" (UID: \"27945ccd-4a29-4f22-a0a2-e1f663869939\") " pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw" Oct 30 17:11:15.761310 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:15.760136 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27945ccd-4a29-4f22-a0a2-e1f663869939-istiod-ca-cert\") pod \"e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw\" (UID: \"27945ccd-4a29-4f22-a0a2-e1f663869939\") " pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw" Oct 30 17:11:15.782322 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:15.782290 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27945ccd-4a29-4f22-a0a2-e1f663869939-credential-socket\") pod \"e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw\" (UID: \"27945ccd-4a29-4f22-a0a2-e1f663869939\") " pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw" Oct 30 17:11:15.847809 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:15.824974 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27945ccd-4a29-4f22-a0a2-e1f663869939-istio-token\") pod \"e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw\" (UID: \"27945ccd-4a29-4f22-a0a2-e1f663869939\") " pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw" Oct 30 17:11:15.847809 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:15.826401 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27945ccd-4a29-4f22-a0a2-e1f663869939-istio-podinfo\") pod \"e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw\" (UID: \"27945ccd-4a29-4f22-a0a2-e1f663869939\") " pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw" Oct 30 17:11:15.847809 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:15.827843 4493 patch_prober.go:29] interesting pod/e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" start-of-body= Oct 30 17:11:15.847809 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:15.827888 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl" podUID="35f0aae9-952b-4239-ae42-f9dfb3dd94be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" Oct 30 17:11:15.847809 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:15.831872 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cnjb\" (UniqueName: \"kubernetes.io/projected/27945ccd-4a29-4f22-a0a2-e1f663869939-kube-api-access-4cnjb\") pod \"e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw\" (UID: \"27945ccd-4a29-4f22-a0a2-e1f663869939\") " pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw" Oct 30 17:11:15.847809 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:15.831950 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" podUID="08baad08-cd72-4305-b92d-c11a84ef276f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:15.871974 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:15.871870 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" podUID="80b9bd9a-30d6-45ff-b318-26fcdc608e5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:15.912585 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:15.912165 4493 patch_prober.go:29] interesting pod/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" start-of-body= Oct 30 17:11:15.912585 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:15.912221 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb" podUID="84f45fe0-793d-4f0d-98d5-f78b9a287c5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" Oct 30 17:11:15.978489 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:15.978401 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw" Oct 30 17:11:16.119808 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:16.117531 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" podUID="0eadc5d4-98c3-4bf2-b3c2-f73fe249182f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 30 17:11:16.139833 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:16.129902 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" Oct 30 17:11:16.139833 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:16.133922 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" podUID="f3e1c8ff-fcc3-4a1f-9730-c617d57b5193" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:16.139833 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:16.134285 4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" podUID="e76172b1-659d-45bd-91c6-1a0ce4062335" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:16.420903 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:16.383951 4493 patch_prober.go:29] interesting pod/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" start-of-body= Oct 30 17:11:16.420903 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:16.384011 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk" podUID="26356b33-b847-4ba2-8b73-cfa37b2b14a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" Oct 30 17:11:16.420903 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:16.402313 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" Oct 30 17:11:16.450062 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:16.431838 4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" podUID="e90f43d3-f5eb-45ab-af1e-340d0ce18966" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:16.638832 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:16.625272 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" podUID="18b72648-f523-405e-b5cd-1266ff13f8f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:16.638832 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:16.626253 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" podUID="14805045-a727-41ba-8bd0-100d41f9f27d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:16.678863 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:16.643876 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" podUID="b5abe703-fee0-4185-aa22-88d1db06392b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:16.678863 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:16.671072 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" podUID="e76172b1-659d-45bd-91c6-1a0ce4062335" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:16.678863 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:16.671433 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" podUID="e43738f3-d55f-4c0a-a364-bf2ac76605c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 17:11:16.829882 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:16.806489 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" Oct 30 17:11:16.833351 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:16.833011 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" podUID="08baad08-cd72-4305-b92d-c11a84ef276f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:16.876982 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:16.872381 4493 patch_prober.go:29] interesting pod/e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" start-of-body= Oct 30 17:11:16.986184 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:16.986085 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl" podUID="35f0aae9-952b-4239-ae42-f9dfb3dd94be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" Oct 30 17:11:17.021494 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:16.986423 4493 patch_prober.go:29] interesting pod/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" start-of-body= Oct 30 17:11:17.021494 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:16.877592 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-xjbtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.5:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:11:17.021494 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:16.977624 4493 patch_prober.go:29] interesting pod/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" start-of-body= Oct 30 17:11:17.021494 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:16.993934 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb" podUID="84f45fe0-793d-4f0d-98d5-f78b9a287c5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" Oct 30 17:11:17.021494 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:16.993982 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8" podUID="56700de4-8e2f-4ede-b163-f152add8e6fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" Oct 30 17:11:17.021494 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:16.994013 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-xjbtt" podUID="b9860f52-b69d-4de3-8999-eff8e90620dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.5:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:17.158378 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:17.149866 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" podUID="f3e1c8ff-fcc3-4a1f-9730-c617d57b5193" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded" Oct 30 17:11:17.401083 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:17.401047 4493 patch_prober.go:29] interesting pod/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" start-of-body= Oct 30 17:11:17.401348 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:17.401320 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8" podUID="56700de4-8e2f-4ede-b163-f152add8e6fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" Oct 30 17:11:17.424078 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:17.424030 4493 patch_prober.go:29] interesting pod/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" start-of-body= Oct 30 17:11:17.424708 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:17.424656 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk" podUID="26356b33-b847-4ba2-8b73-cfa37b2b14a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" Oct 30 17:11:17.425436 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:17.425418 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f844002e-4790-452b-88bf-7e33af30001e" path="/var/lib/kubelet/pods/f844002e-4790-452b-88bf-7e33af30001e/volumes" Oct 30 17:11:17.464681 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:17.464602 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" podUID="0eadc5d4-98c3-4bf2-b3c2-f73fe249182f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:17.546218 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:17.545659 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-xjbtt" Oct 30 17:11:17.551054 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:17.551031 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-xjbtt" Oct 30 17:11:17.551326 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:17.551267 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" podUID="e90f43d3-f5eb-45ab-af1e-340d0ce18966" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Oct 30 17:11:17.700858 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:17.698772 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" podUID="e76172b1-659d-45bd-91c6-1a0ce4062335" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 17:11:17.700858 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:17.699460 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" podUID="e43738f3-d55f-4c0a-a364-bf2ac76605c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 17:11:17.704112 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:17.704085 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" podUID="18b72648-f523-405e-b5cd-1266ff13f8f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 30 17:11:17.704917 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:17.704890 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" podUID="14805045-a727-41ba-8bd0-100d41f9f27d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 17:11:17.705582 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:17.705558 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" podUID="b5abe703-fee0-4185-aa22-88d1db06392b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Oct 30 17:11:17.711079 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:17.711039 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-xjbtt" Oct 30 17:11:17.928674 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:17.928635 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw"] Oct 30 17:11:17.963863 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:17.950050 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl" Oct 30 17:11:17.963863 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:17.942183 4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" podUID="0eadc5d4-98c3-4bf2-b3c2-f73fe249182f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": context deadline exceeded" Oct 30 17:11:17.963863 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:17.953666 4493 patch_prober.go:29] interesting pod/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" start-of-body= Oct 30 17:11:17.963863 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:17.953707 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb" podUID="84f45fe0-793d-4f0d-98d5-f78b9a287c5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.7:15021/healthz/ready\": dial tcp 10.134.1.7:15021: connect: connection refused" Oct 30 17:11:17.981674 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:17.971910 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl" Oct 30 17:11:17.981674 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:17.973263 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" podUID="80b9bd9a-30d6-45ff-b318-26fcdc608e5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 17:11:17.990821 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:17.988412 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" Oct 30 17:11:17.992161 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:17.992138 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl" Oct 30 17:11:17.995360 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:17.995323 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:11:17.997971 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:17.997940 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:11:17.998216 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:17.998191 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:11:18.075813 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:18.071285 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" podUID="08baad08-cd72-4305-b92d-c11a84ef276f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:18.388812 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:18.384923 4493 patch_prober.go:29] interesting pod/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" start-of-body= Oct 30 17:11:18.608925 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:18.601586 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk" podUID="26356b33-b847-4ba2-8b73-cfa37b2b14a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" Oct 30 17:11:18.608925 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:18.533731 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" podUID="0eadc5d4-98c3-4bf2-b3c2-f73fe249182f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 30 17:11:18.608925 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:18.533365 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" Oct 30 17:11:18.608925 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:18.602089 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5a6e039-m5zsq"] Oct 30 17:11:18.608925 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:18.602198 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" Oct 30 17:11:18.608925 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:18.602258 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" Oct 30 17:11:18.608925 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:18.602316 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" Oct 30 17:11:18.608925 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:18.602374 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" Oct 30 17:11:18.608925 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:18.602433 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" Oct 30 17:11:18.622590 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:18.622554 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5a6e039-m5zsq"] Oct 30 17:11:18.657153 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:18.656679 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" Oct 30 17:11:18.659093 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:18.658506 4493 patch_prober.go:29] interesting pod/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" start-of-body= Oct 30 17:11:18.659093 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:18.658574 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8" podUID="56700de4-8e2f-4ede-b163-f152add8e6fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" Oct 30 17:11:18.741057 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:18.740875 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" Oct 30 17:11:18.930809 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:18.930537 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb" Oct 30 17:11:18.937806 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:18.934726 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb" Oct 30 17:11:18.965684 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:18.965142 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd2d5a07-ba42-482e-8a22-d5dec65ddeef" path="/var/lib/kubelet/pods/fd2d5a07-ba42-482e-8a22-d5dec65ddeef/volumes" Oct 30 17:11:18.965684 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:18.965532 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb" Oct 30 17:11:19.102367 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:19.102310 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw" event={"ID":"27945ccd-4a29-4f22-a0a2-e1f663869939","Type":"ContainerStarted","Data":"00877711eaa6547d40d1869bd176550476daecb42e3b523593e66bd1aee01e77"} Oct 30 17:11:19.102559 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:19.102375 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw" event={"ID":"27945ccd-4a29-4f22-a0a2-e1f663869939","Type":"ContainerStarted","Data":"063de8b1639358a36464a2abf015b37c3ad1d03aff6836911666e295e3183e57"} Oct 30 17:11:19.356834 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:19.356740 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8" Oct 30 17:11:19.357274 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:19.357251 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8" Oct 30 17:11:19.357949 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:19.357914 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8" Oct 30 17:11:19.380986 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:19.380960 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk" Oct 30 17:11:19.381360 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:19.381339 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk" Oct 30 17:11:19.381968 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:19.381952 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk" Oct 30 17:11:19.405466 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:19.405398 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw" podStartSLOduration=5.405381567 podStartE2EDuration="5.405381567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:11:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:11:19.166285735 +0000 UTC m=+2906.393460855" watchObservedRunningTime="2026-10-30 17:11:19.405381567 +0000 UTC m=+2906.632556668" Oct 30 17:11:19.978673 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:19.978645 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw" Oct 30 17:11:19.979655 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:19.979632 4493 patch_prober.go:29] interesting pod/e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" start-of-body= Oct 30 17:11:19.979770 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:19.979683 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw" podUID="27945ccd-4a29-4f22-a0a2-e1f663869939" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" Oct 30 17:11:20.978529 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:20.978493 4493 patch_prober.go:29] interesting pod/e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" start-of-body= Oct 30 17:11:20.978703 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:20.978546 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw" podUID="27945ccd-4a29-4f22-a0a2-e1f663869939" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" Oct 30 17:11:21.978693 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:21.978661 4493 patch_prober.go:29] interesting pod/e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" start-of-body= Oct 30 17:11:21.979182 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:21.978718 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw" podUID="27945ccd-4a29-4f22-a0a2-e1f663869939" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" Oct 30 17:11:22.978775 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:22.978741 4493 patch_prober.go:29] interesting pod/e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" start-of-body= Oct 30 17:11:22.979206 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:22.978820 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw" podUID="27945ccd-4a29-4f22-a0a2-e1f663869939" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" Oct 30 17:11:23.978740 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:23.978704 4493 patch_prober.go:29] interesting pod/e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" start-of-body= Oct 30 17:11:23.979058 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:23.979025 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw" podUID="27945ccd-4a29-4f22-a0a2-e1f663869939" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" Oct 30 17:11:24.979222 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:24.979185 4493 patch_prober.go:29] interesting pod/e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" start-of-body= Oct 30 17:11:24.979692 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:24.979243 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw" podUID="27945ccd-4a29-4f22-a0a2-e1f663869939" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" Oct 30 17:11:25.940336 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:25.940296 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt"] Oct 30 17:11:25.959061 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:25.959031 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt" Oct 30 17:11:25.970508 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:25.970483 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt"] Oct 30 17:11:25.974309 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:25.974277 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-d5b798-openshift-default-dockercfg-vgn7p\"" Oct 30 17:11:25.984567 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:25.984538 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw" Oct 30 17:11:25.985002 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:25.984912 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw" Oct 30 17:11:25.985751 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:25.985721 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw" Oct 30 17:11:26.074582 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:26.074533 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f06b3212-a348-4185-b625-530f85fa91bf-credential-socket\") pod \"e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt\" (UID: \"f06b3212-a348-4185-b625-530f85fa91bf\") " pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt" Oct 30 17:11:26.074770 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:26.074621 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f06b3212-a348-4185-b625-530f85fa91bf-istiod-ca-cert\") pod \"e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt\" (UID: \"f06b3212-a348-4185-b625-530f85fa91bf\") " pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt" Oct 30 17:11:26.074770 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:26.074725 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f06b3212-a348-4185-b625-530f85fa91bf-istio-podinfo\") pod \"e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt\" (UID: \"f06b3212-a348-4185-b625-530f85fa91bf\") " pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt" Oct 30 17:11:26.074973 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:26.074888 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f06b3212-a348-4185-b625-530f85fa91bf-istio-token\") pod \"e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt\" (UID: \"f06b3212-a348-4185-b625-530f85fa91bf\") " pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt" Oct 30 17:11:26.075036 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:26.074992 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f06b3212-a348-4185-b625-530f85fa91bf-workload-socket\") pod \"e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt\" (UID: \"f06b3212-a348-4185-b625-530f85fa91bf\") " pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt" Oct 30 17:11:26.075036 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:26.075028 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f06b3212-a348-4185-b625-530f85fa91bf-workload-certs\") pod \"e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt\" (UID: \"f06b3212-a348-4185-b625-530f85fa91bf\") " pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt" Oct 30 17:11:26.075180 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:26.075162 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f06b3212-a348-4185-b625-530f85fa91bf-istio-data\") pod \"e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt\" (UID: \"f06b3212-a348-4185-b625-530f85fa91bf\") " pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt" Oct 30 17:11:26.075248 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:26.075199 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9z9l\" (UniqueName: \"kubernetes.io/projected/f06b3212-a348-4185-b625-530f85fa91bf-kube-api-access-p9z9l\") pod \"e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt\" (UID: \"f06b3212-a348-4185-b625-530f85fa91bf\") " pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt" Oct 30 17:11:26.075308 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:26.075288 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f06b3212-a348-4185-b625-530f85fa91bf-istio-envoy\") pod \"e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt\" (UID: \"f06b3212-a348-4185-b625-530f85fa91bf\") " pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt" Oct 30 17:11:26.177957 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:26.177881 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f06b3212-a348-4185-b625-530f85fa91bf-credential-socket\") pod \"e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt\" (UID: \"f06b3212-a348-4185-b625-530f85fa91bf\") " pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt" Oct 30 17:11:26.178145 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:26.177985 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f06b3212-a348-4185-b625-530f85fa91bf-istiod-ca-cert\") pod \"e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt\" (UID: \"f06b3212-a348-4185-b625-530f85fa91bf\") " pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt" Oct 30 17:11:26.178145 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:26.178056 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f06b3212-a348-4185-b625-530f85fa91bf-istio-podinfo\") pod \"e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt\" (UID: \"f06b3212-a348-4185-b625-530f85fa91bf\") " pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt" Oct 30 17:11:26.178145 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:26.178119 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f06b3212-a348-4185-b625-530f85fa91bf-istio-token\") pod \"e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt\" (UID: \"f06b3212-a348-4185-b625-530f85fa91bf\") " pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt" Oct 30 17:11:26.178312 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:26.178176 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f06b3212-a348-4185-b625-530f85fa91bf-workload-socket\") pod \"e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt\" (UID: \"f06b3212-a348-4185-b625-530f85fa91bf\") " pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt" Oct 30 17:11:26.178312 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:26.178207 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f06b3212-a348-4185-b625-530f85fa91bf-workload-certs\") pod \"e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt\" (UID: \"f06b3212-a348-4185-b625-530f85fa91bf\") " pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt" Oct 30 17:11:26.178312 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:26.178284 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f06b3212-a348-4185-b625-530f85fa91bf-istio-data\") pod \"e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt\" (UID: \"f06b3212-a348-4185-b625-530f85fa91bf\") " pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt" Oct 30 17:11:26.178474 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:26.178312 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9z9l\" (UniqueName: \"kubernetes.io/projected/f06b3212-a348-4185-b625-530f85fa91bf-kube-api-access-p9z9l\") pod \"e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt\" (UID: \"f06b3212-a348-4185-b625-530f85fa91bf\") " pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt" Oct 30 17:11:26.178474 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:26.178326 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f06b3212-a348-4185-b625-530f85fa91bf-credential-socket\") pod \"e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt\" (UID: \"f06b3212-a348-4185-b625-530f85fa91bf\") " pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt" Oct 30 17:11:26.178581 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:26.178547 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f06b3212-a348-4185-b625-530f85fa91bf-workload-socket\") pod \"e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt\" (UID: \"f06b3212-a348-4185-b625-530f85fa91bf\") " pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt" Oct 30 17:11:26.178645 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:26.178576 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f06b3212-a348-4185-b625-530f85fa91bf-istio-envoy\") pod \"e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt\" (UID: \"f06b3212-a348-4185-b625-530f85fa91bf\") " pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt" Oct 30 17:11:26.178645 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:26.178636 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f06b3212-a348-4185-b625-530f85fa91bf-workload-certs\") pod \"e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt\" (UID: \"f06b3212-a348-4185-b625-530f85fa91bf\") " pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt" Oct 30 17:11:26.178746 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:26.178723 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f06b3212-a348-4185-b625-530f85fa91bf-istiod-ca-cert\") pod \"e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt\" (UID: \"f06b3212-a348-4185-b625-530f85fa91bf\") " pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt" Oct 30 17:11:26.178866 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:26.178844 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f06b3212-a348-4185-b625-530f85fa91bf-istio-data\") pod \"e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt\" (UID: \"f06b3212-a348-4185-b625-530f85fa91bf\") " pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt" Oct 30 17:11:26.180912 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:26.180889 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f06b3212-a348-4185-b625-530f85fa91bf-istio-envoy\") pod \"e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt\" (UID: \"f06b3212-a348-4185-b625-530f85fa91bf\") " pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt" Oct 30 17:11:26.181160 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:26.181145 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f06b3212-a348-4185-b625-530f85fa91bf-istio-podinfo\") pod \"e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt\" (UID: \"f06b3212-a348-4185-b625-530f85fa91bf\") " pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt" Oct 30 17:11:26.201637 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:26.201584 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f06b3212-a348-4185-b625-530f85fa91bf-istio-token\") pod \"e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt\" (UID: \"f06b3212-a348-4185-b625-530f85fa91bf\") " pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt" Oct 30 17:11:26.201769 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:26.201746 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9z9l\" (UniqueName: \"kubernetes.io/projected/f06b3212-a348-4185-b625-530f85fa91bf-kube-api-access-p9z9l\") pod \"e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt\" (UID: \"f06b3212-a348-4185-b625-530f85fa91bf\") " pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt" Oct 30 17:11:26.277848 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:26.277817 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt" Oct 30 17:11:26.607563 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:11:26.607532 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf06b3212_a348_4185_b625_530f85fa91bf.slice/crio-7cf66eeb74645492a4db642eaf829f5ffc3caf08bfc3cea28b984baf0eb47ad8 WatchSource:0}: Error finding container 7cf66eeb74645492a4db642eaf829f5ffc3caf08bfc3cea28b984baf0eb47ad8: Status 404 returned error can't find the container with id 7cf66eeb74645492a4db642eaf829f5ffc3caf08bfc3cea28b984baf0eb47ad8 Oct 30 17:11:26.608704 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:26.608676 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt"] Oct 30 17:11:26.610930 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:26.610887 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:11:26.611023 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:26.610981 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:11:26.611065 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:26.611023 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:11:27.204719 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:27.204677 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt" event={"ID":"f06b3212-a348-4185-b625-530f85fa91bf","Type":"ContainerStarted","Data":"2a33aab82a525ef6f843527938c8af09dca8fe5f10791fc2e3639424747e59a9"} Oct 30 17:11:27.205235 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:27.204750 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt" event={"ID":"f06b3212-a348-4185-b625-530f85fa91bf","Type":"ContainerStarted","Data":"7cf66eeb74645492a4db642eaf829f5ffc3caf08bfc3cea28b984baf0eb47ad8"} Oct 30 17:11:27.279809 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:27.278751 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt" podStartSLOduration=2.27873258 podStartE2EDuration="2.27873258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:11:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:11:27.264314164 +0000 UTC m=+2914.491489264" watchObservedRunningTime="2026-10-30 17:11:27.27873258 +0000 UTC m=+2914.505907680" Oct 30 17:11:27.283864 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:27.283830 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt" Oct 30 17:11:27.288886 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:27.288853 4493 patch_prober.go:29] interesting pod/e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused" start-of-body= Oct 30 17:11:27.289049 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:27.288904 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt" podUID="f06b3212-a348-4185-b625-530f85fa91bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused" Oct 30 17:11:27.567889 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:27.554810 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 17:11:28.278873 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:28.278839 4493 patch_prober.go:29] interesting pod/e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused" start-of-body= Oct 30 17:11:28.279257 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:28.278895 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt" podUID="f06b3212-a348-4185-b625-530f85fa91bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused" Oct 30 17:11:29.290521 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:29.290475 4493 patch_prober.go:29] interesting pod/e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused" start-of-body= Oct 30 17:11:29.312173 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:29.312129 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt" podUID="f06b3212-a348-4185-b625-530f85fa91bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused" Oct 30 17:11:29.566213 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:29.565650 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" podUID="e90f43d3-f5eb-45ab-af1e-340d0ce18966" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Oct 30 17:11:29.631084 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:29.631028 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" podUID="18b72648-f523-405e-b5cd-1266ff13f8f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:29.651277 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:29.651239 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" podUID="e76172b1-659d-45bd-91c6-1a0ce4062335" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:29.676365 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:29.676321 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" podUID="e43738f3-d55f-4c0a-a364-bf2ac76605c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 17:11:29.676893 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:29.676434 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" podUID="f3e1c8ff-fcc3-4a1f-9730-c617d57b5193" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:29.808867 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:29.808819 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" podUID="0eadc5d4-98c3-4bf2-b3c2-f73fe249182f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:29.890648 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:29.890607 4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" podUID="f3e1c8ff-fcc3-4a1f-9730-c617d57b5193" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:29.912987 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:29.912941 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" podUID="80b9bd9a-30d6-45ff-b318-26fcdc608e5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:30.075895 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:30.075856 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a7f50df-xftrt"] Oct 30 17:11:30.094466 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:30.094418 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a7f50df-xftrt"] Oct 30 17:11:30.279295 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:30.279201 4493 patch_prober.go:29] interesting pod/e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused" start-of-body= Oct 30 17:11:30.279556 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:30.279526 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt" podUID="f06b3212-a348-4185-b625-530f85fa91bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused" Oct 30 17:11:30.300960 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:30.300909 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" podUID="14805045-a727-41ba-8bd0-100d41f9f27d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:30.569019 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:30.567873 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" podUID="b5abe703-fee0-4185-aa22-88d1db06392b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:30.569319 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:30.569290 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" podUID="e90f43d3-f5eb-45ab-af1e-340d0ce18966" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:30.571344 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:30.570005 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" Oct 30 17:11:30.596424 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:11:30.596277 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 17:11:30.649841 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:30.638932 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" podUID="18b72648-f523-405e-b5cd-1266ff13f8f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 30 17:11:30.652557 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:30.652168 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" podUID="e76172b1-659d-45bd-91c6-1a0ce4062335" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 17:11:30.667306 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:30.653223 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" Oct 30 17:11:30.673941 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:30.673913 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" Oct 30 17:11:30.745378 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:30.745330 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" podUID="e43738f3-d55f-4c0a-a364-bf2ac76605c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 17:11:30.797128 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:30.789933 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" Oct 30 17:11:30.797128 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:30.791097 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" podUID="f3e1c8ff-fcc3-4a1f-9730-c617d57b5193" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:30.797128 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:30.791598 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" Oct 30 17:11:30.816840 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:30.809882 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" podUID="0eadc5d4-98c3-4bf2-b3c2-f73fe249182f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:30.816840 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:30.810541 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" Oct 30 17:11:30.841872 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:30.834799 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" podUID="08baad08-cd72-4305-b92d-c11a84ef276f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:30.915810 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:30.914959 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" podUID="80b9bd9a-30d6-45ff-b318-26fcdc608e5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:30.915810 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:30.915521 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" Oct 30 17:11:31.365824 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:31.317120 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" podUID="14805045-a727-41ba-8bd0-100d41f9f27d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:31.365824 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:31.327979 4493 patch_prober.go:29] interesting pod/e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused" start-of-body= Oct 30 17:11:31.365824 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:31.328029 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt" podUID="f06b3212-a348-4185-b625-530f85fa91bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused" Oct 30 17:11:31.481207 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:31.459108 4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" podUID="18b72648-f523-405e-b5cd-1266ff13f8f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:31.570322 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:31.570281 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" podUID="e90f43d3-f5eb-45ab-af1e-340d0ce18966" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:31.571764 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:31.571732 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" podUID="b5abe703-fee0-4185-aa22-88d1db06392b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:31.586755 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:31.586729 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="136bd37d-7029-4bad-8663-091b30bcf7d4" path="/var/lib/kubelet/pods/136bd37d-7029-4bad-8663-091b30bcf7d4/volumes" Oct 30 17:11:31.587492 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:31.587476 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" Oct 30 17:11:31.588713 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:31.587948 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" Oct 30 17:11:31.617066 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:31.616972 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" Oct 30 17:11:31.640670 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:31.640644 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" Oct 30 17:11:31.653166 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:31.653122 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" podUID="18b72648-f523-405e-b5cd-1266ff13f8f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:31.659253 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:31.653399 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" Oct 30 17:11:31.670401 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:31.670374 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" Oct 30 17:11:31.723913 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:31.723378 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" Oct 30 17:11:31.758838 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:31.758747 4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" podUID="08baad08-cd72-4305-b92d-c11a84ef276f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:31.768107 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:31.768064 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" Oct 30 17:11:31.793448 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:31.793409 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" podUID="f3e1c8ff-fcc3-4a1f-9730-c617d57b5193" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded" Oct 30 17:11:31.852501 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:31.837916 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" podUID="08baad08-cd72-4305-b92d-c11a84ef276f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:31.852501 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:31.838199 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" podUID="0eadc5d4-98c3-4bf2-b3c2-f73fe249182f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 30 17:11:31.852501 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:31.838412 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" Oct 30 17:11:32.335811 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:32.281252 4493 patch_prober.go:29] interesting pod/e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused" start-of-body= Oct 30 17:11:32.335811 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:32.281518 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt" podUID="f06b3212-a348-4185-b625-530f85fa91bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused" Oct 30 17:11:32.335811 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:32.297461 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" Oct 30 17:11:32.501809 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:32.483420 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" Oct 30 17:11:32.710810 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:32.689406 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" Oct 30 17:11:32.885612 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:32.884908 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" podUID="0eadc5d4-98c3-4bf2-b3c2-f73fe249182f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:33.278768 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:33.278722 4493 patch_prober.go:29] interesting pod/e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused" start-of-body= Oct 30 17:11:33.279085 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:33.279044 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt" podUID="f06b3212-a348-4185-b625-530f85fa91bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused" Oct 30 17:11:33.661260 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:33.661196 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" podUID="18b72648-f523-405e-b5cd-1266ff13f8f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:33.673387 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:33.672919 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" podUID="e43738f3-d55f-4c0a-a364-bf2ac76605c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:33.775432 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:33.774258 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" Oct 30 17:11:34.286809 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:34.280107 4493 patch_prober.go:29] interesting pod/e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused" start-of-body= Oct 30 17:11:34.286809 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:34.280164 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt" podUID="f06b3212-a348-4185-b625-530f85fa91bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused" Oct 30 17:11:34.303135 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:34.303100 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" podUID="14805045-a727-41ba-8bd0-100d41f9f27d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 17:11:34.393074 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:34.393030 4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" podUID="14805045-a727-41ba-8bd0-100d41f9f27d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:34.673334 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:34.673290 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" podUID="e43738f3-d55f-4c0a-a364-bf2ac76605c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:34.674451 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:34.674427 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" Oct 30 17:11:34.774963 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:34.774905 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" podUID="0eadc5d4-98c3-4bf2-b3c2-f73fe249182f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:35.018875 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:34.987707 4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" podUID="e43738f3-d55f-4c0a-a364-bf2ac76605c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": context deadline exceeded" Oct 30 17:11:35.285742 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:35.285654 4493 patch_prober.go:29] interesting pod/e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused" start-of-body= Oct 30 17:11:35.320904 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:35.303938 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" podUID="14805045-a727-41ba-8bd0-100d41f9f27d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:35.321464 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:35.321423 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" Oct 30 17:11:35.346351 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:35.342693 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt" podUID="f06b3212-a348-4185-b625-530f85fa91bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused" Oct 30 17:11:35.475425 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:35.475378 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" podUID="f3e1c8ff-fcc3-4a1f-9730-c617d57b5193" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:35.593621 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:35.593515 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" podUID="e90f43d3-f5eb-45ab-af1e-340d0ce18966" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Oct 30 17:11:35.677917 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:35.677862 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" podUID="e43738f3-d55f-4c0a-a364-bf2ac76605c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:35.691070 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:35.691027 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" podUID="e76172b1-659d-45bd-91c6-1a0ce4062335" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:35.691723 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:35.691695 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" podUID="b5abe703-fee0-4185-aa22-88d1db06392b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:35.781712 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:35.779892 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" podUID="0eadc5d4-98c3-4bf2-b3c2-f73fe249182f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:35.781712 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:35.780601 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" Oct 30 17:11:36.001100 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:35.980409 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" Oct 30 17:11:36.303904 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:36.290876 4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" podUID="e76172b1-659d-45bd-91c6-1a0ce4062335" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:36.335471 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:36.335432 4493 patch_prober.go:29] interesting pod/e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused" start-of-body= Oct 30 17:11:36.340774 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:36.340728 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt" podUID="f06b3212-a348-4185-b625-530f85fa91bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused" Oct 30 17:11:36.344129 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:36.341758 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" podUID="14805045-a727-41ba-8bd0-100d41f9f27d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:36.351526 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:36.351501 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" Oct 30 17:11:36.637048 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:36.636966 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" Oct 30 17:11:37.283993 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:37.283963 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt" Oct 30 17:11:37.284431 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:37.284411 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt" Oct 30 17:11:37.285316 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:37.285293 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt" Oct 30 17:11:42.033268 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:42.033231 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efa5edd8-bcdch"] Oct 30 17:11:42.045443 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:42.045412 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efa5edd8-bcdch"] Oct 30 17:11:42.936535 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:42.936492 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0262a990-e664-4be8-8925-6858a7ac443e" path="/var/lib/kubelet/pods/0262a990-e664-4be8-8925-6858a7ac443e/volumes" Oct 30 17:11:56.659126 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:56.659093 4493 scope.go:122] "RemoveContainer" containerID="56354649c121f92c56a239868751842cc87dd67ac06435c92869fd0613f8a1e8" Oct 30 17:11:56.674873 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:56.674848 4493 scope.go:122] "RemoveContainer" containerID="ba667739adfdec4ead3887b6573ad1accb61190d81df09ebbfbe9cfbeb33c3f8" Oct 30 17:11:56.685734 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:56.685715 4493 scope.go:122] "RemoveContainer" containerID="78a9c0a768de39ccbfb96f9e83e15f4d7a8c516758cb8edbabb07d68e5e27439" Oct 30 17:11:56.699624 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:11:56.699601 4493 scope.go:122] "RemoveContainer" containerID="b052fa23578e2658eb99f86c22ebe840fc105a3a96dbf6314d17ec09258f1cb9" Oct 30 17:12:00.342881 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:12:00.342841 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 17:12:24.784230 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:12:24.771888 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 17:12:30.361440 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:12:30.361307 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 17:12:36.827950 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:12:36.827908 4493 patch_prober.go:29] interesting pod/e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:12:36.828598 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:12:36.827982 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg" podUID="c5146890-4f44-4e48-9147-dc215b8195d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:12:40.034341 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:12:40.034300 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79201fa2-p88h9"] Oct 30 17:12:40.045425 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:12:40.045394 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79201fa2-p88h9"] Oct 30 17:12:40.936343 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:12:40.936306 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2408796b-b549-4ac6-a71f-67da68b91a68" path="/var/lib/kubelet/pods/2408796b-b549-4ac6-a71f-67da68b91a68/volumes" Oct 30 17:12:43.358241 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:12:43.357963 4493 generic.go:356] "Generic (PLEG): container finished" podID="c5146890-4f44-4e48-9147-dc215b8195d9" containerID="f3d935c69434a834596a8b60e289ddca19d0a6132e8ffc572e33f8952aa1ae38" exitCode=0 Oct 30 17:12:43.358241 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:12:43.358026 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg" event={"ID":"c5146890-4f44-4e48-9147-dc215b8195d9","Type":"ContainerDied","Data":"f3d935c69434a834596a8b60e289ddca19d0a6132e8ffc572e33f8952aa1ae38"} Oct 30 17:12:43.358900 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:12:43.358538 4493 scope.go:122] "RemoveContainer" containerID="f3d935c69434a834596a8b60e289ddca19d0a6132e8ffc572e33f8952aa1ae38" Oct 30 17:12:43.365288 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:12:43.365249 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:12:43.365408 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:12:43.365346 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:12:43.365408 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:12:43.365395 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:12:44.350682 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:12:44.342144 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw"] Oct 30 17:12:44.395808 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:12:44.371172 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw"] Oct 30 17:12:44.434895 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:12:44.400876 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw" Oct 30 17:12:44.517816 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:12:44.462722 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-24acb2-openshift-default-dockercfg-nrq8m\"" Oct 30 17:12:44.550809 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:12:44.519375 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66c82e49-cc58-4f4c-b090-78dbcbdff588-workload-socket\") pod \"e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw\" (UID: \"66c82e49-cc58-4f4c-b090-78dbcbdff588\") " pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw" Oct 30 17:12:44.550809 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:12:44.519504 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66c82e49-cc58-4f4c-b090-78dbcbdff588-credential-socket\") pod \"e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw\" (UID: \"66c82e49-cc58-4f4c-b090-78dbcbdff588\") " pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw" Oct 30 17:12:44.550809 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:12:44.519554 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66c82e49-cc58-4f4c-b090-78dbcbdff588-istio-data\") pod \"e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw\" (UID: \"66c82e49-cc58-4f4c-b090-78dbcbdff588\") " pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw" Oct 30 17:12:44.550809 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:12:44.519610 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66c82e49-cc58-4f4c-b090-78dbcbdff588-istio-token\") pod \"e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw\" (UID: \"66c82e49-cc58-4f4c-b090-78dbcbdff588\") " pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw" Oct 30 17:12:44.550809 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:12:44.519742 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66c82e49-cc58-4f4c-b090-78dbcbdff588-istio-envoy\") pod \"e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw\" (UID: \"66c82e49-cc58-4f4c-b090-78dbcbdff588\") " pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw" Oct 30 17:12:44.550809 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:12:44.519773 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66c82e49-cc58-4f4c-b090-78dbcbdff588-istio-podinfo\") pod \"e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw\" (UID: \"66c82e49-cc58-4f4c-b090-78dbcbdff588\") " pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw" Oct 30 17:12:44.550809 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:12:44.519920 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66c82e49-cc58-4f4c-b090-78dbcbdff588-workload-certs\") pod \"e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw\" (UID: \"66c82e49-cc58-4f4c-b090-78dbcbdff588\") " pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw" Oct 30 17:12:44.550809 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:12:44.519950 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th87s\" (UniqueName: \"kubernetes.io/projected/66c82e49-cc58-4f4c-b090-78dbcbdff588-kube-api-access-th87s\") pod \"e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw\" (UID: \"66c82e49-cc58-4f4c-b090-78dbcbdff588\") " pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw" Oct 30 17:12:44.550809 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:12:44.519981 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66c82e49-cc58-4f4c-b090-78dbcbdff588-istiod-ca-cert\") pod \"e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw\" (UID: \"66c82e49-cc58-4f4c-b090-78dbcbdff588\") " pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw" Oct 30 17:12:44.648880 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:12:44.627754 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66c82e49-cc58-4f4c-b090-78dbcbdff588-istio-data\") pod \"e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw\" (UID: \"66c82e49-cc58-4f4c-b090-78dbcbdff588\") " pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw" Oct 30 17:12:44.659854 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:12:44.649160 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66c82e49-cc58-4f4c-b090-78dbcbdff588-istio-token\") pod \"e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw\" (UID: \"66c82e49-cc58-4f4c-b090-78dbcbdff588\") " pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw" Oct 30 17:12:44.686038 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:12:44.685997 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66c82e49-cc58-4f4c-b090-78dbcbdff588-istio-envoy\") pod \"e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw\" (UID: \"66c82e49-cc58-4f4c-b090-78dbcbdff588\") " pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw" Oct 30 17:12:44.710804 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:12:44.697914 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66c82e49-cc58-4f4c-b090-78dbcbdff588-istio-podinfo\") pod \"e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw\" (UID: \"66c82e49-cc58-4f4c-b090-78dbcbdff588\") " pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw" Oct 30 17:12:44.723258 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:12:44.711185 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66c82e49-cc58-4f4c-b090-78dbcbdff588-workload-certs\") pod \"e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw\" (UID: \"66c82e49-cc58-4f4c-b090-78dbcbdff588\") " pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw" Oct 30 17:12:44.724090 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:12:44.724061 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th87s\" (UniqueName: \"kubernetes.io/projected/66c82e49-cc58-4f4c-b090-78dbcbdff588-kube-api-access-th87s\") pod \"e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw\" (UID: \"66c82e49-cc58-4f4c-b090-78dbcbdff588\") " pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw" Oct 30 17:12:44.724651 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:12:44.699024 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66c82e49-cc58-4f4c-b090-78dbcbdff588-istio-token\") pod \"e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw\" (UID: \"66c82e49-cc58-4f4c-b090-78dbcbdff588\") " pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw" Oct 30 17:12:44.725585 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:12:44.723926 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg" event={"ID":"c5146890-4f44-4e48-9147-dc215b8195d9","Type":"ContainerStarted","Data":"f552982c48d33c757781f7056c3edbdf024c507caac153eda2ce0f0e5a06df51"} Oct 30 17:12:44.726900 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:12:44.641121 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66c82e49-cc58-4f4c-b090-78dbcbdff588-istio-data\") pod \"e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw\" (UID: \"66c82e49-cc58-4f4c-b090-78dbcbdff588\") " pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw" Oct 30 17:12:44.727065 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:12:44.724843 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66c82e49-cc58-4f4c-b090-78dbcbdff588-workload-certs\") pod \"e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw\" (UID: \"66c82e49-cc58-4f4c-b090-78dbcbdff588\") " pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw" Oct 30 17:12:44.727165 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:12:44.725526 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66c82e49-cc58-4f4c-b090-78dbcbdff588-istiod-ca-cert\") pod \"e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw\" (UID: \"66c82e49-cc58-4f4c-b090-78dbcbdff588\") " pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw" Oct 30 17:12:44.727277 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:12:44.724611 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66c82e49-cc58-4f4c-b090-78dbcbdff588-istiod-ca-cert\") pod \"e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw\" (UID: \"66c82e49-cc58-4f4c-b090-78dbcbdff588\") " pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw" Oct 30 17:12:44.727491 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:12:44.727475 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66c82e49-cc58-4f4c-b090-78dbcbdff588-workload-socket\") pod \"e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw\" (UID: \"66c82e49-cc58-4f4c-b090-78dbcbdff588\") " pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw" Oct 30 17:12:44.731974 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:12:44.731952 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66c82e49-cc58-4f4c-b090-78dbcbdff588-credential-socket\") pod \"e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw\" (UID: \"66c82e49-cc58-4f4c-b090-78dbcbdff588\") " pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw" Oct 30 17:12:44.732642 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:12:44.732397 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66c82e49-cc58-4f4c-b090-78dbcbdff588-credential-socket\") pod \"e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw\" (UID: \"66c82e49-cc58-4f4c-b090-78dbcbdff588\") " pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw" Oct 30 17:12:44.732771 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:12:44.732488 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66c82e49-cc58-4f4c-b090-78dbcbdff588-workload-socket\") pod \"e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw\" (UID: \"66c82e49-cc58-4f4c-b090-78dbcbdff588\") " pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw" Oct 30 17:12:44.791844 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:12:44.784582 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-th87s\" (UniqueName: \"kubernetes.io/projected/66c82e49-cc58-4f4c-b090-78dbcbdff588-kube-api-access-th87s\") pod \"e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw\" (UID: \"66c82e49-cc58-4f4c-b090-78dbcbdff588\") " pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw" Oct 30 17:12:44.792725 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:12:44.792691 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66c82e49-cc58-4f4c-b090-78dbcbdff588-istio-envoy\") pod \"e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw\" (UID: \"66c82e49-cc58-4f4c-b090-78dbcbdff588\") " pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw" Oct 30 17:12:44.808435 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:12:44.808365 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66c82e49-cc58-4f4c-b090-78dbcbdff588-istio-podinfo\") pod \"e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw\" (UID: \"66c82e49-cc58-4f4c-b090-78dbcbdff588\") " pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw" Oct 30 17:12:44.869734 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:12:44.854512 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw" Oct 30 17:12:44.993392 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:12:44.993161 4493 patch_prober.go:29] interesting pod/e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body= Oct 30 17:12:44.993971 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:12:44.993847 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg" podUID="c5146890-4f44-4e48-9147-dc215b8195d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" Oct 30 17:12:45.043808 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:12:45.041483 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg" Oct 30 17:12:45.592935 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:12:45.592904 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw"] Oct 30 17:12:45.594312 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:12:45.594280 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66c82e49_cc58_4f4c_b090_78dbcbdff588.slice/crio-1cabd735f05d364aa6672fb8d27fa6cfb309c531b3ce3d158a93111bfe0f3439 WatchSource:0}: Error finding container 1cabd735f05d364aa6672fb8d27fa6cfb309c531b3ce3d158a93111bfe0f3439: Status 404 returned error can't find the container with id 1cabd735f05d364aa6672fb8d27fa6cfb309c531b3ce3d158a93111bfe0f3439 Oct 30 17:12:45.597715 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:12:45.597681 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:12:45.597846 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:12:45.597764 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:12:45.597907 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:12:45.597842 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:12:45.749809 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:12:45.749696 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw" event={"ID":"66c82e49-cc58-4f4c-b090-78dbcbdff588","Type":"ContainerStarted","Data":"1cabd735f05d364aa6672fb8d27fa6cfb309c531b3ce3d158a93111bfe0f3439"} Oct 30 17:12:45.985829 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:12:45.985315 4493 patch_prober.go:29] interesting pod/e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body= Oct 30 17:12:45.985829 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:12:45.985362 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg" podUID="c5146890-4f44-4e48-9147-dc215b8195d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" Oct 30 17:12:46.760211 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:12:46.760174 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw" event={"ID":"66c82e49-cc58-4f4c-b090-78dbcbdff588","Type":"ContainerStarted","Data":"847c48a76d37d99d788a15fb5702dfb7e31d66d156572538fbc95c640f105702"} Oct 30 17:12:46.870498 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:12:46.870461 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw" Oct 30 17:12:46.878727 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:12:46.878697 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw" Oct 30 17:12:46.995421 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:12:46.995386 4493 patch_prober.go:29] interesting pod/e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body= Oct 30 17:12:46.995776 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:12:46.995745 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg" podUID="c5146890-4f44-4e48-9147-dc215b8195d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" Oct 30 17:12:47.067863 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:12:47.067559 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw" podStartSLOduration=3.067532113 podStartE2EDuration="3.067532113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:12:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:12:46.83222837 +0000 UTC m=+2994.059403471" watchObservedRunningTime="2026-10-30 17:12:47.067532113 +0000 UTC m=+2994.294707214" Oct 30 17:12:47.771765 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:12:47.770710 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw" Oct 30 17:12:47.771765 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:12:47.771612 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw" Oct 30 17:12:47.985546 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:12:47.985513 4493 patch_prober.go:29] interesting pod/e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body= Oct 30 17:12:47.985755 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:12:47.985567 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg" podUID="c5146890-4f44-4e48-9147-dc215b8195d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" Oct 30 17:12:48.985205 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:12:48.985173 4493 patch_prober.go:29] interesting pod/e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body= Oct 30 17:12:48.985615 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:12:48.985230 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg" podUID="c5146890-4f44-4e48-9147-dc215b8195d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" Oct 30 17:12:49.984694 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:12:49.984655 4493 patch_prober.go:29] interesting pod/e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body= Oct 30 17:12:49.984924 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:12:49.984732 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg" podUID="c5146890-4f44-4e48-9147-dc215b8195d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" Oct 30 17:12:50.787201 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:12:50.787167 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg" Oct 30 17:12:50.992005 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:12:50.991967 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg" Oct 30 17:12:50.994133 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:12:50.994110 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg" Oct 30 17:12:54.781039 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:12:54.781013 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-56v9s_d3a1c417-2698-4462-b7ff-1d2e66d7a10f/ovn-acl-logging/0.log" Oct 30 17:12:54.791520 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:12:54.791489 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-56v9s_d3a1c417-2698-4462-b7ff-1d2e66d7a10f/ovn-acl-logging/0.log" Oct 30 17:12:56.762954 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:12:56.762916 4493 scope.go:122] "RemoveContainer" containerID="6c0ab7b5540f82977a83506f613bb1b0f1ab4355c61c78c09a10d8f64ff4aa56" Oct 30 17:12:56.778074 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:12:56.778051 4493 scope.go:122] "RemoveContainer" containerID="84791a8c1aa9ee3e28869fe2e5d77926d098fcc1e2f671fa87ce882043ca9082" Oct 30 17:12:56.792904 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:12:56.792873 4493 scope.go:122] "RemoveContainer" containerID="800b402f1f1906601d64101df2d5630e62bfde1851485129f70f18e5bb9520dd" Oct 30 17:12:56.804285 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:12:56.804262 4493 scope.go:122] "RemoveContainer" containerID="cf4e0bcc91e29125473957f539063d00433d62e34aea80f9c03dcf1ca7000d58" Oct 30 17:12:56.818272 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:12:56.818245 4493 scope.go:122] "RemoveContainer" containerID="77129fa5a1b5bd04b207ca258f437a95bbf0871bec89d90ceec39827fd436cb9" Oct 30 17:13:00.347302 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:13:00.347246 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 17:13:23.036413 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:23.036372 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1ff3a7a-h52f7"] Oct 30 17:13:23.046201 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:23.046171 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1ff3a7a-h52f7"] Oct 30 17:13:23.416610 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:23.416578 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 17:13:24.948581 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:24.948540 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43d23a76-3bc5-44a6-b830-3e838afe638e" path="/var/lib/kubelet/pods/43d23a76-3bc5-44a6-b830-3e838afe638e/volumes" Oct 30 17:13:27.206877 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:27.206841 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f6949a3-l2mvr"] Oct 30 17:13:27.211586 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:27.211566 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6949a3-l2mvr" Oct 30 17:13:27.240547 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:27.240513 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f6949a3-l2mvr"] Oct 30 17:13:27.242701 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:27.242668 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 17:13:27.242701 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:27.242693 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:13:27.242883 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:27.242695 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:13:27.243172 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:27.243155 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:13:27.289132 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:27.289095 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4821fb3-ff5f-4095-ab78-f33f56fc464d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f6949a3-l2mvr\" (UID: \"b4821fb3-ff5f-4095-ab78-f33f56fc464d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6949a3-l2mvr" Oct 30 17:13:27.289461 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:27.289440 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcp5v\" (UniqueName: \"kubernetes.io/projected/b4821fb3-ff5f-4095-ab78-f33f56fc464d-kube-api-access-jcp5v\") pod \"maas-api-revoke-sub-8f6949a3-l2mvr\" (UID: \"b4821fb3-ff5f-4095-ab78-f33f56fc464d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6949a3-l2mvr" Oct 30 17:13:27.289641 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:27.289625 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4821fb3-ff5f-4095-ab78-f33f56fc464d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f6949a3-l2mvr\" (UID: \"b4821fb3-ff5f-4095-ab78-f33f56fc464d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6949a3-l2mvr" Oct 30 17:13:27.325904 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:27.325859 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2cc23d3-fb7n8"] Oct 30 17:13:27.332037 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:27.332010 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2cc23d3-fb7n8" Oct 30 17:13:27.348006 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:27.347979 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2cc23d3-fb7n8"] Oct 30 17:13:27.393908 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:27.393860 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4821fb3-ff5f-4095-ab78-f33f56fc464d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f6949a3-l2mvr\" (UID: \"b4821fb3-ff5f-4095-ab78-f33f56fc464d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6949a3-l2mvr" Oct 30 17:13:27.394186 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:27.393959 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdrdl\" (UniqueName: \"kubernetes.io/projected/a7dfcd4f-5c74-41c1-926f-02a3d1b525e5-kube-api-access-hdrdl\") pod \"maas-api-revoke-sub-e2cc23d3-fb7n8\" (UID: \"a7dfcd4f-5c74-41c1-926f-02a3d1b525e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2cc23d3-fb7n8" Oct 30 17:13:27.394186 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:27.394027 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcp5v\" (UniqueName: \"kubernetes.io/projected/b4821fb3-ff5f-4095-ab78-f33f56fc464d-kube-api-access-jcp5v\") pod \"maas-api-revoke-sub-8f6949a3-l2mvr\" (UID: \"b4821fb3-ff5f-4095-ab78-f33f56fc464d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6949a3-l2mvr" Oct 30 17:13:27.394186 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:27.394097 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4821fb3-ff5f-4095-ab78-f33f56fc464d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f6949a3-l2mvr\" (UID: \"b4821fb3-ff5f-4095-ab78-f33f56fc464d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6949a3-l2mvr" Oct 30 17:13:27.394186 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:27.394132 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7dfcd4f-5c74-41c1-926f-02a3d1b525e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2cc23d3-fb7n8\" (UID: \"a7dfcd4f-5c74-41c1-926f-02a3d1b525e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2cc23d3-fb7n8" Oct 30 17:13:27.394555 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:27.394201 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7dfcd4f-5c74-41c1-926f-02a3d1b525e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2cc23d3-fb7n8\" (UID: \"a7dfcd4f-5c74-41c1-926f-02a3d1b525e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2cc23d3-fb7n8" Oct 30 17:13:27.395520 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:27.395497 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4821fb3-ff5f-4095-ab78-f33f56fc464d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f6949a3-l2mvr\" (UID: \"b4821fb3-ff5f-4095-ab78-f33f56fc464d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6949a3-l2mvr" Oct 30 17:13:27.397619 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:27.397592 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4821fb3-ff5f-4095-ab78-f33f56fc464d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f6949a3-l2mvr\" (UID: \"b4821fb3-ff5f-4095-ab78-f33f56fc464d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6949a3-l2mvr" Oct 30 17:13:27.419498 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:27.419450 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcp5v\" (UniqueName: \"kubernetes.io/projected/b4821fb3-ff5f-4095-ab78-f33f56fc464d-kube-api-access-jcp5v\") pod \"maas-api-revoke-sub-8f6949a3-l2mvr\" (UID: \"b4821fb3-ff5f-4095-ab78-f33f56fc464d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6949a3-l2mvr" Oct 30 17:13:27.498912 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:27.496642 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdrdl\" (UniqueName: \"kubernetes.io/projected/a7dfcd4f-5c74-41c1-926f-02a3d1b525e5-kube-api-access-hdrdl\") pod \"maas-api-revoke-sub-e2cc23d3-fb7n8\" (UID: \"a7dfcd4f-5c74-41c1-926f-02a3d1b525e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2cc23d3-fb7n8" Oct 30 17:13:27.498912 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:27.496777 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7dfcd4f-5c74-41c1-926f-02a3d1b525e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2cc23d3-fb7n8\" (UID: \"a7dfcd4f-5c74-41c1-926f-02a3d1b525e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2cc23d3-fb7n8" Oct 30 17:13:27.498912 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:27.496865 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7dfcd4f-5c74-41c1-926f-02a3d1b525e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2cc23d3-fb7n8\" (UID: \"a7dfcd4f-5c74-41c1-926f-02a3d1b525e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2cc23d3-fb7n8" Oct 30 17:13:27.499313 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:27.499283 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7dfcd4f-5c74-41c1-926f-02a3d1b525e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2cc23d3-fb7n8\" (UID: \"a7dfcd4f-5c74-41c1-926f-02a3d1b525e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2cc23d3-fb7n8" Oct 30 17:13:27.502129 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:27.502105 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7dfcd4f-5c74-41c1-926f-02a3d1b525e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2cc23d3-fb7n8\" (UID: \"a7dfcd4f-5c74-41c1-926f-02a3d1b525e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2cc23d3-fb7n8" Oct 30 17:13:27.519687 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:27.519660 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdrdl\" (UniqueName: \"kubernetes.io/projected/a7dfcd4f-5c74-41c1-926f-02a3d1b525e5-kube-api-access-hdrdl\") pod \"maas-api-revoke-sub-e2cc23d3-fb7n8\" (UID: \"a7dfcd4f-5c74-41c1-926f-02a3d1b525e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2cc23d3-fb7n8" Oct 30 17:13:27.526565 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:27.526538 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6949a3-l2mvr" Oct 30 17:13:27.655992 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:27.655964 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2cc23d3-fb7n8" Oct 30 17:13:27.856540 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:27.855805 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f6949a3-l2mvr"] Oct 30 17:13:27.871717 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:13:27.871686 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4821fb3_ff5f_4095_ab78_f33f56fc464d.slice/crio-20082cefa1259e7fa96427efc168918a89b591debc692eccac3b6f7d21ce0473 WatchSource:0}: Error finding container 20082cefa1259e7fa96427efc168918a89b591debc692eccac3b6f7d21ce0473: Status 404 returned error can't find the container with id 20082cefa1259e7fa96427efc168918a89b591debc692eccac3b6f7d21ce0473 Oct 30 17:13:28.013331 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:28.013301 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2cc23d3-fb7n8"] Oct 30 17:13:28.106670 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:13:28.106639 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7dfcd4f_5c74_41c1_926f_02a3d1b525e5.slice/crio-aad3dbeb79a699528103bae14e42593575912abedc57cb3e83051ab2df83f6a0 WatchSource:0}: Error finding container aad3dbeb79a699528103bae14e42593575912abedc57cb3e83051ab2df83f6a0: Status 404 returned error can't find the container with id aad3dbeb79a699528103bae14e42593575912abedc57cb3e83051ab2df83f6a0 Oct 30 17:13:28.144677 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:28.144651 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0fbf5ac-69flm"] Oct 30 17:13:28.150947 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:28.150925 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fbf5ac-69flm" Oct 30 17:13:28.171834 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:28.171806 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0fbf5ac-69flm"] Oct 30 17:13:28.208424 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:28.208389 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aad374be-4bce-4729-bf69-73a4e855b79b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0fbf5ac-69flm\" (UID: \"aad374be-4bce-4729-bf69-73a4e855b79b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fbf5ac-69flm" Oct 30 17:13:28.208970 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:28.208584 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4484\" (UniqueName: \"kubernetes.io/projected/aad374be-4bce-4729-bf69-73a4e855b79b-kube-api-access-w4484\") pod \"maas-api-revoke-sub-f0fbf5ac-69flm\" (UID: \"aad374be-4bce-4729-bf69-73a4e855b79b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fbf5ac-69flm" Oct 30 17:13:28.208970 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:28.208842 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aad374be-4bce-4729-bf69-73a4e855b79b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0fbf5ac-69flm\" (UID: \"aad374be-4bce-4729-bf69-73a4e855b79b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fbf5ac-69flm" Oct 30 17:13:28.219821 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:28.219778 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6949a3-l2mvr" event={"ID":"b4821fb3-ff5f-4095-ab78-f33f56fc464d","Type":"ContainerStarted","Data":"20082cefa1259e7fa96427efc168918a89b591debc692eccac3b6f7d21ce0473"} Oct 30 17:13:28.221488 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:28.221467 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2cc23d3-fb7n8" event={"ID":"a7dfcd4f-5c74-41c1-926f-02a3d1b525e5","Type":"ContainerStarted","Data":"aad3dbeb79a699528103bae14e42593575912abedc57cb3e83051ab2df83f6a0"} Oct 30 17:13:28.309909 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:28.309800 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aad374be-4bce-4729-bf69-73a4e855b79b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0fbf5ac-69flm\" (UID: \"aad374be-4bce-4729-bf69-73a4e855b79b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fbf5ac-69flm" Oct 30 17:13:28.310056 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:28.309940 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4484\" (UniqueName: \"kubernetes.io/projected/aad374be-4bce-4729-bf69-73a4e855b79b-kube-api-access-w4484\") pod \"maas-api-revoke-sub-f0fbf5ac-69flm\" (UID: \"aad374be-4bce-4729-bf69-73a4e855b79b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fbf5ac-69flm" Oct 30 17:13:28.310134 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:28.310115 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aad374be-4bce-4729-bf69-73a4e855b79b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0fbf5ac-69flm\" (UID: \"aad374be-4bce-4729-bf69-73a4e855b79b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fbf5ac-69flm" Oct 30 17:13:28.310667 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:28.310647 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aad374be-4bce-4729-bf69-73a4e855b79b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0fbf5ac-69flm\" (UID: \"aad374be-4bce-4729-bf69-73a4e855b79b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fbf5ac-69flm" Oct 30 17:13:28.312851 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:28.312829 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aad374be-4bce-4729-bf69-73a4e855b79b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0fbf5ac-69flm\" (UID: \"aad374be-4bce-4729-bf69-73a4e855b79b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fbf5ac-69flm" Oct 30 17:13:28.333023 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:28.332993 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4484\" (UniqueName: \"kubernetes.io/projected/aad374be-4bce-4729-bf69-73a4e855b79b-kube-api-access-w4484\") pod \"maas-api-revoke-sub-f0fbf5ac-69flm\" (UID: \"aad374be-4bce-4729-bf69-73a4e855b79b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fbf5ac-69flm" Oct 30 17:13:28.465854 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:28.465826 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fbf5ac-69flm" Oct 30 17:13:28.868374 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:28.868342 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0fbf5ac-69flm"] Oct 30 17:13:29.292808 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:29.291894 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fbf5ac-69flm" event={"ID":"aad374be-4bce-4729-bf69-73a4e855b79b","Type":"ContainerStarted","Data":"09a97e4a22ebd46145678441d310371204d8ccfbc258bd52e5c111594431047e"} Oct 30 17:13:29.350912 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:29.350871 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6949a3-l2mvr" event={"ID":"b4821fb3-ff5f-4095-ab78-f33f56fc464d","Type":"ContainerStarted","Data":"85f85e13ee8b9bcaa6953c50189126b749b1fa31ec160862c8103c167d6263cc"} Oct 30 17:13:29.496757 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:29.416627 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6949a3-l2mvr" podStartSLOduration=2.4166052430000002 podStartE2EDuration="2.416605243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:13:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:13:29.406834575 +0000 UTC m=+3036.634009685" watchObservedRunningTime="2026-10-30 17:13:29.416605243 +0000 UTC m=+3036.643780344" Oct 30 17:13:29.496757 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:29.438124 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2cc23d3-fb7n8" event={"ID":"a7dfcd4f-5c74-41c1-926f-02a3d1b525e5","Type":"ContainerStarted","Data":"61557f4679b0e265b57f611a179bd705cc85b3baa5904e363f88b609179820f9"} Oct 30 17:13:29.575398 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:29.575338 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2cc23d3-fb7n8" podStartSLOduration=2.575315915 podStartE2EDuration="2.575315915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:13:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:13:29.486387214 +0000 UTC m=+3036.713562316" watchObservedRunningTime="2026-10-30 17:13:29.575315915 +0000 UTC m=+3036.802491017" Oct 30 17:13:30.342347 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:13:30.342306 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 17:13:30.450305 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:30.450268 4493 generic.go:356] "Generic (PLEG): container finished" podID="aad374be-4bce-4729-bf69-73a4e855b79b" containerID="b82046b32be1ba0aff8189faba1a0520f8aa3c793af33125a4e621e370149644" exitCode=0 Oct 30 17:13:30.450511 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:30.450376 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fbf5ac-69flm" event={"ID":"aad374be-4bce-4729-bf69-73a4e855b79b","Type":"ContainerDied","Data":"b82046b32be1ba0aff8189faba1a0520f8aa3c793af33125a4e621e370149644"} Oct 30 17:13:30.453719 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:30.453521 4493 generic.go:356] "Generic (PLEG): container finished" podID="b4821fb3-ff5f-4095-ab78-f33f56fc464d" containerID="85f85e13ee8b9bcaa6953c50189126b749b1fa31ec160862c8103c167d6263cc" exitCode=0 Oct 30 17:13:30.453719 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:30.453640 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6949a3-l2mvr" event={"ID":"b4821fb3-ff5f-4095-ab78-f33f56fc464d","Type":"ContainerDied","Data":"85f85e13ee8b9bcaa6953c50189126b749b1fa31ec160862c8103c167d6263cc"} Oct 30 17:13:30.455878 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:30.455855 4493 generic.go:356] "Generic (PLEG): container finished" podID="a7dfcd4f-5c74-41c1-926f-02a3d1b525e5" containerID="61557f4679b0e265b57f611a179bd705cc85b3baa5904e363f88b609179820f9" exitCode=0 Oct 30 17:13:30.456029 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:30.455939 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2cc23d3-fb7n8" event={"ID":"a7dfcd4f-5c74-41c1-926f-02a3d1b525e5","Type":"ContainerDied","Data":"61557f4679b0e265b57f611a179bd705cc85b3baa5904e363f88b609179820f9"} Oct 30 17:13:31.671568 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:31.671526 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6949a3-l2mvr" Oct 30 17:13:31.736566 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:31.736536 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2cc23d3-fb7n8" Oct 30 17:13:31.741524 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:31.741491 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fbf5ac-69flm" Oct 30 17:13:31.764360 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:31.764336 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4821fb3-ff5f-4095-ab78-f33f56fc464d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4821fb3-ff5f-4095-ab78-f33f56fc464d-maas-api-openshift-ca\") pod \"b4821fb3-ff5f-4095-ab78-f33f56fc464d\" (UID: \"b4821fb3-ff5f-4095-ab78-f33f56fc464d\") " Oct 30 17:13:31.764497 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:31.764377 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aad374be-4bce-4729-bf69-73a4e855b79b-kube-api-access-w4484\" (UniqueName: \"kubernetes.io/projected/aad374be-4bce-4729-bf69-73a4e855b79b-kube-api-access-w4484\") pod \"aad374be-4bce-4729-bf69-73a4e855b79b\" (UID: \"aad374be-4bce-4729-bf69-73a4e855b79b\") " Oct 30 17:13:31.764600 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:31.764579 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aad374be-4bce-4729-bf69-73a4e855b79b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aad374be-4bce-4729-bf69-73a4e855b79b-maas-api-xks-ca\") pod \"aad374be-4bce-4729-bf69-73a4e855b79b\" (UID: \"aad374be-4bce-4729-bf69-73a4e855b79b\") " Oct 30 17:13:31.764715 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:31.764699 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4821fb3-ff5f-4095-ab78-f33f56fc464d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4821fb3-ff5f-4095-ab78-f33f56fc464d-maas-api-xks-ca\") pod \"b4821fb3-ff5f-4095-ab78-f33f56fc464d\" (UID: \"b4821fb3-ff5f-4095-ab78-f33f56fc464d\") " Oct 30 17:13:31.764802 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:31.764705 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4821fb3-ff5f-4095-ab78-f33f56fc464d-maas-api-openshift-ca" pod "b4821fb3-ff5f-4095-ab78-f33f56fc464d" (UID: "b4821fb3-ff5f-4095-ab78-f33f56fc464d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:13:31.764802 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:31.764736 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7dfcd4f-5c74-41c1-926f-02a3d1b525e5-kube-api-access-hdrdl\" (UniqueName: \"kubernetes.io/projected/a7dfcd4f-5c74-41c1-926f-02a3d1b525e5-kube-api-access-hdrdl\") pod \"a7dfcd4f-5c74-41c1-926f-02a3d1b525e5\" (UID: \"a7dfcd4f-5c74-41c1-926f-02a3d1b525e5\") " Oct 30 17:13:31.764921 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:31.764892 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7dfcd4f-5c74-41c1-926f-02a3d1b525e5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7dfcd4f-5c74-41c1-926f-02a3d1b525e5-maas-api-openshift-ca\") pod \"a7dfcd4f-5c74-41c1-926f-02a3d1b525e5\" (UID: \"a7dfcd4f-5c74-41c1-926f-02a3d1b525e5\") " Oct 30 17:13:31.764982 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:31.764938 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4821fb3-ff5f-4095-ab78-f33f56fc464d-kube-api-access-jcp5v\" (UniqueName: \"kubernetes.io/projected/b4821fb3-ff5f-4095-ab78-f33f56fc464d-kube-api-access-jcp5v\") pod \"b4821fb3-ff5f-4095-ab78-f33f56fc464d\" (UID: \"b4821fb3-ff5f-4095-ab78-f33f56fc464d\") " Oct 30 17:13:31.764982 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:31.764967 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aad374be-4bce-4729-bf69-73a4e855b79b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aad374be-4bce-4729-bf69-73a4e855b79b-maas-api-openshift-ca\") pod \"aad374be-4bce-4729-bf69-73a4e855b79b\" (UID: \"aad374be-4bce-4729-bf69-73a4e855b79b\") " Oct 30 17:13:31.765109 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:31.765089 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7dfcd4f-5c74-41c1-926f-02a3d1b525e5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7dfcd4f-5c74-41c1-926f-02a3d1b525e5-maas-api-xks-ca\") pod \"a7dfcd4f-5c74-41c1-926f-02a3d1b525e5\" (UID: \"a7dfcd4f-5c74-41c1-926f-02a3d1b525e5\") " Oct 30 17:13:31.765289 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:31.765264 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7dfcd4f-5c74-41c1-926f-02a3d1b525e5-maas-api-openshift-ca" pod "a7dfcd4f-5c74-41c1-926f-02a3d1b525e5" (UID: "a7dfcd4f-5c74-41c1-926f-02a3d1b525e5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:13:31.765409 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:31.765389 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aad374be-4bce-4729-bf69-73a4e855b79b-maas-api-openshift-ca" pod "aad374be-4bce-4729-bf69-73a4e855b79b" (UID: "aad374be-4bce-4729-bf69-73a4e855b79b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:13:31.765705 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:31.765688 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4821fb3-ff5f-4095-ab78-f33f56fc464d-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:13:31.765828 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:31.765711 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7dfcd4f-5c74-41c1-926f-02a3d1b525e5-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:13:31.765828 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:31.765726 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aad374be-4bce-4729-bf69-73a4e855b79b-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:13:31.768203 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:31.768177 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aad374be-4bce-4729-bf69-73a4e855b79b-maas-api-xks-ca" pod "aad374be-4bce-4729-bf69-73a4e855b79b" (UID: "aad374be-4bce-4729-bf69-73a4e855b79b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:13:31.768510 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:31.768481 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7dfcd4f-5c74-41c1-926f-02a3d1b525e5-maas-api-xks-ca" pod "a7dfcd4f-5c74-41c1-926f-02a3d1b525e5" (UID: "a7dfcd4f-5c74-41c1-926f-02a3d1b525e5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:13:31.768617 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:31.768578 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4821fb3-ff5f-4095-ab78-f33f56fc464d-kube-api-access-jcp5v" pod "b4821fb3-ff5f-4095-ab78-f33f56fc464d" (UID: "b4821fb3-ff5f-4095-ab78-f33f56fc464d"). InnerVolumeSpecName "kube-api-access-jcp5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:13:31.768677 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:31.768608 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4821fb3-ff5f-4095-ab78-f33f56fc464d-maas-api-xks-ca" pod "b4821fb3-ff5f-4095-ab78-f33f56fc464d" (UID: "b4821fb3-ff5f-4095-ab78-f33f56fc464d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:13:31.768677 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:31.768623 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7dfcd4f-5c74-41c1-926f-02a3d1b525e5-kube-api-access-hdrdl" pod "a7dfcd4f-5c74-41c1-926f-02a3d1b525e5" (UID: "a7dfcd4f-5c74-41c1-926f-02a3d1b525e5"). InnerVolumeSpecName "kube-api-access-hdrdl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:13:31.768677 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:31.768636 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aad374be-4bce-4729-bf69-73a4e855b79b-kube-api-access-w4484" pod "aad374be-4bce-4729-bf69-73a4e855b79b" (UID: "aad374be-4bce-4729-bf69-73a4e855b79b"). InnerVolumeSpecName "kube-api-access-w4484". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:13:31.867195 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:31.867149 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aad374be-4bce-4729-bf69-73a4e855b79b-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:13:31.867195 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:31.867183 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4821fb3-ff5f-4095-ab78-f33f56fc464d-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:13:31.867195 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:31.867197 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdrdl\" (UniqueName: \"kubernetes.io/projected/a7dfcd4f-5c74-41c1-926f-02a3d1b525e5-kube-api-access-hdrdl\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:13:31.867515 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:31.867213 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcp5v\" (UniqueName: \"kubernetes.io/projected/b4821fb3-ff5f-4095-ab78-f33f56fc464d-kube-api-access-jcp5v\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:13:31.867515 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:31.867227 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7dfcd4f-5c74-41c1-926f-02a3d1b525e5-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:13:31.867515 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:31.867242 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4484\" (UniqueName: \"kubernetes.io/projected/aad374be-4bce-4729-bf69-73a4e855b79b-kube-api-access-w4484\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:13:32.474801 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:32.474750 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fbf5ac-69flm" event={"ID":"aad374be-4bce-4729-bf69-73a4e855b79b","Type":"ContainerDied","Data":"09a97e4a22ebd46145678441d310371204d8ccfbc258bd52e5c111594431047e"} Oct 30 17:13:32.474965 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:32.474810 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09a97e4a22ebd46145678441d310371204d8ccfbc258bd52e5c111594431047e" Oct 30 17:13:32.474965 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:32.474752 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0fbf5ac-69flm" Oct 30 17:13:32.476393 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:32.476370 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6949a3-l2mvr" Oct 30 17:13:32.476549 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:32.476407 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f6949a3-l2mvr" event={"ID":"b4821fb3-ff5f-4095-ab78-f33f56fc464d","Type":"ContainerDied","Data":"20082cefa1259e7fa96427efc168918a89b591debc692eccac3b6f7d21ce0473"} Oct 30 17:13:32.476549 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:32.476457 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20082cefa1259e7fa96427efc168918a89b591debc692eccac3b6f7d21ce0473" Oct 30 17:13:32.478166 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:32.478143 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2cc23d3-fb7n8" Oct 30 17:13:32.478299 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:32.478192 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2cc23d3-fb7n8" event={"ID":"a7dfcd4f-5c74-41c1-926f-02a3d1b525e5","Type":"ContainerDied","Data":"aad3dbeb79a699528103bae14e42593575912abedc57cb3e83051ab2df83f6a0"} Oct 30 17:13:32.478299 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:32.478231 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aad3dbeb79a699528103bae14e42593575912abedc57cb3e83051ab2df83f6a0" Oct 30 17:13:37.997482 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:37.997452 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8efa9d2-t2m6h"] Oct 30 17:13:37.998139 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:37.998124 4493 cpu_manager.go:414] "containerMap: removing container" podUID="a7dfcd4f-5c74-41c1-926f-02a3d1b525e5" containerName="revoke-keys" Oct 30 17:13:37.998195 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:37.998141 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7dfcd4f-5c74-41c1-926f-02a3d1b525e5" containerName="revoke-keys" Oct 30 17:13:37.998195 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:37.998159 4493 cpu_manager.go:414] "containerMap: removing container" podUID="b4821fb3-ff5f-4095-ab78-f33f56fc464d" containerName="revoke-keys" Oct 30 17:13:37.998195 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:37.998164 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4821fb3-ff5f-4095-ab78-f33f56fc464d" containerName="revoke-keys" Oct 30 17:13:37.998195 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:37.998189 4493 cpu_manager.go:414] "containerMap: removing container" podUID="aad374be-4bce-4729-bf69-73a4e855b79b" containerName="revoke-keys" Oct 30 17:13:37.998195 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:37.998194 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aad374be-4bce-4729-bf69-73a4e855b79b" containerName="revoke-keys" Oct 30 17:13:37.998378 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:37.998286 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="aad374be-4bce-4729-bf69-73a4e855b79b" containerName="revoke-keys" Oct 30 17:13:37.998378 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:37.998298 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7dfcd4f-5c74-41c1-926f-02a3d1b525e5" containerName="revoke-keys" Oct 30 17:13:37.998378 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:37.998307 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4821fb3-ff5f-4095-ab78-f33f56fc464d" containerName="revoke-keys" Oct 30 17:13:38.003097 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:38.003077 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8efa9d2-t2m6h" Oct 30 17:13:38.030440 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:38.030412 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:13:38.037208 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:38.036655 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:13:38.037208 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:38.036713 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:13:38.037208 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:38.036840 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 17:13:38.038549 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:38.038521 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8efa9d2-t2m6h"] Oct 30 17:13:38.081436 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:38.081402 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94e42682-3d3b-4fca-8bc7-e6c6807950f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8efa9d2-t2m6h\" (UID: \"94e42682-3d3b-4fca-8bc7-e6c6807950f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8efa9d2-t2m6h" Oct 30 17:13:38.081661 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:38.081557 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt8x5\" (UniqueName: \"kubernetes.io/projected/94e42682-3d3b-4fca-8bc7-e6c6807950f5-kube-api-access-kt8x5\") pod \"maas-api-revoke-sub-a8efa9d2-t2m6h\" (UID: \"94e42682-3d3b-4fca-8bc7-e6c6807950f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8efa9d2-t2m6h" Oct 30 17:13:38.081755 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:38.081663 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94e42682-3d3b-4fca-8bc7-e6c6807950f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8efa9d2-t2m6h\" (UID: \"94e42682-3d3b-4fca-8bc7-e6c6807950f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8efa9d2-t2m6h" Oct 30 17:13:38.183710 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:38.183676 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94e42682-3d3b-4fca-8bc7-e6c6807950f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8efa9d2-t2m6h\" (UID: \"94e42682-3d3b-4fca-8bc7-e6c6807950f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8efa9d2-t2m6h" Oct 30 17:13:38.183910 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:38.183804 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt8x5\" (UniqueName: \"kubernetes.io/projected/94e42682-3d3b-4fca-8bc7-e6c6807950f5-kube-api-access-kt8x5\") pod \"maas-api-revoke-sub-a8efa9d2-t2m6h\" (UID: \"94e42682-3d3b-4fca-8bc7-e6c6807950f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8efa9d2-t2m6h" Oct 30 17:13:38.183910 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:38.183836 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94e42682-3d3b-4fca-8bc7-e6c6807950f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8efa9d2-t2m6h\" (UID: \"94e42682-3d3b-4fca-8bc7-e6c6807950f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8efa9d2-t2m6h" Oct 30 17:13:38.184486 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:38.184464 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94e42682-3d3b-4fca-8bc7-e6c6807950f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8efa9d2-t2m6h\" (UID: \"94e42682-3d3b-4fca-8bc7-e6c6807950f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8efa9d2-t2m6h" Oct 30 17:13:38.186454 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:38.186424 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94e42682-3d3b-4fca-8bc7-e6c6807950f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8efa9d2-t2m6h\" (UID: \"94e42682-3d3b-4fca-8bc7-e6c6807950f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8efa9d2-t2m6h" Oct 30 17:13:38.209683 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:38.209652 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt8x5\" (UniqueName: \"kubernetes.io/projected/94e42682-3d3b-4fca-8bc7-e6c6807950f5-kube-api-access-kt8x5\") pod \"maas-api-revoke-sub-a8efa9d2-t2m6h\" (UID: \"94e42682-3d3b-4fca-8bc7-e6c6807950f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8efa9d2-t2m6h" Oct 30 17:13:38.226836 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:38.226768 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c81cd66-lqh87"] Oct 30 17:13:38.231306 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:38.231287 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c81cd66-lqh87" Oct 30 17:13:38.342849 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:38.303034 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87rzf\" (UniqueName: \"kubernetes.io/projected/e0fad0bd-9b34-443a-8262-ec55c07fc934-kube-api-access-87rzf\") pod \"maas-api-revoke-sub-1c81cd66-lqh87\" (UID: \"e0fad0bd-9b34-443a-8262-ec55c07fc934\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c81cd66-lqh87" Oct 30 17:13:38.342849 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:38.303321 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0fad0bd-9b34-443a-8262-ec55c07fc934-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c81cd66-lqh87\" (UID: \"e0fad0bd-9b34-443a-8262-ec55c07fc934\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c81cd66-lqh87" Oct 30 17:13:38.342849 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:38.303485 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0fad0bd-9b34-443a-8262-ec55c07fc934-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c81cd66-lqh87\" (UID: \"e0fad0bd-9b34-443a-8262-ec55c07fc934\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c81cd66-lqh87" Oct 30 17:13:38.342849 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:38.326901 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8efa9d2-t2m6h" Oct 30 17:13:38.344655 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:38.344623 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c81cd66-lqh87"] Oct 30 17:13:38.404763 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:38.404629 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0fad0bd-9b34-443a-8262-ec55c07fc934-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c81cd66-lqh87\" (UID: \"e0fad0bd-9b34-443a-8262-ec55c07fc934\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c81cd66-lqh87" Oct 30 17:13:38.404763 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:38.404707 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87rzf\" (UniqueName: \"kubernetes.io/projected/e0fad0bd-9b34-443a-8262-ec55c07fc934-kube-api-access-87rzf\") pod \"maas-api-revoke-sub-1c81cd66-lqh87\" (UID: \"e0fad0bd-9b34-443a-8262-ec55c07fc934\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c81cd66-lqh87" Oct 30 17:13:38.405114 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:38.404967 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0fad0bd-9b34-443a-8262-ec55c07fc934-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c81cd66-lqh87\" (UID: \"e0fad0bd-9b34-443a-8262-ec55c07fc934\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c81cd66-lqh87" Oct 30 17:13:38.405954 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:38.405924 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0fad0bd-9b34-443a-8262-ec55c07fc934-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c81cd66-lqh87\" (UID: \"e0fad0bd-9b34-443a-8262-ec55c07fc934\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c81cd66-lqh87" Oct 30 17:13:38.413956 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:38.411070 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0fad0bd-9b34-443a-8262-ec55c07fc934-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c81cd66-lqh87\" (UID: \"e0fad0bd-9b34-443a-8262-ec55c07fc934\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c81cd66-lqh87" Oct 30 17:13:38.433462 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:38.433434 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87rzf\" (UniqueName: \"kubernetes.io/projected/e0fad0bd-9b34-443a-8262-ec55c07fc934-kube-api-access-87rzf\") pod \"maas-api-revoke-sub-1c81cd66-lqh87\" (UID: \"e0fad0bd-9b34-443a-8262-ec55c07fc934\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c81cd66-lqh87" Oct 30 17:13:38.613280 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:38.613200 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c81cd66-lqh87" Oct 30 17:13:38.673181 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:38.673150 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8efa9d2-t2m6h"] Oct 30 17:13:38.932145 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:13:38.932113 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0fad0bd_9b34_443a_8262_ec55c07fc934.slice/crio-209fb28d99b34965606321315619c1d293a5f388410da00dbe7e7d12494ec49e WatchSource:0}: Error finding container 209fb28d99b34965606321315619c1d293a5f388410da00dbe7e7d12494ec49e: Status 404 returned error can't find the container with id 209fb28d99b34965606321315619c1d293a5f388410da00dbe7e7d12494ec49e Oct 30 17:13:38.939477 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:38.939449 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c81cd66-lqh87"] Oct 30 17:13:39.576685 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:39.575797 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c81cd66-lqh87" event={"ID":"e0fad0bd-9b34-443a-8262-ec55c07fc934","Type":"ContainerStarted","Data":"cd6f261f02ee181f14b0484da034be39ac5174909da70cc4bcc581c547c93979"} Oct 30 17:13:39.576685 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:39.575859 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c81cd66-lqh87" event={"ID":"e0fad0bd-9b34-443a-8262-ec55c07fc934","Type":"ContainerStarted","Data":"209fb28d99b34965606321315619c1d293a5f388410da00dbe7e7d12494ec49e"} Oct 30 17:13:39.579846 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:39.578726 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8efa9d2-t2m6h" event={"ID":"94e42682-3d3b-4fca-8bc7-e6c6807950f5","Type":"ContainerStarted","Data":"671265411c99153246976c6497e300737c6ae9d34dc235183f0964a5c971b61b"} Oct 30 17:13:39.579846 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:39.578770 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8efa9d2-t2m6h" event={"ID":"94e42682-3d3b-4fca-8bc7-e6c6807950f5","Type":"ContainerStarted","Data":"f3748cec0b201acc05022c5830803fd289b4b8feaa9279908bd71d9386fd1978"} Oct 30 17:13:39.621167 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:39.621115 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c81cd66-lqh87" podStartSLOduration=1.6211013859999999 podStartE2EDuration="1.621101386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:13:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:13:39.616125796 +0000 UTC m=+3046.843300897" watchObservedRunningTime="2026-10-30 17:13:39.621101386 +0000 UTC m=+3046.848276485" Oct 30 17:13:39.659610 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:39.659549 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8efa9d2-t2m6h" podStartSLOduration=2.65953347 podStartE2EDuration="2.65953347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:13:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:13:39.655468171 +0000 UTC m=+3046.882643272" watchObservedRunningTime="2026-10-30 17:13:39.65953347 +0000 UTC m=+3046.886708569" Oct 30 17:13:40.188512 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:40.188471 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ef1fb16-mnsb6"] Oct 30 17:13:40.197714 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:40.197686 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ef1fb16-mnsb6" Oct 30 17:13:40.223175 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:40.223140 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ef1fb16-mnsb6"] Oct 30 17:13:40.352720 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:40.349519 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbnz4\" (UniqueName: \"kubernetes.io/projected/711c8b22-46bd-4989-8a27-829c81465f5a-kube-api-access-hbnz4\") pod \"maas-api-revoke-sub-1ef1fb16-mnsb6\" (UID: \"711c8b22-46bd-4989-8a27-829c81465f5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ef1fb16-mnsb6" Oct 30 17:13:40.369166 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:40.358764 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/711c8b22-46bd-4989-8a27-829c81465f5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ef1fb16-mnsb6\" (UID: \"711c8b22-46bd-4989-8a27-829c81465f5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ef1fb16-mnsb6" Oct 30 17:13:40.369166 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:40.358954 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/711c8b22-46bd-4989-8a27-829c81465f5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ef1fb16-mnsb6\" (UID: \"711c8b22-46bd-4989-8a27-829c81465f5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ef1fb16-mnsb6" Oct 30 17:13:40.462533 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:40.461443 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbnz4\" (UniqueName: \"kubernetes.io/projected/711c8b22-46bd-4989-8a27-829c81465f5a-kube-api-access-hbnz4\") pod \"maas-api-revoke-sub-1ef1fb16-mnsb6\" (UID: \"711c8b22-46bd-4989-8a27-829c81465f5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ef1fb16-mnsb6" Oct 30 17:13:40.462533 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:40.461493 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/711c8b22-46bd-4989-8a27-829c81465f5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ef1fb16-mnsb6\" (UID: \"711c8b22-46bd-4989-8a27-829c81465f5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ef1fb16-mnsb6" Oct 30 17:13:40.462533 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:40.461593 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/711c8b22-46bd-4989-8a27-829c81465f5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ef1fb16-mnsb6\" (UID: \"711c8b22-46bd-4989-8a27-829c81465f5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ef1fb16-mnsb6" Oct 30 17:13:40.462533 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:40.462464 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/711c8b22-46bd-4989-8a27-829c81465f5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ef1fb16-mnsb6\" (UID: \"711c8b22-46bd-4989-8a27-829c81465f5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ef1fb16-mnsb6" Oct 30 17:13:40.466264 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:40.466239 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/711c8b22-46bd-4989-8a27-829c81465f5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ef1fb16-mnsb6\" (UID: \"711c8b22-46bd-4989-8a27-829c81465f5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ef1fb16-mnsb6" Oct 30 17:13:40.486326 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:40.486292 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbnz4\" (UniqueName: \"kubernetes.io/projected/711c8b22-46bd-4989-8a27-829c81465f5a-kube-api-access-hbnz4\") pod \"maas-api-revoke-sub-1ef1fb16-mnsb6\" (UID: \"711c8b22-46bd-4989-8a27-829c81465f5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ef1fb16-mnsb6" Oct 30 17:13:40.527988 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:40.527959 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ef1fb16-mnsb6" Oct 30 17:13:40.595085 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:40.595056 4493 generic.go:356] "Generic (PLEG): container finished" podID="94e42682-3d3b-4fca-8bc7-e6c6807950f5" containerID="671265411c99153246976c6497e300737c6ae9d34dc235183f0964a5c971b61b" exitCode=0 Oct 30 17:13:40.595451 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:40.595277 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8efa9d2-t2m6h" event={"ID":"94e42682-3d3b-4fca-8bc7-e6c6807950f5","Type":"ContainerDied","Data":"671265411c99153246976c6497e300737c6ae9d34dc235183f0964a5c971b61b"} Oct 30 17:13:40.597499 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:40.597206 4493 generic.go:356] "Generic (PLEG): container finished" podID="e0fad0bd-9b34-443a-8262-ec55c07fc934" containerID="cd6f261f02ee181f14b0484da034be39ac5174909da70cc4bcc581c547c93979" exitCode=0 Oct 30 17:13:40.597499 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:40.597384 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c81cd66-lqh87" event={"ID":"e0fad0bd-9b34-443a-8262-ec55c07fc934","Type":"ContainerDied","Data":"cd6f261f02ee181f14b0484da034be39ac5174909da70cc4bcc581c547c93979"} Oct 30 17:13:40.850300 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:40.850271 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ef1fb16-mnsb6"] Oct 30 17:13:40.850838 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:13:40.850808 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod711c8b22_46bd_4989_8a27_829c81465f5a.slice/crio-e08d1879aaa27e6f303d2495ade5fd197e9220680670f4e149d3bc79c91b12ee WatchSource:0}: Error finding container e08d1879aaa27e6f303d2495ade5fd197e9220680670f4e149d3bc79c91b12ee: Status 404 returned error can't find the container with id e08d1879aaa27e6f303d2495ade5fd197e9220680670f4e149d3bc79c91b12ee Oct 30 17:13:41.609256 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:41.609151 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ef1fb16-mnsb6" event={"ID":"711c8b22-46bd-4989-8a27-829c81465f5a","Type":"ContainerStarted","Data":"b7686aa2eecd9fbf06ebb07c5c620d95347aeeb4b1576053d78546518d9a6cbb"} Oct 30 17:13:41.609256 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:41.609246 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ef1fb16-mnsb6" event={"ID":"711c8b22-46bd-4989-8a27-829c81465f5a","Type":"ContainerStarted","Data":"e08d1879aaa27e6f303d2495ade5fd197e9220680670f4e149d3bc79c91b12ee"} Oct 30 17:13:41.661002 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:41.660944 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ef1fb16-mnsb6" podStartSLOduration=1.660926154 podStartE2EDuration="1.660926154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:13:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:13:41.659135854 +0000 UTC m=+3048.886310953" watchObservedRunningTime="2026-10-30 17:13:41.660926154 +0000 UTC m=+3048.888101256" Oct 30 17:13:41.854458 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:41.854432 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8efa9d2-t2m6h" Oct 30 17:13:41.859433 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:41.859383 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c81cd66-lqh87" Oct 30 17:13:41.882680 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:41.882656 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94e42682-3d3b-4fca-8bc7-e6c6807950f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94e42682-3d3b-4fca-8bc7-e6c6807950f5-maas-api-xks-ca\") pod \"94e42682-3d3b-4fca-8bc7-e6c6807950f5\" (UID: \"94e42682-3d3b-4fca-8bc7-e6c6807950f5\") " Oct 30 17:13:41.882826 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:41.882757 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0fad0bd-9b34-443a-8262-ec55c07fc934-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0fad0bd-9b34-443a-8262-ec55c07fc934-maas-api-openshift-ca\") pod \"e0fad0bd-9b34-443a-8262-ec55c07fc934\" (UID: \"e0fad0bd-9b34-443a-8262-ec55c07fc934\") " Oct 30 17:13:41.882938 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:41.882916 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94e42682-3d3b-4fca-8bc7-e6c6807950f5-kube-api-access-kt8x5\" (UniqueName: \"kubernetes.io/projected/94e42682-3d3b-4fca-8bc7-e6c6807950f5-kube-api-access-kt8x5\") pod \"94e42682-3d3b-4fca-8bc7-e6c6807950f5\" (UID: \"94e42682-3d3b-4fca-8bc7-e6c6807950f5\") " Oct 30 17:13:41.883068 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:41.883055 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0fad0bd-9b34-443a-8262-ec55c07fc934-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0fad0bd-9b34-443a-8262-ec55c07fc934-maas-api-xks-ca\") pod \"e0fad0bd-9b34-443a-8262-ec55c07fc934\" (UID: \"e0fad0bd-9b34-443a-8262-ec55c07fc934\") " Oct 30 17:13:41.883132 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:41.883091 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94e42682-3d3b-4fca-8bc7-e6c6807950f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94e42682-3d3b-4fca-8bc7-e6c6807950f5-maas-api-openshift-ca\") pod \"94e42682-3d3b-4fca-8bc7-e6c6807950f5\" (UID: \"94e42682-3d3b-4fca-8bc7-e6c6807950f5\") " Oct 30 17:13:41.883177 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:41.883130 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0fad0bd-9b34-443a-8262-ec55c07fc934-maas-api-openshift-ca" pod "e0fad0bd-9b34-443a-8262-ec55c07fc934" (UID: "e0fad0bd-9b34-443a-8262-ec55c07fc934"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:13:41.883177 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:41.883172 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0fad0bd-9b34-443a-8262-ec55c07fc934-kube-api-access-87rzf\" (UniqueName: \"kubernetes.io/projected/e0fad0bd-9b34-443a-8262-ec55c07fc934-kube-api-access-87rzf\") pod \"e0fad0bd-9b34-443a-8262-ec55c07fc934\" (UID: \"e0fad0bd-9b34-443a-8262-ec55c07fc934\") " Oct 30 17:13:41.883553 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:41.883530 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94e42682-3d3b-4fca-8bc7-e6c6807950f5-maas-api-openshift-ca" pod "94e42682-3d3b-4fca-8bc7-e6c6807950f5" (UID: "94e42682-3d3b-4fca-8bc7-e6c6807950f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:13:41.883815 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:41.883778 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94e42682-3d3b-4fca-8bc7-e6c6807950f5-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:13:41.883959 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:41.883935 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0fad0bd-9b34-443a-8262-ec55c07fc934-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:13:41.885230 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:41.885202 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94e42682-3d3b-4fca-8bc7-e6c6807950f5-kube-api-access-kt8x5" pod "94e42682-3d3b-4fca-8bc7-e6c6807950f5" (UID: "94e42682-3d3b-4fca-8bc7-e6c6807950f5"). InnerVolumeSpecName "kube-api-access-kt8x5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:13:41.885521 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:41.885502 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0fad0bd-9b34-443a-8262-ec55c07fc934-kube-api-access-87rzf" pod "e0fad0bd-9b34-443a-8262-ec55c07fc934" (UID: "e0fad0bd-9b34-443a-8262-ec55c07fc934"). InnerVolumeSpecName "kube-api-access-87rzf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:13:41.885645 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:41.885629 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0fad0bd-9b34-443a-8262-ec55c07fc934-maas-api-xks-ca" pod "e0fad0bd-9b34-443a-8262-ec55c07fc934" (UID: "e0fad0bd-9b34-443a-8262-ec55c07fc934"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:13:41.885694 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:41.885635 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94e42682-3d3b-4fca-8bc7-e6c6807950f5-maas-api-xks-ca" pod "94e42682-3d3b-4fca-8bc7-e6c6807950f5" (UID: "94e42682-3d3b-4fca-8bc7-e6c6807950f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:13:41.986775 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:41.986720 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kt8x5\" (UniqueName: \"kubernetes.io/projected/94e42682-3d3b-4fca-8bc7-e6c6807950f5-kube-api-access-kt8x5\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:13:41.986933 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:41.986778 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0fad0bd-9b34-443a-8262-ec55c07fc934-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:13:41.986933 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:41.986827 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87rzf\" (UniqueName: \"kubernetes.io/projected/e0fad0bd-9b34-443a-8262-ec55c07fc934-kube-api-access-87rzf\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:13:41.986933 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:41.986838 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94e42682-3d3b-4fca-8bc7-e6c6807950f5-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:13:42.619864 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:42.619820 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8efa9d2-t2m6h" event={"ID":"94e42682-3d3b-4fca-8bc7-e6c6807950f5","Type":"ContainerDied","Data":"f3748cec0b201acc05022c5830803fd289b4b8feaa9279908bd71d9386fd1978"} Oct 30 17:13:42.620362 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:42.619887 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3748cec0b201acc05022c5830803fd289b4b8feaa9279908bd71d9386fd1978" Oct 30 17:13:42.620362 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:42.619836 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8efa9d2-t2m6h" Oct 30 17:13:42.621969 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:42.621944 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c81cd66-lqh87" event={"ID":"e0fad0bd-9b34-443a-8262-ec55c07fc934","Type":"ContainerDied","Data":"209fb28d99b34965606321315619c1d293a5f388410da00dbe7e7d12494ec49e"} Oct 30 17:13:42.621969 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:42.621956 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c81cd66-lqh87" Oct 30 17:13:42.622179 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:42.621990 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="209fb28d99b34965606321315619c1d293a5f388410da00dbe7e7d12494ec49e" Oct 30 17:13:42.623907 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:42.623877 4493 generic.go:356] "Generic (PLEG): container finished" podID="711c8b22-46bd-4989-8a27-829c81465f5a" containerID="b7686aa2eecd9fbf06ebb07c5c620d95347aeeb4b1576053d78546518d9a6cbb" exitCode=0 Oct 30 17:13:42.624033 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:42.623917 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ef1fb16-mnsb6" event={"ID":"711c8b22-46bd-4989-8a27-829c81465f5a","Type":"ContainerDied","Data":"b7686aa2eecd9fbf06ebb07c5c620d95347aeeb4b1576053d78546518d9a6cbb"} Oct 30 17:13:43.818329 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:43.818290 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ef1fb16-mnsb6" Oct 30 17:13:43.927687 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:43.927643 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/711c8b22-46bd-4989-8a27-829c81465f5a-kube-api-access-hbnz4\" (UniqueName: \"kubernetes.io/projected/711c8b22-46bd-4989-8a27-829c81465f5a-kube-api-access-hbnz4\") pod \"711c8b22-46bd-4989-8a27-829c81465f5a\" (UID: \"711c8b22-46bd-4989-8a27-829c81465f5a\") " Oct 30 17:13:43.927911 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:43.927743 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/711c8b22-46bd-4989-8a27-829c81465f5a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/711c8b22-46bd-4989-8a27-829c81465f5a-maas-api-openshift-ca\") pod \"711c8b22-46bd-4989-8a27-829c81465f5a\" (UID: \"711c8b22-46bd-4989-8a27-829c81465f5a\") " Oct 30 17:13:43.927911 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:43.927814 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/711c8b22-46bd-4989-8a27-829c81465f5a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/711c8b22-46bd-4989-8a27-829c81465f5a-maas-api-xks-ca\") pod \"711c8b22-46bd-4989-8a27-829c81465f5a\" (UID: \"711c8b22-46bd-4989-8a27-829c81465f5a\") " Oct 30 17:13:43.928228 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:43.928200 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/711c8b22-46bd-4989-8a27-829c81465f5a-maas-api-openshift-ca" pod "711c8b22-46bd-4989-8a27-829c81465f5a" (UID: "711c8b22-46bd-4989-8a27-829c81465f5a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:13:43.928759 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:43.928734 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/711c8b22-46bd-4989-8a27-829c81465f5a-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:13:43.930415 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:43.930386 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/711c8b22-46bd-4989-8a27-829c81465f5a-maas-api-xks-ca" pod "711c8b22-46bd-4989-8a27-829c81465f5a" (UID: "711c8b22-46bd-4989-8a27-829c81465f5a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:13:43.931166 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:43.931139 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/711c8b22-46bd-4989-8a27-829c81465f5a-kube-api-access-hbnz4" pod "711c8b22-46bd-4989-8a27-829c81465f5a" (UID: "711c8b22-46bd-4989-8a27-829c81465f5a"). InnerVolumeSpecName "kube-api-access-hbnz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:13:44.052924 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:44.052214 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbnz4\" (UniqueName: \"kubernetes.io/projected/711c8b22-46bd-4989-8a27-829c81465f5a-kube-api-access-hbnz4\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:13:44.052924 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:44.052256 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/711c8b22-46bd-4989-8a27-829c81465f5a-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:13:44.095464 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:44.095418 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d1960dd-55b85"] Oct 30 17:13:44.100640 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:44.100611 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d1960dd-55b85"] Oct 30 17:13:44.644132 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:44.644102 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ef1fb16-mnsb6" Oct 30 17:13:44.644417 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:44.644098 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ef1fb16-mnsb6" event={"ID":"711c8b22-46bd-4989-8a27-829c81465f5a","Type":"ContainerDied","Data":"e08d1879aaa27e6f303d2495ade5fd197e9220680670f4e149d3bc79c91b12ee"} Oct 30 17:13:44.644417 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:44.644275 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e08d1879aaa27e6f303d2495ade5fd197e9220680670f4e149d3bc79c91b12ee" Oct 30 17:13:44.935051 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:44.934971 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45117888-42f0-4c94-a55b-892b703d1889" path="/var/lib/kubelet/pods/45117888-42f0-4c94-a55b-892b703d1889/volumes" Oct 30 17:13:50.658440 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:50.658403 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-e1267c/e2e-usage-sim-53fe84-6c9ffdddfd-t4swq"] Oct 30 17:13:50.658896 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:50.658702 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-extmod-shr-user-e1267c/e2e-usage-sim-53fe84-6c9ffdddfd-t4swq" podUID="0da18196-46f9-493f-aad4-bb3bb825363a" containerName="sim" containerID="cri-o://05a4ecef3af0bcbcbc90f09eef6a47b8be853a6c32acab6a1447430dbf06d346" gracePeriod=30 Oct 30 17:13:51.220807 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:51.220219 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-e1267c/e2e-usage-sim-53fe84-6c9ffdddfd-t4swq" Oct 30 17:13:51.225942 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:51.222366 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0da18196-46f9-493f-aad4-bb3bb825363a-kube-api-access-m8g56\" (UniqueName: \"kubernetes.io/projected/0da18196-46f9-493f-aad4-bb3bb825363a-kube-api-access-m8g56\") pod \"0da18196-46f9-493f-aad4-bb3bb825363a\" (UID: \"0da18196-46f9-493f-aad4-bb3bb825363a\") " Oct 30 17:13:51.225942 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:51.223231 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0da18196-46f9-493f-aad4-bb3bb825363a-tls\" (UniqueName: \"kubernetes.io/secret/0da18196-46f9-493f-aad4-bb3bb825363a-tls\") pod \"0da18196-46f9-493f-aad4-bb3bb825363a\" (UID: \"0da18196-46f9-493f-aad4-bb3bb825363a\") " Oct 30 17:13:51.229821 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:51.226532 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0da18196-46f9-493f-aad4-bb3bb825363a-kube-api-access-m8g56" pod "0da18196-46f9-493f-aad4-bb3bb825363a" (UID: "0da18196-46f9-493f-aad4-bb3bb825363a"). InnerVolumeSpecName "kube-api-access-m8g56". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:13:51.237850 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:51.231050 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0da18196-46f9-493f-aad4-bb3bb825363a-tls" pod "0da18196-46f9-493f-aad4-bb3bb825363a" (UID: "0da18196-46f9-493f-aad4-bb3bb825363a"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:13:51.377859 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:51.372214 4493 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/0da18196-46f9-493f-aad4-bb3bb825363a-tls\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:13:51.377859 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:51.372250 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8g56\" (UniqueName: \"kubernetes.io/projected/0da18196-46f9-493f-aad4-bb3bb825363a-kube-api-access-m8g56\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:13:51.724900 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:51.724862 4493 generic.go:356] "Generic (PLEG): container finished" podID="0da18196-46f9-493f-aad4-bb3bb825363a" containerID="05a4ecef3af0bcbcbc90f09eef6a47b8be853a6c32acab6a1447430dbf06d346" exitCode=0 Oct 30 17:13:51.725422 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:51.724939 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-e1267c/e2e-usage-sim-53fe84-6c9ffdddfd-t4swq" Oct 30 17:13:51.725422 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:51.724942 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-e1267c/e2e-usage-sim-53fe84-6c9ffdddfd-t4swq" event={"ID":"0da18196-46f9-493f-aad4-bb3bb825363a","Type":"ContainerDied","Data":"05a4ecef3af0bcbcbc90f09eef6a47b8be853a6c32acab6a1447430dbf06d346"} Oct 30 17:13:51.725422 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:51.725005 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-e1267c/e2e-usage-sim-53fe84-6c9ffdddfd-t4swq" event={"ID":"0da18196-46f9-493f-aad4-bb3bb825363a","Type":"ContainerDied","Data":"404d6f7dd40317435162d2159c79cd89f73bcfb3965415e64ec2b3a30a79ef3f"} Oct 30 17:13:51.725422 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:51.725044 4493 scope.go:122] "RemoveContainer" containerID="05a4ecef3af0bcbcbc90f09eef6a47b8be853a6c32acab6a1447430dbf06d346" Oct 30 17:13:51.740248 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:51.740227 4493 scope.go:122] "RemoveContainer" containerID="05a4ecef3af0bcbcbc90f09eef6a47b8be853a6c32acab6a1447430dbf06d346" Oct 30 17:13:51.740548 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:13:51.740524 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05a4ecef3af0bcbcbc90f09eef6a47b8be853a6c32acab6a1447430dbf06d346\": container with ID starting with 05a4ecef3af0bcbcbc90f09eef6a47b8be853a6c32acab6a1447430dbf06d346 not found: ID does not exist" containerID="05a4ecef3af0bcbcbc90f09eef6a47b8be853a6c32acab6a1447430dbf06d346" Oct 30 17:13:51.740632 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:51.740558 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05a4ecef3af0bcbcbc90f09eef6a47b8be853a6c32acab6a1447430dbf06d346"} err="failed to get container status \"05a4ecef3af0bcbcbc90f09eef6a47b8be853a6c32acab6a1447430dbf06d346\": rpc error: code = NotFound desc = could not find container \"05a4ecef3af0bcbcbc90f09eef6a47b8be853a6c32acab6a1447430dbf06d346\": container with ID starting with 05a4ecef3af0bcbcbc90f09eef6a47b8be853a6c32acab6a1447430dbf06d346 not found: ID does not exist" Oct 30 17:13:51.783494 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:51.783460 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-e1267c/e2e-usage-sim-53fe84-6c9ffdddfd-t4swq"] Oct 30 17:13:51.795050 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:51.795026 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-extmod-shr-user-e1267c/e2e-usage-sim-53fe84-6c9ffdddfd-t4swq"] Oct 30 17:13:53.211085 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:53.211041 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0da18196-46f9-493f-aad4-bb3bb825363a" path="/var/lib/kubelet/pods/0da18196-46f9-493f-aad4-bb3bb825363a/volumes" Oct 30 17:13:55.034492 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:55.034460 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90d54b1f-nkmqg"] Oct 30 17:13:55.045651 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:55.045624 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90d54b1f-nkmqg"] Oct 30 17:13:56.883553 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:56.883518 4493 scope.go:122] "RemoveContainer" containerID="26123186e843d2f0aa42d61ec1dd3877077c137d7fb39d975ad3670007c56f51" Oct 30 17:13:56.935290 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:13:56.935253 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="decec274-c815-4e33-948b-7966e1f420b0" path="/var/lib/kubelet/pods/decec274-c815-4e33-948b-7966e1f420b0/volumes" Oct 30 17:14:00.344746 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:14:00.344699 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 17:14:08.038640 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:08.038606 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a3d9d25-j5cs8"] Oct 30 17:14:08.050452 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:08.050424 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a3d9d25-j5cs8"] Oct 30 17:14:08.936218 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:08.936163 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c41b6331-1e79-4bf4-8f1c-7728a330202b" path="/var/lib/kubelet/pods/c41b6331-1e79-4bf4-8f1c-7728a330202b/volumes" Oct 30 17:14:17.380844 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:17.380808 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6"] Oct 30 17:14:17.381710 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:17.381688 4493 cpu_manager.go:414] "containerMap: removing container" podUID="711c8b22-46bd-4989-8a27-829c81465f5a" containerName="revoke-keys" Oct 30 17:14:17.381843 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:17.381712 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="711c8b22-46bd-4989-8a27-829c81465f5a" containerName="revoke-keys" Oct 30 17:14:17.381843 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:17.381742 4493 cpu_manager.go:414] "containerMap: removing container" podUID="94e42682-3d3b-4fca-8bc7-e6c6807950f5" containerName="revoke-keys" Oct 30 17:14:17.381843 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:17.381749 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94e42682-3d3b-4fca-8bc7-e6c6807950f5" containerName="revoke-keys" Oct 30 17:14:17.381843 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:17.381800 4493 cpu_manager.go:414] "containerMap: removing container" podUID="e0fad0bd-9b34-443a-8262-ec55c07fc934" containerName="revoke-keys" Oct 30 17:14:17.381843 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:17.381808 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0fad0bd-9b34-443a-8262-ec55c07fc934" containerName="revoke-keys" Oct 30 17:14:17.381843 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:17.381827 4493 cpu_manager.go:414] "containerMap: removing container" podUID="0da18196-46f9-493f-aad4-bb3bb825363a" containerName="sim" Oct 30 17:14:17.381843 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:17.381834 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0da18196-46f9-493f-aad4-bb3bb825363a" containerName="sim" Oct 30 17:14:17.382232 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:17.382009 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="711c8b22-46bd-4989-8a27-829c81465f5a" containerName="revoke-keys" Oct 30 17:14:17.382232 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:17.382027 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="94e42682-3d3b-4fca-8bc7-e6c6807950f5" containerName="revoke-keys" Oct 30 17:14:17.382232 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:17.382041 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0fad0bd-9b34-443a-8262-ec55c07fc934" containerName="revoke-keys" Oct 30 17:14:17.382232 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:17.382050 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="0da18196-46f9-493f-aad4-bb3bb825363a" containerName="sim" Oct 30 17:14:17.391064 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:17.387548 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6" Oct 30 17:14:17.410358 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:17.409878 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-e15d1f-openshift-default-dockercfg-jgp98\"" Oct 30 17:14:17.425655 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:17.425626 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6"] Oct 30 17:14:17.448311 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:17.448276 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba3aa602-ca3b-4d37-9156-7c95b17a99cd-istio-podinfo\") pod \"e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6\" (UID: \"ba3aa602-ca3b-4d37-9156-7c95b17a99cd\") " pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6" Oct 30 17:14:17.450209 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:17.450186 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba3aa602-ca3b-4d37-9156-7c95b17a99cd-istio-token\") pod \"e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6\" (UID: \"ba3aa602-ca3b-4d37-9156-7c95b17a99cd\") " pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6" Oct 30 17:14:17.450460 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:17.450443 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba3aa602-ca3b-4d37-9156-7c95b17a99cd-istiod-ca-cert\") pod \"e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6\" (UID: \"ba3aa602-ca3b-4d37-9156-7c95b17a99cd\") " pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6" Oct 30 17:14:17.450632 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:17.450616 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba3aa602-ca3b-4d37-9156-7c95b17a99cd-istio-data\") pod \"e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6\" (UID: \"ba3aa602-ca3b-4d37-9156-7c95b17a99cd\") " pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6" Oct 30 17:14:17.450823 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:17.450803 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6swd8\" (UniqueName: \"kubernetes.io/projected/ba3aa602-ca3b-4d37-9156-7c95b17a99cd-kube-api-access-6swd8\") pod \"e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6\" (UID: \"ba3aa602-ca3b-4d37-9156-7c95b17a99cd\") " pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6" Oct 30 17:14:17.450965 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:17.450949 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba3aa602-ca3b-4d37-9156-7c95b17a99cd-workload-socket\") pod \"e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6\" (UID: \"ba3aa602-ca3b-4d37-9156-7c95b17a99cd\") " pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6" Oct 30 17:14:17.451288 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:17.451269 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba3aa602-ca3b-4d37-9156-7c95b17a99cd-credential-socket\") pod \"e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6\" (UID: \"ba3aa602-ca3b-4d37-9156-7c95b17a99cd\") " pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6" Oct 30 17:14:17.451393 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:17.451322 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba3aa602-ca3b-4d37-9156-7c95b17a99cd-istio-envoy\") pod \"e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6\" (UID: \"ba3aa602-ca3b-4d37-9156-7c95b17a99cd\") " pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6" Oct 30 17:14:17.451393 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:17.451381 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba3aa602-ca3b-4d37-9156-7c95b17a99cd-workload-certs\") pod \"e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6\" (UID: \"ba3aa602-ca3b-4d37-9156-7c95b17a99cd\") " pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6" Oct 30 17:14:17.553370 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:17.553329 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba3aa602-ca3b-4d37-9156-7c95b17a99cd-istio-token\") pod \"e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6\" (UID: \"ba3aa602-ca3b-4d37-9156-7c95b17a99cd\") " pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6" Oct 30 17:14:17.553749 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:17.553725 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba3aa602-ca3b-4d37-9156-7c95b17a99cd-istiod-ca-cert\") pod \"e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6\" (UID: \"ba3aa602-ca3b-4d37-9156-7c95b17a99cd\") " pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6" Oct 30 17:14:17.568955 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:17.568915 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba3aa602-ca3b-4d37-9156-7c95b17a99cd-istio-data\") pod \"e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6\" (UID: \"ba3aa602-ca3b-4d37-9156-7c95b17a99cd\") " pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6" Oct 30 17:14:17.569208 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:17.569183 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6swd8\" (UniqueName: \"kubernetes.io/projected/ba3aa602-ca3b-4d37-9156-7c95b17a99cd-kube-api-access-6swd8\") pod \"e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6\" (UID: \"ba3aa602-ca3b-4d37-9156-7c95b17a99cd\") " pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6" Oct 30 17:14:17.569348 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:17.569334 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba3aa602-ca3b-4d37-9156-7c95b17a99cd-workload-socket\") pod \"e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6\" (UID: \"ba3aa602-ca3b-4d37-9156-7c95b17a99cd\") " pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6" Oct 30 17:14:17.569561 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:17.569544 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba3aa602-ca3b-4d37-9156-7c95b17a99cd-credential-socket\") pod \"e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6\" (UID: \"ba3aa602-ca3b-4d37-9156-7c95b17a99cd\") " pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6" Oct 30 17:14:17.569717 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:17.569703 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba3aa602-ca3b-4d37-9156-7c95b17a99cd-istio-envoy\") pod \"e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6\" (UID: \"ba3aa602-ca3b-4d37-9156-7c95b17a99cd\") " pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6" Oct 30 17:14:17.569916 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:17.569890 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba3aa602-ca3b-4d37-9156-7c95b17a99cd-workload-certs\") pod \"e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6\" (UID: \"ba3aa602-ca3b-4d37-9156-7c95b17a99cd\") " pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6" Oct 30 17:14:17.570141 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:17.570127 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba3aa602-ca3b-4d37-9156-7c95b17a99cd-istio-podinfo\") pod \"e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6\" (UID: \"ba3aa602-ca3b-4d37-9156-7c95b17a99cd\") " pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6" Oct 30 17:14:17.580817 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:17.580772 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba3aa602-ca3b-4d37-9156-7c95b17a99cd-istiod-ca-cert\") pod \"e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6\" (UID: \"ba3aa602-ca3b-4d37-9156-7c95b17a99cd\") " pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6" Oct 30 17:14:17.586869 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:17.581123 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba3aa602-ca3b-4d37-9156-7c95b17a99cd-istio-data\") pod \"e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6\" (UID: \"ba3aa602-ca3b-4d37-9156-7c95b17a99cd\") " pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6" Oct 30 17:14:17.587066 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:17.581349 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba3aa602-ca3b-4d37-9156-7c95b17a99cd-workload-socket\") pod \"e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6\" (UID: \"ba3aa602-ca3b-4d37-9156-7c95b17a99cd\") " pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6" Oct 30 17:14:17.633579 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:17.597346 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba3aa602-ca3b-4d37-9156-7c95b17a99cd-istio-podinfo\") pod \"e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6\" (UID: \"ba3aa602-ca3b-4d37-9156-7c95b17a99cd\") " pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6" Oct 30 17:14:17.633579 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:17.602069 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba3aa602-ca3b-4d37-9156-7c95b17a99cd-workload-certs\") pod \"e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6\" (UID: \"ba3aa602-ca3b-4d37-9156-7c95b17a99cd\") " pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6" Oct 30 17:14:17.634290 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:17.634262 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba3aa602-ca3b-4d37-9156-7c95b17a99cd-credential-socket\") pod \"e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6\" (UID: \"ba3aa602-ca3b-4d37-9156-7c95b17a99cd\") " pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6" Oct 30 17:14:17.748835 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:17.736576 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba3aa602-ca3b-4d37-9156-7c95b17a99cd-istio-envoy\") pod \"e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6\" (UID: \"ba3aa602-ca3b-4d37-9156-7c95b17a99cd\") " pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6" Oct 30 17:14:17.780147 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:17.780111 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6swd8\" (UniqueName: \"kubernetes.io/projected/ba3aa602-ca3b-4d37-9156-7c95b17a99cd-kube-api-access-6swd8\") pod \"e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6\" (UID: \"ba3aa602-ca3b-4d37-9156-7c95b17a99cd\") " pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6" Oct 30 17:14:17.890508 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:17.890420 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba3aa602-ca3b-4d37-9156-7c95b17a99cd-istio-token\") pod \"e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6\" (UID: \"ba3aa602-ca3b-4d37-9156-7c95b17a99cd\") " pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6" Oct 30 17:14:18.024602 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:18.021276 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6" Oct 30 17:14:18.553466 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:14:18.553426 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba3aa602_ca3b_4d37_9156_7c95b17a99cd.slice/crio-a8ca382523d5cf30997edd3a04af93a751610bf00b9f9aea0fa50f186c099de4 WatchSource:0}: Error finding container a8ca382523d5cf30997edd3a04af93a751610bf00b9f9aea0fa50f186c099de4: Status 404 returned error can't find the container with id a8ca382523d5cf30997edd3a04af93a751610bf00b9f9aea0fa50f186c099de4 Oct 30 17:14:18.557215 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:18.557172 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:14:18.557328 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:18.557263 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:14:18.557328 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:18.557304 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:14:18.563988 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:18.563960 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6"] Oct 30 17:14:18.954616 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:18.954578 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6" event={"ID":"ba3aa602-ca3b-4d37-9156-7c95b17a99cd","Type":"ContainerStarted","Data":"33f6e9b7a3fe74fad17b82952f1bd31a08384ffaf3a4a9c54df250e09ebb4b54"} Oct 30 17:14:18.954865 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:18.954686 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6" event={"ID":"ba3aa602-ca3b-4d37-9156-7c95b17a99cd","Type":"ContainerStarted","Data":"a8ca382523d5cf30997edd3a04af93a751610bf00b9f9aea0fa50f186c099de4"} Oct 30 17:14:19.021599 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:19.021539 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6" podStartSLOduration=2.021519359 podStartE2EDuration="2.021519359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:14:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:14:19.011339947 +0000 UTC m=+3086.238515047" watchObservedRunningTime="2026-10-30 17:14:19.021519359 +0000 UTC m=+3086.248694461" Oct 30 17:14:19.022631 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:19.022602 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6" Oct 30 17:14:19.024003 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:19.023975 4493 patch_prober.go:29] interesting pod/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.19:15021/healthz/ready\": dial tcp 10.134.1.19:15021: connect: connection refused" start-of-body= Oct 30 17:14:19.024114 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:19.024030 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6" podUID="ba3aa602-ca3b-4d37-9156-7c95b17a99cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.19:15021/healthz/ready\": dial tcp 10.134.1.19:15021: connect: connection refused" Oct 30 17:14:20.027701 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:20.027668 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6" Oct 30 17:14:20.970906 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:20.970878 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6" Oct 30 17:14:20.971748 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:20.971727 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6" Oct 30 17:14:26.645810 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:26.645523 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 17:14:30.373037 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:14:30.372997 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 17:14:36.038005 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:36.037962 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5fff639-nw275"] Oct 30 17:14:36.048971 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:36.048936 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5fff639-nw275"] Oct 30 17:14:36.939040 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:36.938997 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86a15b69-0911-4261-a2a4-08198bfb9142" path="/var/lib/kubelet/pods/86a15b69-0911-4261-a2a4-08198bfb9142/volumes" Oct 30 17:14:41.425149 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:41.425111 4493 generic.go:356] "Generic (PLEG): container finished" podID="66c82e49-cc58-4f4c-b090-78dbcbdff588" containerID="847c48a76d37d99d788a15fb5702dfb7e31d66d156572538fbc95c640f105702" exitCode=0 Oct 30 17:14:41.425621 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:41.425178 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw" event={"ID":"66c82e49-cc58-4f4c-b090-78dbcbdff588","Type":"ContainerDied","Data":"847c48a76d37d99d788a15fb5702dfb7e31d66d156572538fbc95c640f105702"} Oct 30 17:14:41.425621 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:41.425558 4493 scope.go:122] "RemoveContainer" containerID="847c48a76d37d99d788a15fb5702dfb7e31d66d156572538fbc95c640f105702" Oct 30 17:14:41.427441 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:41.427408 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:14:41.427562 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:41.427482 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:14:41.427562 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:41.427517 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:14:42.442556 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:42.442506 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw" event={"ID":"66c82e49-cc58-4f4c-b090-78dbcbdff588","Type":"ContainerStarted","Data":"8aa60a7cf4135b49980a4a40788696424a55f27cc66f1a1db3c113a34ca1fe17"} Oct 30 17:14:42.870335 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:42.870244 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw" Oct 30 17:14:42.870498 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:42.870435 4493 patch_prober.go:29] interesting pod/e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.12:15021/healthz/ready\": dial tcp 10.134.1.12:15021: connect: connection refused" start-of-body= Oct 30 17:14:42.870498 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:42.870487 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw" podUID="66c82e49-cc58-4f4c-b090-78dbcbdff588" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.12:15021/healthz/ready\": dial tcp 10.134.1.12:15021: connect: connection refused" Oct 30 17:14:43.870599 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:43.870562 4493 patch_prober.go:29] interesting pod/e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.12:15021/healthz/ready\": dial tcp 10.134.1.12:15021: connect: connection refused" start-of-body= Oct 30 17:14:43.871077 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:43.870626 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw" podUID="66c82e49-cc58-4f4c-b090-78dbcbdff588" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.12:15021/healthz/ready\": dial tcp 10.134.1.12:15021: connect: connection refused" Oct 30 17:14:44.870513 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:44.870465 4493 patch_prober.go:29] interesting pod/e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.12:15021/healthz/ready\": dial tcp 10.134.1.12:15021: connect: connection refused" start-of-body= Oct 30 17:14:44.870695 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:44.870555 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw" podUID="66c82e49-cc58-4f4c-b090-78dbcbdff588" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.12:15021/healthz/ready\": dial tcp 10.134.1.12:15021: connect: connection refused" Oct 30 17:14:45.870376 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:45.870341 4493 patch_prober.go:29] interesting pod/e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.12:15021/healthz/ready\": dial tcp 10.134.1.12:15021: connect: connection refused" start-of-body= Oct 30 17:14:45.870861 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:45.870420 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw" podUID="66c82e49-cc58-4f4c-b090-78dbcbdff588" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.12:15021/healthz/ready\": dial tcp 10.134.1.12:15021: connect: connection refused" Oct 30 17:14:46.870549 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:46.870506 4493 patch_prober.go:29] interesting pod/e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.12:15021/healthz/ready\": dial tcp 10.134.1.12:15021: connect: connection refused" start-of-body= Oct 30 17:14:46.870917 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:46.870566 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw" podUID="66c82e49-cc58-4f4c-b090-78dbcbdff588" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.12:15021/healthz/ready\": dial tcp 10.134.1.12:15021: connect: connection refused" Oct 30 17:14:47.771259 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:47.771225 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw" Oct 30 17:14:47.902053 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:47.902021 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw" Oct 30 17:14:48.035448 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:48.035353 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3820b22-w45fd"] Oct 30 17:14:48.040103 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:48.040074 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3820b22-w45fd"] Oct 30 17:14:48.497216 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:48.497184 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw" Oct 30 17:14:48.935088 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:48.935058 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44f0bfa4-b78a-4cad-99e5-53bfc3979a2a" path="/var/lib/kubelet/pods/44f0bfa4-b78a-4cad-99e5-53bfc3979a2a/volumes" Oct 30 17:14:56.949970 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:56.949941 4493 scope.go:122] "RemoveContainer" containerID="9b1052ea46c2b365c5cd42ac33c75b7597c8c56aa3a7e708c8e93a87751fe8e3" Oct 30 17:14:56.964092 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:56.963902 4493 scope.go:122] "RemoveContainer" containerID="3a9ef4f86f495b704cdbff0652445bd70f294f07ac41d6bc8a00281ddd2f57d4" Oct 30 17:14:57.002816 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:57.000955 4493 scope.go:122] "RemoveContainer" containerID="47c52c1f3d0a8e8d452180fc23d1164ff75cb42bfdb6a62066aa1c2a671d41ac" Oct 30 17:14:57.584997 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:57.584961 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-24acb2/test-model-24acb2-kserve-946ff6bb7-h8cjs"] Oct 30 17:14:57.591207 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:57.591182 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-24acb2/test-model-24acb2-kserve-946ff6bb7-h8cjs" Oct 30 17:14:57.626755 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:57.624745 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-24acb2\"/\"kube-root-ca.crt\"" Oct 30 17:14:57.626755 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:57.625179 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-24acb2\"/\"test-model-24acb2-kserve-self-signed-certs\"" Oct 30 17:14:57.626755 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:57.625374 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-24acb2\"/\"openshift-service-ca.crt\"" Oct 30 17:14:57.626755 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:57.625657 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-24acb2\"/\"default-dockercfg-fc947\"" Oct 30 17:14:57.634165 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:57.634012 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-24acb2/test-model-24acb2-kserve-946ff6bb7-h8cjs"] Oct 30 17:14:57.737031 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:57.732847 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd1ff57b-1a55-4ef7-95f6-ee9389a00d99-model-cache\") pod \"test-model-24acb2-kserve-946ff6bb7-h8cjs\" (UID: \"cd1ff57b-1a55-4ef7-95f6-ee9389a00d99\") " pod="ai-tenant-e2e-inf-b-24acb2/test-model-24acb2-kserve-946ff6bb7-h8cjs" Oct 30 17:14:57.737031 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:57.732935 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd1ff57b-1a55-4ef7-95f6-ee9389a00d99-tmp-dir\") pod \"test-model-24acb2-kserve-946ff6bb7-h8cjs\" (UID: \"cd1ff57b-1a55-4ef7-95f6-ee9389a00d99\") " pod="ai-tenant-e2e-inf-b-24acb2/test-model-24acb2-kserve-946ff6bb7-h8cjs" Oct 30 17:14:57.737031 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:57.732984 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq74z\" (UniqueName: \"kubernetes.io/projected/cd1ff57b-1a55-4ef7-95f6-ee9389a00d99-kube-api-access-jq74z\") pod \"test-model-24acb2-kserve-946ff6bb7-h8cjs\" (UID: \"cd1ff57b-1a55-4ef7-95f6-ee9389a00d99\") " pod="ai-tenant-e2e-inf-b-24acb2/test-model-24acb2-kserve-946ff6bb7-h8cjs" Oct 30 17:14:57.737031 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:57.733032 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd1ff57b-1a55-4ef7-95f6-ee9389a00d99-tls-certs\") pod \"test-model-24acb2-kserve-946ff6bb7-h8cjs\" (UID: \"cd1ff57b-1a55-4ef7-95f6-ee9389a00d99\") " pod="ai-tenant-e2e-inf-b-24acb2/test-model-24acb2-kserve-946ff6bb7-h8cjs" Oct 30 17:14:57.737031 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:57.733085 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd1ff57b-1a55-4ef7-95f6-ee9389a00d99-dshm\") pod \"test-model-24acb2-kserve-946ff6bb7-h8cjs\" (UID: \"cd1ff57b-1a55-4ef7-95f6-ee9389a00d99\") " pod="ai-tenant-e2e-inf-b-24acb2/test-model-24acb2-kserve-946ff6bb7-h8cjs" Oct 30 17:14:57.737031 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:57.733244 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd1ff57b-1a55-4ef7-95f6-ee9389a00d99-home\") pod \"test-model-24acb2-kserve-946ff6bb7-h8cjs\" (UID: \"cd1ff57b-1a55-4ef7-95f6-ee9389a00d99\") " pod="ai-tenant-e2e-inf-b-24acb2/test-model-24acb2-kserve-946ff6bb7-h8cjs" Oct 30 17:14:57.835859 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:57.835006 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd1ff57b-1a55-4ef7-95f6-ee9389a00d99-model-cache\") pod \"test-model-24acb2-kserve-946ff6bb7-h8cjs\" (UID: \"cd1ff57b-1a55-4ef7-95f6-ee9389a00d99\") " pod="ai-tenant-e2e-inf-b-24acb2/test-model-24acb2-kserve-946ff6bb7-h8cjs" Oct 30 17:14:57.835859 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:57.835098 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd1ff57b-1a55-4ef7-95f6-ee9389a00d99-tmp-dir\") pod \"test-model-24acb2-kserve-946ff6bb7-h8cjs\" (UID: \"cd1ff57b-1a55-4ef7-95f6-ee9389a00d99\") " pod="ai-tenant-e2e-inf-b-24acb2/test-model-24acb2-kserve-946ff6bb7-h8cjs" Oct 30 17:14:57.835859 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:57.835145 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq74z\" (UniqueName: \"kubernetes.io/projected/cd1ff57b-1a55-4ef7-95f6-ee9389a00d99-kube-api-access-jq74z\") pod \"test-model-24acb2-kserve-946ff6bb7-h8cjs\" (UID: \"cd1ff57b-1a55-4ef7-95f6-ee9389a00d99\") " pod="ai-tenant-e2e-inf-b-24acb2/test-model-24acb2-kserve-946ff6bb7-h8cjs" Oct 30 17:14:57.835859 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:57.835195 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd1ff57b-1a55-4ef7-95f6-ee9389a00d99-tls-certs\") pod \"test-model-24acb2-kserve-946ff6bb7-h8cjs\" (UID: \"cd1ff57b-1a55-4ef7-95f6-ee9389a00d99\") " pod="ai-tenant-e2e-inf-b-24acb2/test-model-24acb2-kserve-946ff6bb7-h8cjs" Oct 30 17:14:57.835859 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:57.835248 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd1ff57b-1a55-4ef7-95f6-ee9389a00d99-dshm\") pod \"test-model-24acb2-kserve-946ff6bb7-h8cjs\" (UID: \"cd1ff57b-1a55-4ef7-95f6-ee9389a00d99\") " pod="ai-tenant-e2e-inf-b-24acb2/test-model-24acb2-kserve-946ff6bb7-h8cjs" Oct 30 17:14:57.835859 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:57.835412 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd1ff57b-1a55-4ef7-95f6-ee9389a00d99-home\") pod \"test-model-24acb2-kserve-946ff6bb7-h8cjs\" (UID: \"cd1ff57b-1a55-4ef7-95f6-ee9389a00d99\") " pod="ai-tenant-e2e-inf-b-24acb2/test-model-24acb2-kserve-946ff6bb7-h8cjs" Oct 30 17:14:57.835859 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:57.835443 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd1ff57b-1a55-4ef7-95f6-ee9389a00d99-model-cache\") pod \"test-model-24acb2-kserve-946ff6bb7-h8cjs\" (UID: \"cd1ff57b-1a55-4ef7-95f6-ee9389a00d99\") " pod="ai-tenant-e2e-inf-b-24acb2/test-model-24acb2-kserve-946ff6bb7-h8cjs" Oct 30 17:14:57.835859 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:57.835737 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd1ff57b-1a55-4ef7-95f6-ee9389a00d99-home\") pod \"test-model-24acb2-kserve-946ff6bb7-h8cjs\" (UID: \"cd1ff57b-1a55-4ef7-95f6-ee9389a00d99\") " pod="ai-tenant-e2e-inf-b-24acb2/test-model-24acb2-kserve-946ff6bb7-h8cjs" Oct 30 17:14:57.836432 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:57.836211 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd1ff57b-1a55-4ef7-95f6-ee9389a00d99-tmp-dir\") pod \"test-model-24acb2-kserve-946ff6bb7-h8cjs\" (UID: \"cd1ff57b-1a55-4ef7-95f6-ee9389a00d99\") " pod="ai-tenant-e2e-inf-b-24acb2/test-model-24acb2-kserve-946ff6bb7-h8cjs" Oct 30 17:14:57.847812 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:57.841572 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd1ff57b-1a55-4ef7-95f6-ee9389a00d99-tls-certs\") pod \"test-model-24acb2-kserve-946ff6bb7-h8cjs\" (UID: \"cd1ff57b-1a55-4ef7-95f6-ee9389a00d99\") " pod="ai-tenant-e2e-inf-b-24acb2/test-model-24acb2-kserve-946ff6bb7-h8cjs" Oct 30 17:14:57.847812 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:57.843586 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd1ff57b-1a55-4ef7-95f6-ee9389a00d99-dshm\") pod \"test-model-24acb2-kserve-946ff6bb7-h8cjs\" (UID: \"cd1ff57b-1a55-4ef7-95f6-ee9389a00d99\") " pod="ai-tenant-e2e-inf-b-24acb2/test-model-24acb2-kserve-946ff6bb7-h8cjs" Oct 30 17:14:57.873814 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:57.873445 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq74z\" (UniqueName: \"kubernetes.io/projected/cd1ff57b-1a55-4ef7-95f6-ee9389a00d99-kube-api-access-jq74z\") pod \"test-model-24acb2-kserve-946ff6bb7-h8cjs\" (UID: \"cd1ff57b-1a55-4ef7-95f6-ee9389a00d99\") " pod="ai-tenant-e2e-inf-b-24acb2/test-model-24acb2-kserve-946ff6bb7-h8cjs" Oct 30 17:14:57.908807 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:57.908122 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-24acb2/test-model-24acb2-kserve-946ff6bb7-h8cjs" Oct 30 17:14:58.440311 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:14:58.440274 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcd1ff57b_1a55_4ef7_95f6_ee9389a00d99.slice/crio-b814e24362b807a9f41dfac9d23de70d87291ca610d88428f0e344975e5d07ba WatchSource:0}: Error finding container b814e24362b807a9f41dfac9d23de70d87291ca610d88428f0e344975e5d07ba: Status 404 returned error can't find the container with id b814e24362b807a9f41dfac9d23de70d87291ca610d88428f0e344975e5d07ba Oct 30 17:14:58.442654 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:58.442628 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-24acb2/test-model-24acb2-kserve-946ff6bb7-h8cjs"] Oct 30 17:14:58.632371 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:58.632322 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-24acb2/test-model-24acb2-kserve-946ff6bb7-h8cjs" event={"ID":"cd1ff57b-1a55-4ef7-95f6-ee9389a00d99","Type":"ContainerStarted","Data":"7025845d11a69595f9dca24bea1b1681eaf5b1f436eecd5dd4f95b6bb52ba2af"} Oct 30 17:14:58.632610 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:58.632596 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-24acb2/test-model-24acb2-kserve-946ff6bb7-h8cjs" event={"ID":"cd1ff57b-1a55-4ef7-95f6-ee9389a00d99","Type":"ContainerStarted","Data":"b814e24362b807a9f41dfac9d23de70d87291ca610d88428f0e344975e5d07ba"} Oct 30 17:14:58.680605 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:58.680540 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-24acb2/test-model-24acb2-kserve-946ff6bb7-h8cjs" podStartSLOduration=1.6805229659999998 podStartE2EDuration="1.680522966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:14:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:14:58.67472182 +0000 UTC m=+3125.901896922" watchObservedRunningTime="2026-10-30 17:14:58.680522966 +0000 UTC m=+3125.907698069" Oct 30 17:14:58.909731 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:14:58.909699 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-24acb2/test-model-24acb2-kserve-946ff6bb7-h8cjs" Oct 30 17:15:00.147450 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.147399 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5758b7-29889675-hshgf"] Oct 30 17:15:00.153021 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.152996 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5758b7-29889675-hshgf" Oct 30 17:15:00.156673 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.156649 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-13c495-29889675-79wjh"] Oct 30 17:15:00.168394 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.167936 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2589b5-29889675-h7lhw"] Oct 30 17:15:00.169968 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.169948 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-13c495-29889675-79wjh" Oct 30 17:15:00.179808 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.177011 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 17:15:00.229936 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.229887 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7wsh\" (UniqueName: \"kubernetes.io/projected/b72976e0-c501-44b4-b42c-5dfa26192e87-kube-api-access-x7wsh\") pod \"maas-api-key-cleanup-e2e-extmod-b-13c495-29889675-79wjh\" (UID: \"b72976e0-c501-44b4-b42c-5dfa26192e87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-13c495-29889675-79wjh" Oct 30 17:15:00.230205 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.230184 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhkfk\" (UniqueName: \"kubernetes.io/projected/68c22006-b1b5-4f15-b504-be63d753819c-kube-api-access-vhkfk\") pod \"maas-api-key-cleanup-e2e-extmod-b-5758b7-29889675-hshgf\" (UID: \"68c22006-b1b5-4f15-b504-be63d753819c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5758b7-29889675-hshgf" Oct 30 17:15:00.230427 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.230411 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68c22006-b1b5-4f15-b504-be63d753819c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-5758b7-29889675-hshgf\" (UID: \"68c22006-b1b5-4f15-b504-be63d753819c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5758b7-29889675-hshgf" Oct 30 17:15:00.230549 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.230535 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b72976e0-c501-44b4-b42c-5dfa26192e87-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-13c495-29889675-79wjh\" (UID: \"b72976e0-c501-44b4-b42c-5dfa26192e87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-13c495-29889675-79wjh" Oct 30 17:15:00.234523 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.234478 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:15:00.235014 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.234992 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:15:00.244111 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.235381 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f94fb5-29889675-6w42p"] Oct 30 17:15:00.252664 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.235466 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2589b5-29889675-h7lhw" Oct 30 17:15:00.301539 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.300985 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f94fb5-29889675-6w42p" Oct 30 17:15:00.301751 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.300881 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-abda73-29889675-8gqs6"] Oct 30 17:15:00.307648 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.307621 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6acc78-29889675-664bj"] Oct 30 17:15:00.309083 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.309063 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-abda73-29889675-8gqs6" Oct 30 17:15:00.322357 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.322338 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d5b798-29889675-hhqqx"] Oct 30 17:15:00.323122 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.323103 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6acc78-29889675-664bj" Oct 30 17:15:00.340953 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.340898 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7wsh\" (UniqueName: \"kubernetes.io/projected/b72976e0-c501-44b4-b42c-5dfa26192e87-kube-api-access-x7wsh\") pod \"maas-api-key-cleanup-e2e-extmod-b-13c495-29889675-79wjh\" (UID: \"b72976e0-c501-44b4-b42c-5dfa26192e87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-13c495-29889675-79wjh" Oct 30 17:15:00.342568 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.342544 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhkfk\" (UniqueName: \"kubernetes.io/projected/68c22006-b1b5-4f15-b504-be63d753819c-kube-api-access-vhkfk\") pod \"maas-api-key-cleanup-e2e-extmod-b-5758b7-29889675-hshgf\" (UID: \"68c22006-b1b5-4f15-b504-be63d753819c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5758b7-29889675-hshgf" Oct 30 17:15:00.354493 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.347298 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d97qs\" (UniqueName: \"kubernetes.io/projected/e0665578-a6c7-4b36-ba0f-5144760dce17-kube-api-access-d97qs\") pod \"maas-api-key-cleanup-e2e-extmod-a-f94fb5-29889675-6w42p\" (UID: \"e0665578-a6c7-4b36-ba0f-5144760dce17\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f94fb5-29889675-6w42p" Oct 30 17:15:00.354493 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.347438 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68c22006-b1b5-4f15-b504-be63d753819c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-5758b7-29889675-hshgf\" (UID: \"68c22006-b1b5-4f15-b504-be63d753819c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5758b7-29889675-hshgf" Oct 30 17:15:00.354493 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.347483 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b72976e0-c501-44b4-b42c-5dfa26192e87-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-13c495-29889675-79wjh\" (UID: \"b72976e0-c501-44b4-b42c-5dfa26192e87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-13c495-29889675-79wjh" Oct 30 17:15:00.354493 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.347531 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/045dd183-6322-48cb-85df-fa7cc872f8cd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-2589b5-29889675-h7lhw\" (UID: \"045dd183-6322-48cb-85df-fa7cc872f8cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2589b5-29889675-h7lhw" Oct 30 17:15:00.354493 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.347576 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0665578-a6c7-4b36-ba0f-5144760dce17-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-f94fb5-29889675-6w42p\" (UID: \"e0665578-a6c7-4b36-ba0f-5144760dce17\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f94fb5-29889675-6w42p" Oct 30 17:15:00.354493 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.347631 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25dd3114-a3ba-4976-8f11-611923dc090f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-abda73-29889675-8gqs6\" (UID: \"25dd3114-a3ba-4976-8f11-611923dc090f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-abda73-29889675-8gqs6" Oct 30 17:15:00.354493 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.347760 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dk4t\" (UniqueName: \"kubernetes.io/projected/045dd183-6322-48cb-85df-fa7cc872f8cd-kube-api-access-6dk4t\") pod \"maas-api-key-cleanup-e2e-extmod-a-2589b5-29889675-h7lhw\" (UID: \"045dd183-6322-48cb-85df-fa7cc872f8cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2589b5-29889675-h7lhw" Oct 30 17:15:00.354493 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.347860 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2597\" (UniqueName: \"kubernetes.io/projected/25dd3114-a3ba-4976-8f11-611923dc090f-kube-api-access-q2597\") pod \"maas-api-key-cleanup-e2e-extmod-b-abda73-29889675-8gqs6\" (UID: \"25dd3114-a3ba-4976-8f11-611923dc090f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-abda73-29889675-8gqs6" Oct 30 17:15:00.354493 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.353765 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68c22006-b1b5-4f15-b504-be63d753819c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-5758b7-29889675-hshgf\" (UID: \"68c22006-b1b5-4f15-b504-be63d753819c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5758b7-29889675-hshgf" Oct 30 17:15:00.354493 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.354233 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b72976e0-c501-44b4-b42c-5dfa26192e87-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-13c495-29889675-79wjh\" (UID: \"b72976e0-c501-44b4-b42c-5dfa26192e87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-13c495-29889675-79wjh" Oct 30 17:15:00.373680 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.370449 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-43197b-29889675-mm8kj"] Oct 30 17:15:00.373680 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.371936 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d5b798-29889675-hhqqx" Oct 30 17:15:00.377597 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.377578 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3106d6-29889675-dtkjj"] Oct 30 17:15:00.379932 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.379911 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-43197b-29889675-mm8kj" Oct 30 17:15:00.385993 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.385968 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhkfk\" (UniqueName: \"kubernetes.io/projected/68c22006-b1b5-4f15-b504-be63d753819c-kube-api-access-vhkfk\") pod \"maas-api-key-cleanup-e2e-extmod-b-5758b7-29889675-hshgf\" (UID: \"68c22006-b1b5-4f15-b504-be63d753819c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5758b7-29889675-hshgf" Oct 30 17:15:00.386240 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.386217 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7wsh\" (UniqueName: \"kubernetes.io/projected/b72976e0-c501-44b4-b42c-5dfa26192e87-kube-api-access-x7wsh\") pod \"maas-api-key-cleanup-e2e-extmod-b-13c495-29889675-79wjh\" (UID: \"b72976e0-c501-44b4-b42c-5dfa26192e87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-13c495-29889675-79wjh" Oct 30 17:15:00.390329 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.390306 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5c6b3c-29889675-zznjt"] Oct 30 17:15:00.390446 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.390422 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3106d6-29889675-dtkjj" Oct 30 17:15:00.397617 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.397566 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5c6b3c-29889675-zznjt" Oct 30 17:15:00.397830 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.397733 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5758b7-29889675-hshgf"] Oct 30 17:15:00.398056 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.398045 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2589b5-29889675-h7lhw"] Oct 30 17:15:00.398167 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.398158 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-13c495-29889675-79wjh"] Oct 30 17:15:00.398265 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.398256 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cf4621-29889675-vl2xw"] Oct 30 17:15:00.402318 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.402302 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cf4621-29889675-vl2xw" Oct 30 17:15:00.402689 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.402462 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f94fb5-29889675-6w42p"] Oct 30 17:15:00.409842 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.409825 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dd715e-29889675-sjdts"] Oct 30 17:15:00.416245 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.416228 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dd715e-29889675-sjdts" Oct 30 17:15:00.420196 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.420178 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cb6251-29889675-tpztt"] Oct 30 17:15:00.433849 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.433676 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889675-nmmn8"] Oct 30 17:15:00.435228 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.435104 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cb6251-29889675-tpztt" Oct 30 17:15:00.439573 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:15:00.438478 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 17:15:00.450684 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.450641 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ff2bc89-4ecd-41d6-86ca-4111cba66bbd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-43197b-29889675-mm8kj\" (UID: \"5ff2bc89-4ecd-41d6-86ca-4111cba66bbd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-43197b-29889675-mm8kj" Oct 30 17:15:00.451031 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.451011 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d97qs\" (UniqueName: \"kubernetes.io/projected/e0665578-a6c7-4b36-ba0f-5144760dce17-kube-api-access-d97qs\") pod \"maas-api-key-cleanup-e2e-extmod-a-f94fb5-29889675-6w42p\" (UID: \"e0665578-a6c7-4b36-ba0f-5144760dce17\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f94fb5-29889675-6w42p" Oct 30 17:15:00.451209 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.451192 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e13c30d1-8241-414d-b8fb-a31f796a805a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-cf4621-29889675-vl2xw\" (UID: \"e13c30d1-8241-414d-b8fb-a31f796a805a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cf4621-29889675-vl2xw" Oct 30 17:15:00.451328 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.451315 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65k76\" (UniqueName: \"kubernetes.io/projected/c4086c1e-112f-4518-8ae2-3fa5526f9357-kube-api-access-65k76\") pod \"maas-api-key-cleanup-e2e-inf-a-6acc78-29889675-664bj\" (UID: \"c4086c1e-112f-4518-8ae2-3fa5526f9357\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6acc78-29889675-664bj" Oct 30 17:15:00.451502 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.451488 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/045dd183-6322-48cb-85df-fa7cc872f8cd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-2589b5-29889675-h7lhw\" (UID: \"045dd183-6322-48cb-85df-fa7cc872f8cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2589b5-29889675-h7lhw" Oct 30 17:15:00.451609 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.451594 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4086c1e-112f-4518-8ae2-3fa5526f9357-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-6acc78-29889675-664bj\" (UID: \"c4086c1e-112f-4518-8ae2-3fa5526f9357\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6acc78-29889675-664bj" Oct 30 17:15:00.451727 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.451715 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0665578-a6c7-4b36-ba0f-5144760dce17-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-f94fb5-29889675-6w42p\" (UID: \"e0665578-a6c7-4b36-ba0f-5144760dce17\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f94fb5-29889675-6w42p" Oct 30 17:15:00.451855 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.451841 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c296662-aa00-4ee9-90e5-2829d66ce1be-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-3106d6-29889675-dtkjj\" (UID: \"5c296662-aa00-4ee9-90e5-2829d66ce1be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3106d6-29889675-dtkjj" Oct 30 17:15:00.451966 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.451954 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkc7m\" (UniqueName: \"kubernetes.io/projected/632128dd-dd61-4807-8e04-6c809d0e4711-kube-api-access-zkc7m\") pod \"maas-api-key-cleanup-e2e-extmod-a-d5b798-29889675-hhqqx\" (UID: \"632128dd-dd61-4807-8e04-6c809d0e4711\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d5b798-29889675-hhqqx" Oct 30 17:15:00.452086 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.452073 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25dd3114-a3ba-4976-8f11-611923dc090f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-abda73-29889675-8gqs6\" (UID: \"25dd3114-a3ba-4976-8f11-611923dc090f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-abda73-29889675-8gqs6" Oct 30 17:15:00.452279 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.452264 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x7zk\" (UniqueName: \"kubernetes.io/projected/89814242-301f-48ab-a666-b8574e1e8116-kube-api-access-8x7zk\") pod \"maas-api-key-cleanup-e2e-extmod-b-5c6b3c-29889675-zznjt\" (UID: \"89814242-301f-48ab-a666-b8574e1e8116\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5c6b3c-29889675-zznjt" Oct 30 17:15:00.452442 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.452430 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dk4t\" (UniqueName: \"kubernetes.io/projected/045dd183-6322-48cb-85df-fa7cc872f8cd-kube-api-access-6dk4t\") pod \"maas-api-key-cleanup-e2e-extmod-a-2589b5-29889675-h7lhw\" (UID: \"045dd183-6322-48cb-85df-fa7cc872f8cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2589b5-29889675-h7lhw" Oct 30 17:15:00.452538 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.452526 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89814242-301f-48ab-a666-b8574e1e8116-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-5c6b3c-29889675-zznjt\" (UID: \"89814242-301f-48ab-a666-b8574e1e8116\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5c6b3c-29889675-zznjt" Oct 30 17:15:00.452660 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.452647 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/632128dd-dd61-4807-8e04-6c809d0e4711-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-d5b798-29889675-hhqqx\" (UID: \"632128dd-dd61-4807-8e04-6c809d0e4711\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d5b798-29889675-hhqqx" Oct 30 17:15:00.452771 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.452759 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2597\" (UniqueName: \"kubernetes.io/projected/25dd3114-a3ba-4976-8f11-611923dc090f-kube-api-access-q2597\") pod \"maas-api-key-cleanup-e2e-extmod-b-abda73-29889675-8gqs6\" (UID: \"25dd3114-a3ba-4976-8f11-611923dc090f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-abda73-29889675-8gqs6" Oct 30 17:15:00.452927 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.452912 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4n8n\" (UniqueName: \"kubernetes.io/projected/e13c30d1-8241-414d-b8fb-a31f796a805a-kube-api-access-m4n8n\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-cf4621-29889675-vl2xw\" (UID: \"e13c30d1-8241-414d-b8fb-a31f796a805a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cf4621-29889675-vl2xw" Oct 30 17:15:00.453054 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.453041 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcxv2\" (UniqueName: \"kubernetes.io/projected/5c296662-aa00-4ee9-90e5-2829d66ce1be-kube-api-access-tcxv2\") pod \"maas-api-key-cleanup-e2e-extmod-a-3106d6-29889675-dtkjj\" (UID: \"5c296662-aa00-4ee9-90e5-2829d66ce1be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3106d6-29889675-dtkjj" Oct 30 17:15:00.453178 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.453164 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvhrz\" (UniqueName: \"kubernetes.io/projected/5ff2bc89-4ecd-41d6-86ca-4111cba66bbd-kube-api-access-zvhrz\") pod \"maas-api-key-cleanup-e2e-extmod-b-43197b-29889675-mm8kj\" (UID: \"5ff2bc89-4ecd-41d6-86ca-4111cba66bbd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-43197b-29889675-mm8kj" Oct 30 17:15:00.454302 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.454282 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/045dd183-6322-48cb-85df-fa7cc872f8cd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-2589b5-29889675-h7lhw\" (UID: \"045dd183-6322-48cb-85df-fa7cc872f8cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2589b5-29889675-h7lhw" Oct 30 17:15:00.455051 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.455030 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0665578-a6c7-4b36-ba0f-5144760dce17-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-f94fb5-29889675-6w42p\" (UID: \"e0665578-a6c7-4b36-ba0f-5144760dce17\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f94fb5-29889675-6w42p" Oct 30 17:15:00.455800 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.455762 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25dd3114-a3ba-4976-8f11-611923dc090f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-abda73-29889675-8gqs6\" (UID: \"25dd3114-a3ba-4976-8f11-611923dc090f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-abda73-29889675-8gqs6" Oct 30 17:15:00.457035 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.456775 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6acc78-29889675-664bj"] Oct 30 17:15:00.457035 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.456913 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889675-nmmn8" Oct 30 17:15:00.463023 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.463005 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5c6b3c-29889675-zznjt"] Oct 30 17:15:00.477860 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.477841 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8bff6e-29889675-q2zhc"] Oct 30 17:15:00.480161 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.480131 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d97qs\" (UniqueName: \"kubernetes.io/projected/e0665578-a6c7-4b36-ba0f-5144760dce17-kube-api-access-d97qs\") pod \"maas-api-key-cleanup-e2e-extmod-a-f94fb5-29889675-6w42p\" (UID: \"e0665578-a6c7-4b36-ba0f-5144760dce17\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f94fb5-29889675-6w42p" Oct 30 17:15:00.482284 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.482262 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2597\" (UniqueName: \"kubernetes.io/projected/25dd3114-a3ba-4976-8f11-611923dc090f-kube-api-access-q2597\") pod \"maas-api-key-cleanup-e2e-extmod-b-abda73-29889675-8gqs6\" (UID: \"25dd3114-a3ba-4976-8f11-611923dc090f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-abda73-29889675-8gqs6" Oct 30 17:15:00.482871 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.482852 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cf4621-29889675-vl2xw"] Oct 30 17:15:00.482983 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.482950 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8bff6e-29889675-q2zhc" Oct 30 17:15:00.483630 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.483609 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dk4t\" (UniqueName: \"kubernetes.io/projected/045dd183-6322-48cb-85df-fa7cc872f8cd-kube-api-access-6dk4t\") pod \"maas-api-key-cleanup-e2e-extmod-a-2589b5-29889675-h7lhw\" (UID: \"045dd183-6322-48cb-85df-fa7cc872f8cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2589b5-29889675-h7lhw" Oct 30 17:15:00.487208 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.487190 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889675-d8tcq"] Oct 30 17:15:00.492195 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.492178 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889675-d8tcq" Oct 30 17:15:00.492848 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.492832 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-43197b-29889675-mm8kj"] Oct 30 17:15:00.498703 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.498680 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-06e7f6-29889675-m9wc6"] Oct 30 17:15:00.502356 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.502336 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-06e7f6-29889675-m9wc6" Oct 30 17:15:00.504018 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.503944 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889675-kxmbh"] Oct 30 17:15:00.511262 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.511235 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cb6251-29889675-tpztt"] Oct 30 17:15:00.511389 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.511374 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889675-kxmbh" Oct 30 17:15:00.517166 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.517145 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24acb2-29889675-wttj8"] Oct 30 17:15:00.523911 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.523889 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24acb2-29889675-wttj8" Oct 30 17:15:00.524628 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.524614 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-abda73-29889675-8gqs6"] Oct 30 17:15:00.535887 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.535866 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5758b7-29889675-hshgf" Oct 30 17:15:00.537625 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.537605 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d5b798-29889675-hhqqx"] Oct 30 17:15:00.544873 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.544441 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8bff6e-29889675-q2zhc"] Oct 30 17:15:00.566452 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.566422 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889675-kxmbh"] Oct 30 17:15:00.567405 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.567377 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-06e7f6-29889675-m9wc6"] Oct 30 17:15:00.570341 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.570316 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c296662-aa00-4ee9-90e5-2829d66ce1be-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-3106d6-29889675-dtkjj\" (UID: \"5c296662-aa00-4ee9-90e5-2829d66ce1be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3106d6-29889675-dtkjj" Oct 30 17:15:00.570519 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.570500 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkc7m\" (UniqueName: \"kubernetes.io/projected/632128dd-dd61-4807-8e04-6c809d0e4711-kube-api-access-zkc7m\") pod \"maas-api-key-cleanup-e2e-extmod-a-d5b798-29889675-hhqqx\" (UID: \"632128dd-dd61-4807-8e04-6c809d0e4711\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d5b798-29889675-hhqqx" Oct 30 17:15:00.570666 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.570652 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x7zk\" (UniqueName: \"kubernetes.io/projected/89814242-301f-48ab-a666-b8574e1e8116-kube-api-access-8x7zk\") pod \"maas-api-key-cleanup-e2e-extmod-b-5c6b3c-29889675-zznjt\" (UID: \"89814242-301f-48ab-a666-b8574e1e8116\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5c6b3c-29889675-zznjt" Oct 30 17:15:00.570843 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.570825 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac7c8bb8-9c6e-4b0e-bb84-2e70fc863d96-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-dd715e-29889675-sjdts\" (UID: \"ac7c8bb8-9c6e-4b0e-bb84-2e70fc863d96\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dd715e-29889675-sjdts" Oct 30 17:15:00.571001 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.570985 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89814242-301f-48ab-a666-b8574e1e8116-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-5c6b3c-29889675-zznjt\" (UID: \"89814242-301f-48ab-a666-b8574e1e8116\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5c6b3c-29889675-zznjt" Oct 30 17:15:00.571132 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.571118 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57140245-e2df-4ee9-9344-1de9618511d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-2d365c-29889675-nmmn8\" (UID: \"57140245-e2df-4ee9-9344-1de9618511d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889675-nmmn8" Oct 30 17:15:00.571243 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.571228 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9988e846-2f00-4c0a-b72e-b2943e79f01b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-18519c-29889675-kxmbh\" (UID: \"9988e846-2f00-4c0a-b72e-b2943e79f01b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889675-kxmbh" Oct 30 17:15:00.571381 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.571337 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/632128dd-dd61-4807-8e04-6c809d0e4711-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-d5b798-29889675-hhqqx\" (UID: \"632128dd-dd61-4807-8e04-6c809d0e4711\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d5b798-29889675-hhqqx" Oct 30 17:15:00.571511 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.571496 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4n8n\" (UniqueName: \"kubernetes.io/projected/e13c30d1-8241-414d-b8fb-a31f796a805a-kube-api-access-m4n8n\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-cf4621-29889675-vl2xw\" (UID: \"e13c30d1-8241-414d-b8fb-a31f796a805a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cf4621-29889675-vl2xw" Oct 30 17:15:00.571664 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.571642 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d2cd04d-1a67-45a1-8934-f9d4c9435523-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-06e7f6-29889675-m9wc6\" (UID: \"7d2cd04d-1a67-45a1-8934-f9d4c9435523\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-06e7f6-29889675-m9wc6" Oct 30 17:15:00.571796 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.571767 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrs54\" (UniqueName: \"kubernetes.io/projected/ac7c8bb8-9c6e-4b0e-bb84-2e70fc863d96-kube-api-access-zrs54\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-dd715e-29889675-sjdts\" (UID: \"ac7c8bb8-9c6e-4b0e-bb84-2e70fc863d96\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dd715e-29889675-sjdts" Oct 30 17:15:00.571921 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.571906 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcxv2\" (UniqueName: \"kubernetes.io/projected/5c296662-aa00-4ee9-90e5-2829d66ce1be-kube-api-access-tcxv2\") pod \"maas-api-key-cleanup-e2e-extmod-a-3106d6-29889675-dtkjj\" (UID: \"5c296662-aa00-4ee9-90e5-2829d66ce1be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3106d6-29889675-dtkjj" Oct 30 17:15:00.572046 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.572031 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2gh4\" (UniqueName: \"kubernetes.io/projected/40a120b1-71f5-40ab-8faf-da2052ab0a56-kube-api-access-h2gh4\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-8bff6e-29889675-q2zhc\" (UID: \"40a120b1-71f5-40ab-8faf-da2052ab0a56\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8bff6e-29889675-q2zhc" Oct 30 17:15:00.572325 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.572167 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvhrz\" (UniqueName: \"kubernetes.io/projected/5ff2bc89-4ecd-41d6-86ca-4111cba66bbd-kube-api-access-zvhrz\") pod \"maas-api-key-cleanup-e2e-extmod-b-43197b-29889675-mm8kj\" (UID: \"5ff2bc89-4ecd-41d6-86ca-4111cba66bbd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-43197b-29889675-mm8kj" Oct 30 17:15:00.574640 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.572608 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89814242-301f-48ab-a666-b8574e1e8116-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-5c6b3c-29889675-zznjt\" (UID: \"89814242-301f-48ab-a666-b8574e1e8116\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5c6b3c-29889675-zznjt" Oct 30 17:15:00.574640 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.574225 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/632128dd-dd61-4807-8e04-6c809d0e4711-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-d5b798-29889675-hhqqx\" (UID: \"632128dd-dd61-4807-8e04-6c809d0e4711\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d5b798-29889675-hhqqx" Oct 30 17:15:00.574640 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.574587 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ff2bc89-4ecd-41d6-86ca-4111cba66bbd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-43197b-29889675-mm8kj\" (UID: \"5ff2bc89-4ecd-41d6-86ca-4111cba66bbd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-43197b-29889675-mm8kj" Oct 30 17:15:00.574876 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.574640 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kzfv\" (UniqueName: \"kubernetes.io/projected/9988e846-2f00-4c0a-b72e-b2943e79f01b-kube-api-access-7kzfv\") pod \"maas-api-key-cleanup-e2e-worker-w2-18519c-29889675-kxmbh\" (UID: \"9988e846-2f00-4c0a-b72e-b2943e79f01b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889675-kxmbh" Oct 30 17:15:00.574876 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.574702 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhspq\" (UniqueName: \"kubernetes.io/projected/53dc2f29-cd5a-4a6b-8e97-cc2d60925ade-kube-api-access-lhspq\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-cb6251-29889675-tpztt\" (UID: \"53dc2f29-cd5a-4a6b-8e97-cc2d60925ade\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cb6251-29889675-tpztt" Oct 30 17:15:00.574876 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.574743 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43be3031-1e16-4687-97f9-f7af506d98b5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f3503b-29889675-d8tcq\" (UID: \"43be3031-1e16-4687-97f9-f7af506d98b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889675-d8tcq" Oct 30 17:15:00.574876 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.574849 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40a120b1-71f5-40ab-8faf-da2052ab0a56-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-8bff6e-29889675-q2zhc\" (UID: \"40a120b1-71f5-40ab-8faf-da2052ab0a56\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8bff6e-29889675-q2zhc" Oct 30 17:15:00.575085 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.574899 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e13c30d1-8241-414d-b8fb-a31f796a805a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-cf4621-29889675-vl2xw\" (UID: \"e13c30d1-8241-414d-b8fb-a31f796a805a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cf4621-29889675-vl2xw" Oct 30 17:15:00.575085 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.574955 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65k76\" (UniqueName: \"kubernetes.io/projected/c4086c1e-112f-4518-8ae2-3fa5526f9357-kube-api-access-65k76\") pod \"maas-api-key-cleanup-e2e-inf-a-6acc78-29889675-664bj\" (UID: \"c4086c1e-112f-4518-8ae2-3fa5526f9357\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6acc78-29889675-664bj" Oct 30 17:15:00.575209 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.575092 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lppp\" (UniqueName: \"kubernetes.io/projected/57140245-e2df-4ee9-9344-1de9618511d0-kube-api-access-6lppp\") pod \"maas-api-key-cleanup-e2e-worker-w3-2d365c-29889675-nmmn8\" (UID: \"57140245-e2df-4ee9-9344-1de9618511d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889675-nmmn8" Oct 30 17:15:00.575209 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.575150 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53dc2f29-cd5a-4a6b-8e97-cc2d60925ade-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-cb6251-29889675-tpztt\" (UID: \"53dc2f29-cd5a-4a6b-8e97-cc2d60925ade\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cb6251-29889675-tpztt" Oct 30 17:15:00.575209 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.575194 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdzqc\" (UniqueName: \"kubernetes.io/projected/43be3031-1e16-4687-97f9-f7af506d98b5-kube-api-access-sdzqc\") pod \"maas-api-key-cleanup-e2e-worker-w0-f3503b-29889675-d8tcq\" (UID: \"43be3031-1e16-4687-97f9-f7af506d98b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889675-d8tcq" Oct 30 17:15:00.575634 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.575608 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg6lv\" (UniqueName: \"kubernetes.io/projected/7d2cd04d-1a67-45a1-8934-f9d4c9435523-kube-api-access-mg6lv\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-06e7f6-29889675-m9wc6\" (UID: \"7d2cd04d-1a67-45a1-8934-f9d4c9435523\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-06e7f6-29889675-m9wc6" Oct 30 17:15:00.575885 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.575865 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4086c1e-112f-4518-8ae2-3fa5526f9357-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-6acc78-29889675-664bj\" (UID: \"c4086c1e-112f-4518-8ae2-3fa5526f9357\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6acc78-29889675-664bj" Oct 30 17:15:00.577163 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.577130 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e13c30d1-8241-414d-b8fb-a31f796a805a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-cf4621-29889675-vl2xw\" (UID: \"e13c30d1-8241-414d-b8fb-a31f796a805a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cf4621-29889675-vl2xw" Oct 30 17:15:00.577299 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.577206 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ff2bc89-4ecd-41d6-86ca-4111cba66bbd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-43197b-29889675-mm8kj\" (UID: \"5ff2bc89-4ecd-41d6-86ca-4111cba66bbd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-43197b-29889675-mm8kj" Oct 30 17:15:00.577378 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.577311 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4086c1e-112f-4518-8ae2-3fa5526f9357-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-6acc78-29889675-664bj\" (UID: \"c4086c1e-112f-4518-8ae2-3fa5526f9357\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6acc78-29889675-664bj" Oct 30 17:15:00.577742 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.577723 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c296662-aa00-4ee9-90e5-2829d66ce1be-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-3106d6-29889675-dtkjj\" (UID: \"5c296662-aa00-4ee9-90e5-2829d66ce1be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3106d6-29889675-dtkjj" Oct 30 17:15:00.581340 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.581321 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889675-d8tcq"] Oct 30 17:15:00.587864 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.587845 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3106d6-29889675-dtkjj"] Oct 30 17:15:00.595822 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.595764 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dd715e-29889675-sjdts"] Oct 30 17:15:00.597303 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.597115 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-13c495-29889675-79wjh" Oct 30 17:15:00.601167 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.600352 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889675-nmmn8"] Oct 30 17:15:00.605576 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.605158 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x7zk\" (UniqueName: \"kubernetes.io/projected/89814242-301f-48ab-a666-b8574e1e8116-kube-api-access-8x7zk\") pod \"maas-api-key-cleanup-e2e-extmod-b-5c6b3c-29889675-zznjt\" (UID: \"89814242-301f-48ab-a666-b8574e1e8116\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5c6b3c-29889675-zznjt" Oct 30 17:15:00.607290 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.606878 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvhrz\" (UniqueName: \"kubernetes.io/projected/5ff2bc89-4ecd-41d6-86ca-4111cba66bbd-kube-api-access-zvhrz\") pod \"maas-api-key-cleanup-e2e-extmod-b-43197b-29889675-mm8kj\" (UID: \"5ff2bc89-4ecd-41d6-86ca-4111cba66bbd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-43197b-29889675-mm8kj" Oct 30 17:15:00.607906 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.607689 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4n8n\" (UniqueName: \"kubernetes.io/projected/e13c30d1-8241-414d-b8fb-a31f796a805a-kube-api-access-m4n8n\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-cf4621-29889675-vl2xw\" (UID: \"e13c30d1-8241-414d-b8fb-a31f796a805a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cf4621-29889675-vl2xw" Oct 30 17:15:00.611065 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.611025 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65k76\" (UniqueName: \"kubernetes.io/projected/c4086c1e-112f-4518-8ae2-3fa5526f9357-kube-api-access-65k76\") pod \"maas-api-key-cleanup-e2e-inf-a-6acc78-29889675-664bj\" (UID: \"c4086c1e-112f-4518-8ae2-3fa5526f9357\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6acc78-29889675-664bj" Oct 30 17:15:00.611362 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.611330 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcxv2\" (UniqueName: \"kubernetes.io/projected/5c296662-aa00-4ee9-90e5-2829d66ce1be-kube-api-access-tcxv2\") pod \"maas-api-key-cleanup-e2e-extmod-a-3106d6-29889675-dtkjj\" (UID: \"5c296662-aa00-4ee9-90e5-2829d66ce1be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3106d6-29889675-dtkjj" Oct 30 17:15:00.616510 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.613940 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkc7m\" (UniqueName: \"kubernetes.io/projected/632128dd-dd61-4807-8e04-6c809d0e4711-kube-api-access-zkc7m\") pod \"maas-api-key-cleanup-e2e-extmod-a-d5b798-29889675-hhqqx\" (UID: \"632128dd-dd61-4807-8e04-6c809d0e4711\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d5b798-29889675-hhqqx" Oct 30 17:15:00.619189 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.619122 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24acb2-29889675-wttj8"] Oct 30 17:15:00.620050 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.620023 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2589b5-29889675-h7lhw" Oct 30 17:15:00.667078 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.662681 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f94fb5-29889675-6w42p" Oct 30 17:15:00.680399 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.680087 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac7c8bb8-9c6e-4b0e-bb84-2e70fc863d96-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-dd715e-29889675-sjdts\" (UID: \"ac7c8bb8-9c6e-4b0e-bb84-2e70fc863d96\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dd715e-29889675-sjdts" Oct 30 17:15:00.680399 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.680166 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57140245-e2df-4ee9-9344-1de9618511d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-2d365c-29889675-nmmn8\" (UID: \"57140245-e2df-4ee9-9344-1de9618511d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889675-nmmn8" Oct 30 17:15:00.680399 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.680201 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9988e846-2f00-4c0a-b72e-b2943e79f01b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-18519c-29889675-kxmbh\" (UID: \"9988e846-2f00-4c0a-b72e-b2943e79f01b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889675-kxmbh" Oct 30 17:15:00.680399 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.680267 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d2cd04d-1a67-45a1-8934-f9d4c9435523-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-06e7f6-29889675-m9wc6\" (UID: \"7d2cd04d-1a67-45a1-8934-f9d4c9435523\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-06e7f6-29889675-m9wc6" Oct 30 17:15:00.680399 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.680300 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrs54\" (UniqueName: \"kubernetes.io/projected/ac7c8bb8-9c6e-4b0e-bb84-2e70fc863d96-kube-api-access-zrs54\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-dd715e-29889675-sjdts\" (UID: \"ac7c8bb8-9c6e-4b0e-bb84-2e70fc863d96\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dd715e-29889675-sjdts" Oct 30 17:15:00.680399 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.680344 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2gh4\" (UniqueName: \"kubernetes.io/projected/40a120b1-71f5-40ab-8faf-da2052ab0a56-kube-api-access-h2gh4\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-8bff6e-29889675-q2zhc\" (UID: \"40a120b1-71f5-40ab-8faf-da2052ab0a56\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8bff6e-29889675-q2zhc" Oct 30 17:15:00.680399 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.680380 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49589f83-94f8-4e46-808f-cb013c6adfeb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-24acb2-29889675-wttj8\" (UID: \"49589f83-94f8-4e46-808f-cb013c6adfeb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24acb2-29889675-wttj8" Oct 30 17:15:00.680895 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.680437 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kzfv\" (UniqueName: \"kubernetes.io/projected/9988e846-2f00-4c0a-b72e-b2943e79f01b-kube-api-access-7kzfv\") pod \"maas-api-key-cleanup-e2e-worker-w2-18519c-29889675-kxmbh\" (UID: \"9988e846-2f00-4c0a-b72e-b2943e79f01b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889675-kxmbh" Oct 30 17:15:00.680895 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.680481 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhspq\" (UniqueName: \"kubernetes.io/projected/53dc2f29-cd5a-4a6b-8e97-cc2d60925ade-kube-api-access-lhspq\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-cb6251-29889675-tpztt\" (UID: \"53dc2f29-cd5a-4a6b-8e97-cc2d60925ade\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cb6251-29889675-tpztt" Oct 30 17:15:00.680895 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.680519 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43be3031-1e16-4687-97f9-f7af506d98b5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f3503b-29889675-d8tcq\" (UID: \"43be3031-1e16-4687-97f9-f7af506d98b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889675-d8tcq" Oct 30 17:15:00.680895 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.680570 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40a120b1-71f5-40ab-8faf-da2052ab0a56-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-8bff6e-29889675-q2zhc\" (UID: \"40a120b1-71f5-40ab-8faf-da2052ab0a56\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8bff6e-29889675-q2zhc" Oct 30 17:15:00.680895 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.680696 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lppp\" (UniqueName: \"kubernetes.io/projected/57140245-e2df-4ee9-9344-1de9618511d0-kube-api-access-6lppp\") pod \"maas-api-key-cleanup-e2e-worker-w3-2d365c-29889675-nmmn8\" (UID: \"57140245-e2df-4ee9-9344-1de9618511d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889675-nmmn8" Oct 30 17:15:00.680895 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.680729 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53dc2f29-cd5a-4a6b-8e97-cc2d60925ade-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-cb6251-29889675-tpztt\" (UID: \"53dc2f29-cd5a-4a6b-8e97-cc2d60925ade\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cb6251-29889675-tpztt" Oct 30 17:15:00.680895 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.680759 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdzqc\" (UniqueName: \"kubernetes.io/projected/43be3031-1e16-4687-97f9-f7af506d98b5-kube-api-access-sdzqc\") pod \"maas-api-key-cleanup-e2e-worker-w0-f3503b-29889675-d8tcq\" (UID: \"43be3031-1e16-4687-97f9-f7af506d98b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889675-d8tcq" Oct 30 17:15:00.680895 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.680807 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gccm\" (UniqueName: \"kubernetes.io/projected/49589f83-94f8-4e46-808f-cb013c6adfeb-kube-api-access-9gccm\") pod \"maas-api-key-cleanup-e2e-inf-b-24acb2-29889675-wttj8\" (UID: \"49589f83-94f8-4e46-808f-cb013c6adfeb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24acb2-29889675-wttj8" Oct 30 17:15:00.680895 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.680837 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg6lv\" (UniqueName: \"kubernetes.io/projected/7d2cd04d-1a67-45a1-8934-f9d4c9435523-kube-api-access-mg6lv\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-06e7f6-29889675-m9wc6\" (UID: \"7d2cd04d-1a67-45a1-8934-f9d4c9435523\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-06e7f6-29889675-m9wc6" Oct 30 17:15:00.685422 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.682102 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac7c8bb8-9c6e-4b0e-bb84-2e70fc863d96-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-dd715e-29889675-sjdts\" (UID: \"ac7c8bb8-9c6e-4b0e-bb84-2e70fc863d96\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dd715e-29889675-sjdts" Oct 30 17:15:00.685422 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.682917 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57140245-e2df-4ee9-9344-1de9618511d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-2d365c-29889675-nmmn8\" (UID: \"57140245-e2df-4ee9-9344-1de9618511d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889675-nmmn8" Oct 30 17:15:00.685422 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.683331 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9988e846-2f00-4c0a-b72e-b2943e79f01b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-18519c-29889675-kxmbh\" (UID: \"9988e846-2f00-4c0a-b72e-b2943e79f01b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889675-kxmbh" Oct 30 17:15:00.685422 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.683527 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43be3031-1e16-4687-97f9-f7af506d98b5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f3503b-29889675-d8tcq\" (UID: \"43be3031-1e16-4687-97f9-f7af506d98b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889675-d8tcq" Oct 30 17:15:00.685422 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.684172 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40a120b1-71f5-40ab-8faf-da2052ab0a56-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-8bff6e-29889675-q2zhc\" (UID: \"40a120b1-71f5-40ab-8faf-da2052ab0a56\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8bff6e-29889675-q2zhc" Oct 30 17:15:00.685422 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.684323 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d2cd04d-1a67-45a1-8934-f9d4c9435523-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-06e7f6-29889675-m9wc6\" (UID: \"7d2cd04d-1a67-45a1-8934-f9d4c9435523\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-06e7f6-29889675-m9wc6" Oct 30 17:15:00.685422 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.685319 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53dc2f29-cd5a-4a6b-8e97-cc2d60925ade-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-cb6251-29889675-tpztt\" (UID: \"53dc2f29-cd5a-4a6b-8e97-cc2d60925ade\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cb6251-29889675-tpztt" Oct 30 17:15:00.685822 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.685530 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-abda73-29889675-8gqs6" Oct 30 17:15:00.780226 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.766377 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6acc78-29889675-664bj" Oct 30 17:15:00.812091 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.807653 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrs54\" (UniqueName: \"kubernetes.io/projected/ac7c8bb8-9c6e-4b0e-bb84-2e70fc863d96-kube-api-access-zrs54\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-dd715e-29889675-sjdts\" (UID: \"ac7c8bb8-9c6e-4b0e-bb84-2e70fc863d96\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dd715e-29889675-sjdts" Oct 30 17:15:00.813151 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.813119 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdzqc\" (UniqueName: \"kubernetes.io/projected/43be3031-1e16-4687-97f9-f7af506d98b5-kube-api-access-sdzqc\") pod \"maas-api-key-cleanup-e2e-worker-w0-f3503b-29889675-d8tcq\" (UID: \"43be3031-1e16-4687-97f9-f7af506d98b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889675-d8tcq" Oct 30 17:15:00.817487 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.814983 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gccm\" (UniqueName: \"kubernetes.io/projected/49589f83-94f8-4e46-808f-cb013c6adfeb-kube-api-access-9gccm\") pod \"maas-api-key-cleanup-e2e-inf-b-24acb2-29889675-wttj8\" (UID: \"49589f83-94f8-4e46-808f-cb013c6adfeb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24acb2-29889675-wttj8" Oct 30 17:15:00.817487 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.815405 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg6lv\" (UniqueName: \"kubernetes.io/projected/7d2cd04d-1a67-45a1-8934-f9d4c9435523-kube-api-access-mg6lv\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-06e7f6-29889675-m9wc6\" (UID: \"7d2cd04d-1a67-45a1-8934-f9d4c9435523\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-06e7f6-29889675-m9wc6" Oct 30 17:15:00.817487 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.815405 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lppp\" (UniqueName: \"kubernetes.io/projected/57140245-e2df-4ee9-9344-1de9618511d0-kube-api-access-6lppp\") pod \"maas-api-key-cleanup-e2e-worker-w3-2d365c-29889675-nmmn8\" (UID: \"57140245-e2df-4ee9-9344-1de9618511d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889675-nmmn8" Oct 30 17:15:00.817487 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.816170 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-43197b-29889675-mm8kj" Oct 30 17:15:00.817487 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.816325 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49589f83-94f8-4e46-808f-cb013c6adfeb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-24acb2-29889675-wttj8\" (UID: \"49589f83-94f8-4e46-808f-cb013c6adfeb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24acb2-29889675-wttj8" Oct 30 17:15:00.817487 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.817070 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3106d6-29889675-dtkjj" Oct 30 17:15:00.817487 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.817159 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d5b798-29889675-hhqqx" Oct 30 17:15:00.817487 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.817477 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49589f83-94f8-4e46-808f-cb013c6adfeb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-24acb2-29889675-wttj8\" (UID: \"49589f83-94f8-4e46-808f-cb013c6adfeb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24acb2-29889675-wttj8" Oct 30 17:15:00.823356 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.823328 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kzfv\" (UniqueName: \"kubernetes.io/projected/9988e846-2f00-4c0a-b72e-b2943e79f01b-kube-api-access-7kzfv\") pod \"maas-api-key-cleanup-e2e-worker-w2-18519c-29889675-kxmbh\" (UID: \"9988e846-2f00-4c0a-b72e-b2943e79f01b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889675-kxmbh" Oct 30 17:15:00.828192 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.828129 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5c6b3c-29889675-zznjt" Oct 30 17:15:00.828533 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.828505 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhspq\" (UniqueName: \"kubernetes.io/projected/53dc2f29-cd5a-4a6b-8e97-cc2d60925ade-kube-api-access-lhspq\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-cb6251-29889675-tpztt\" (UID: \"53dc2f29-cd5a-4a6b-8e97-cc2d60925ade\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cb6251-29889675-tpztt" Oct 30 17:15:00.828636 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.828615 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2gh4\" (UniqueName: \"kubernetes.io/projected/40a120b1-71f5-40ab-8faf-da2052ab0a56-kube-api-access-h2gh4\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-8bff6e-29889675-q2zhc\" (UID: \"40a120b1-71f5-40ab-8faf-da2052ab0a56\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8bff6e-29889675-q2zhc" Oct 30 17:15:00.839089 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.839060 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cf4621-29889675-vl2xw" Oct 30 17:15:00.846457 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.846397 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gccm\" (UniqueName: \"kubernetes.io/projected/49589f83-94f8-4e46-808f-cb013c6adfeb-kube-api-access-9gccm\") pod \"maas-api-key-cleanup-e2e-inf-b-24acb2-29889675-wttj8\" (UID: \"49589f83-94f8-4e46-808f-cb013c6adfeb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24acb2-29889675-wttj8" Oct 30 17:15:00.882319 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.882281 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dd715e-29889675-sjdts" Oct 30 17:15:00.888140 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.888113 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cb6251-29889675-tpztt" Oct 30 17:15:00.899165 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.899125 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889675-nmmn8" Oct 30 17:15:00.920955 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.920677 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8bff6e-29889675-q2zhc" Oct 30 17:15:00.951998 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.950346 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889675-d8tcq" Oct 30 17:15:00.997814 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:00.997375 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-06e7f6-29889675-m9wc6" Oct 30 17:15:01.105498 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:01.105240 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889675-kxmbh" Oct 30 17:15:01.144161 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:01.144046 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24acb2-29889675-wttj8" Oct 30 17:15:01.316712 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:01.316677 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-13c495-29889675-79wjh"] Oct 30 17:15:01.326104 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:01.326072 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5758b7-29889675-hshgf"] Oct 30 17:15:01.688420 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:01.688389 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2589b5-29889675-h7lhw"] Oct 30 17:15:01.709704 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:01.709672 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2589b5-29889675-h7lhw" event={"ID":"045dd183-6322-48cb-85df-fa7cc872f8cd","Type":"ContainerStarted","Data":"a2bac27902e2208992516933f74af78c8b95e7fdaecc86b75e3be7b6c638364b"} Oct 30 17:15:01.711522 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:01.711470 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-13c495-29889675-79wjh" event={"ID":"b72976e0-c501-44b4-b42c-5dfa26192e87","Type":"ContainerStarted","Data":"80baf4e07396d3da64b406253cbc1f2c5aca71b159b921b1f69f94d8cfb4c6fe"} Oct 30 17:15:01.713629 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:01.713607 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5758b7-29889675-hshgf" event={"ID":"68c22006-b1b5-4f15-b504-be63d753819c","Type":"ContainerStarted","Data":"f58e55958ca3cb4a9ef60cce1c455976558cfbead0006bcb347f963d4dd75a4e"} Oct 30 17:15:01.731675 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:01.731646 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cf4621-29889675-vl2xw"] Oct 30 17:15:01.784345 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:01.784312 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3106d6-29889675-dtkjj"] Oct 30 17:15:01.817639 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:15:01.817604 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode13c30d1_8241_414d_b8fb_a31f796a805a.slice/crio-02892a1e40d0072539a1bbd1f0adfa80655f7affdbcf45410e302f480cb33793 WatchSource:0}: Error finding container 02892a1e40d0072539a1bbd1f0adfa80655f7affdbcf45410e302f480cb33793: Status 404 returned error can't find the container with id 02892a1e40d0072539a1bbd1f0adfa80655f7affdbcf45410e302f480cb33793 Oct 30 17:15:01.840421 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:15:01.840381 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4086c1e_112f_4518_8ae2_3fa5526f9357.slice/crio-49220f40f51b730842615758b0fa20f2e9cd17f4672239c2ef3549f3c92a9ac5 WatchSource:0}: Error finding container 49220f40f51b730842615758b0fa20f2e9cd17f4672239c2ef3549f3c92a9ac5: Status 404 returned error can't find the container with id 49220f40f51b730842615758b0fa20f2e9cd17f4672239c2ef3549f3c92a9ac5 Oct 30 17:15:01.850724 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:01.850674 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6acc78-29889675-664bj"] Oct 30 17:15:01.860428 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:01.860402 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f94fb5-29889675-6w42p"] Oct 30 17:15:01.866911 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:01.866890 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889675-d8tcq"] Oct 30 17:15:01.873173 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:01.873140 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cb6251-29889675-tpztt"] Oct 30 17:15:01.887778 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:01.887088 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-43197b-29889675-mm8kj"] Oct 30 17:15:01.942125 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:01.941891 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5c6b3c-29889675-zznjt"] Oct 30 17:15:01.952082 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:01.952055 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-abda73-29889675-8gqs6"] Oct 30 17:15:01.963575 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:01.963475 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889675-nmmn8"] Oct 30 17:15:01.969686 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:01.969658 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d5b798-29889675-hhqqx"] Oct 30 17:15:01.992533 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:01.992505 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24acb2-29889675-wttj8"] Oct 30 17:15:02.015871 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:02.015736 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-06e7f6-29889675-m9wc6"] Oct 30 17:15:02.022064 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:02.022039 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889675-kxmbh"] Oct 30 17:15:02.138940 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:02.138915 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dd715e-29889675-sjdts"] Oct 30 17:15:02.146192 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:02.146165 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8bff6e-29889675-q2zhc"] Oct 30 17:15:02.146398 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:15:02.146318 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d2cd04d_1a67_45a1_8934_f9d4c9435523.slice/crio-7979981d2aa15242228c3746654b0ba39d1e39e416f588e4c5be3a8909d33da8 WatchSource:0}: Error finding container 7979981d2aa15242228c3746654b0ba39d1e39e416f588e4c5be3a8909d33da8: Status 404 returned error can't find the container with id 7979981d2aa15242228c3746654b0ba39d1e39e416f588e4c5be3a8909d33da8 Oct 30 17:15:02.148747 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:15:02.148720 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9988e846_2f00_4c0a_b72e_b2943e79f01b.slice/crio-2bc4d6d3e595e9fdea11817fb7f184e02732341271cf5c6401125fc2d98ff2e9 WatchSource:0}: Error finding container 2bc4d6d3e595e9fdea11817fb7f184e02732341271cf5c6401125fc2d98ff2e9: Status 404 returned error can't find the container with id 2bc4d6d3e595e9fdea11817fb7f184e02732341271cf5c6401125fc2d98ff2e9 Oct 30 17:15:02.734070 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:02.733988 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889675-kxmbh" event={"ID":"9988e846-2f00-4c0a-b72e-b2943e79f01b","Type":"ContainerStarted","Data":"2bc4d6d3e595e9fdea11817fb7f184e02732341271cf5c6401125fc2d98ff2e9"} Oct 30 17:15:02.736682 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:02.736607 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f94fb5-29889675-6w42p" event={"ID":"e0665578-a6c7-4b36-ba0f-5144760dce17","Type":"ContainerStarted","Data":"286b223fdaa8f36f362c17760172878c07dd1d97fd223bc4f787848283aa8903"} Oct 30 17:15:02.740862 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:02.740314 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8bff6e-29889675-q2zhc" event={"ID":"40a120b1-71f5-40ab-8faf-da2052ab0a56","Type":"ContainerStarted","Data":"69aa4f2f7fc330138ca66573efcb5932b9b1592bdee1f5f3dd98cde4ab05c380"} Oct 30 17:15:02.743703 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:02.742196 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-abda73-29889675-8gqs6" event={"ID":"25dd3114-a3ba-4976-8f11-611923dc090f","Type":"ContainerStarted","Data":"6059761abf4f1f5f2c4ca6507eaec0a54e6ac1cb8c9752daa2add833da49e916"} Oct 30 17:15:02.745604 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:02.745579 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889675-nmmn8" event={"ID":"57140245-e2df-4ee9-9344-1de9618511d0","Type":"ContainerStarted","Data":"a6d8ec3d8ae273dddb8a9c88cbb2b05f75bc2047959d7dc62d43dc2a7b74937d"} Oct 30 17:15:02.753227 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:02.752265 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-06e7f6-29889675-m9wc6" event={"ID":"7d2cd04d-1a67-45a1-8934-f9d4c9435523","Type":"ContainerStarted","Data":"7979981d2aa15242228c3746654b0ba39d1e39e416f588e4c5be3a8909d33da8"} Oct 30 17:15:02.755129 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:02.755102 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3106d6-29889675-dtkjj" event={"ID":"5c296662-aa00-4ee9-90e5-2829d66ce1be","Type":"ContainerStarted","Data":"2769a2bd72d663fd14194b0f6d3153024d44441192903add7b095bed3f838f7a"} Oct 30 17:15:02.759145 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:02.759116 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cb6251-29889675-tpztt" event={"ID":"53dc2f29-cd5a-4a6b-8e97-cc2d60925ade","Type":"ContainerStarted","Data":"afea8a23d9d8b0e0dbec5de6341670189e36813a2a5335afff3d0e00824a3b88"} Oct 30 17:15:02.765038 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:02.764964 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cf4621-29889675-vl2xw" event={"ID":"e13c30d1-8241-414d-b8fb-a31f796a805a","Type":"ContainerStarted","Data":"02892a1e40d0072539a1bbd1f0adfa80655f7affdbcf45410e302f480cb33793"} Oct 30 17:15:02.766777 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:02.766689 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d5b798-29889675-hhqqx" event={"ID":"632128dd-dd61-4807-8e04-6c809d0e4711","Type":"ContainerStarted","Data":"a8930bd07c2f8af414ac1c4ab3f69edf444fe172c7b61b5c247f4ee051e5e24c"} Oct 30 17:15:02.769716 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:02.769687 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-13c495-29889675-79wjh" event={"ID":"b72976e0-c501-44b4-b42c-5dfa26192e87","Type":"ContainerStarted","Data":"151c09cc8938f2358b6fb660540937f0372b1012ce572b891507f4db0b148093"} Oct 30 17:15:02.771385 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:02.771361 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6acc78-29889675-664bj" event={"ID":"c4086c1e-112f-4518-8ae2-3fa5526f9357","Type":"ContainerStarted","Data":"49220f40f51b730842615758b0fa20f2e9cd17f4672239c2ef3549f3c92a9ac5"} Oct 30 17:15:02.772683 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:02.772662 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-43197b-29889675-mm8kj" event={"ID":"5ff2bc89-4ecd-41d6-86ca-4111cba66bbd","Type":"ContainerStarted","Data":"af1c9a919fe9b6872ec410a19db9b2897f40d92b98ca854402a878b02d67944b"} Oct 30 17:15:02.774259 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:02.774198 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5758b7-29889675-hshgf" event={"ID":"68c22006-b1b5-4f15-b504-be63d753819c","Type":"ContainerStarted","Data":"5d90450533837efa9afd050f682dbcc0e679b7288d1f7daaa16b0c45078e5c24"} Oct 30 17:15:02.775404 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:02.775383 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5c6b3c-29889675-zznjt" event={"ID":"89814242-301f-48ab-a666-b8574e1e8116","Type":"ContainerStarted","Data":"f8780fd09918d0ec00f49628fc24d972f1587be7eef0141326344387f431a8d2"} Oct 30 17:15:02.776538 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:02.776518 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24acb2-29889675-wttj8" event={"ID":"49589f83-94f8-4e46-808f-cb013c6adfeb","Type":"ContainerStarted","Data":"9e8519c338871447d8b1578d2570b403fcd9b46157cb52e30bc191d8ac7cbee6"} Oct 30 17:15:02.777693 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:02.777670 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889675-d8tcq" event={"ID":"43be3031-1e16-4687-97f9-f7af506d98b5","Type":"ContainerStarted","Data":"8943b60b629e463b0685c7666bde22517bdf7d22f79cc483e1d2ca3cccacbed2"} Oct 30 17:15:02.778951 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:02.778776 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dd715e-29889675-sjdts" event={"ID":"ac7c8bb8-9c6e-4b0e-bb84-2e70fc863d96","Type":"ContainerStarted","Data":"155b7e3fc2f6d2da48c61bd027aaa1759647f783605103ecc483c2b57d9e6518"} Oct 30 17:15:02.798165 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:02.798111 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2589b5-29889675-h7lhw" podStartSLOduration=2.798096188 podStartE2EDuration="2.798096188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:02.796976096 +0000 UTC m=+3130.024151197" watchObservedRunningTime="2026-10-30 17:15:02.798096188 +0000 UTC m=+3130.025271288" Oct 30 17:15:02.841326 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:02.841268 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-13c495-29889675-79wjh" podStartSLOduration=2.841253174 podStartE2EDuration="2.841253174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:02.836595702 +0000 UTC m=+3130.063770803" watchObservedRunningTime="2026-10-30 17:15:02.841253174 +0000 UTC m=+3130.068428273" Oct 30 17:15:02.881916 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:02.881854 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6acc78-29889675-664bj" podStartSLOduration=2.881838226 podStartE2EDuration="2.881838226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:02.877203192 +0000 UTC m=+3130.104378291" watchObservedRunningTime="2026-10-30 17:15:02.881838226 +0000 UTC m=+3130.109013326" Oct 30 17:15:02.918777 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:02.918719 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5758b7-29889675-hshgf" podStartSLOduration=2.918704538 podStartE2EDuration="2.918704538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:02.916299701 +0000 UTC m=+3130.143474804" watchObservedRunningTime="2026-10-30 17:15:02.918704538 +0000 UTC m=+3130.145879638" Oct 30 17:15:02.960098 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:02.960035 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cf4621-29889675-vl2xw" podStartSLOduration=2.960018874 podStartE2EDuration="2.960018874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:02.956577406 +0000 UTC m=+3130.183752529" watchObservedRunningTime="2026-10-30 17:15:02.960018874 +0000 UTC m=+3130.187193993" Oct 30 17:15:03.791076 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:03.790176 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dd715e-29889675-sjdts" event={"ID":"ac7c8bb8-9c6e-4b0e-bb84-2e70fc863d96","Type":"ContainerStarted","Data":"6ccb6efc2e22d38876475a47f9d341bdbf0ceeb298464627bc3492a278425340"} Oct 30 17:15:03.792183 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:03.792151 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889675-kxmbh" event={"ID":"9988e846-2f00-4c0a-b72e-b2943e79f01b","Type":"ContainerStarted","Data":"691b82e2c38c207e8fac733e08eaee4bff58961f187a0eba8c1e70feada3a7e0"} Oct 30 17:15:03.793722 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:03.793699 4493 generic.go:356] "Generic (PLEG): container finished" podID="e0665578-a6c7-4b36-ba0f-5144760dce17" containerID="62281486a085c3e18cccc01c2767e5066d5312a8bc1e3ea794e2331c7ff2cd5c" exitCode=60 Oct 30 17:15:03.793884 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:03.793795 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f94fb5-29889675-6w42p" event={"ID":"e0665578-a6c7-4b36-ba0f-5144760dce17","Type":"ContainerDied","Data":"62281486a085c3e18cccc01c2767e5066d5312a8bc1e3ea794e2331c7ff2cd5c"} Oct 30 17:15:03.794057 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:03.794043 4493 scope.go:122] "RemoveContainer" containerID="62281486a085c3e18cccc01c2767e5066d5312a8bc1e3ea794e2331c7ff2cd5c" Oct 30 17:15:03.795628 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:03.795600 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-06e7f6-29889675-m9wc6" event={"ID":"7d2cd04d-1a67-45a1-8934-f9d4c9435523","Type":"ContainerStarted","Data":"3e48671cc4bc635875b9ed1a979af3856883b3097591540901a9806ced06a138"} Oct 30 17:15:03.797955 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:03.797921 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3106d6-29889675-dtkjj" event={"ID":"5c296662-aa00-4ee9-90e5-2829d66ce1be","Type":"ContainerStarted","Data":"04dfd24f4a1fe68bd17769ff7fe6c1006ed8cfe00911c2d7f801d93368e447cb"} Oct 30 17:15:03.799671 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:03.799652 4493 generic.go:356] "Generic (PLEG): container finished" podID="045dd183-6322-48cb-85df-fa7cc872f8cd" containerID="9808e2a9895e7e7b3a958f63e24caa14f4b1d15f496362603e46cbbbacc8a7e8" exitCode=60 Oct 30 17:15:03.799766 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:03.799724 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2589b5-29889675-h7lhw" event={"ID":"045dd183-6322-48cb-85df-fa7cc872f8cd","Type":"ContainerDied","Data":"9808e2a9895e7e7b3a958f63e24caa14f4b1d15f496362603e46cbbbacc8a7e8"} Oct 30 17:15:03.800120 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:03.800103 4493 scope.go:122] "RemoveContainer" containerID="9808e2a9895e7e7b3a958f63e24caa14f4b1d15f496362603e46cbbbacc8a7e8" Oct 30 17:15:03.817285 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:03.816204 4493 generic.go:356] "Generic (PLEG): container finished" podID="53dc2f29-cd5a-4a6b-8e97-cc2d60925ade" containerID="96701ff6c65a04092dad8689a51e31a830d3d2853a5fe22b7a6fb6e66cb8b982" exitCode=60 Oct 30 17:15:03.817285 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:03.816288 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cb6251-29889675-tpztt" event={"ID":"53dc2f29-cd5a-4a6b-8e97-cc2d60925ade","Type":"ContainerDied","Data":"96701ff6c65a04092dad8689a51e31a830d3d2853a5fe22b7a6fb6e66cb8b982"} Oct 30 17:15:03.817285 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:03.816711 4493 scope.go:122] "RemoveContainer" containerID="96701ff6c65a04092dad8689a51e31a830d3d2853a5fe22b7a6fb6e66cb8b982" Oct 30 17:15:03.819686 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:03.819657 4493 generic.go:356] "Generic (PLEG): container finished" podID="e13c30d1-8241-414d-b8fb-a31f796a805a" containerID="6338d2fe6325c17a86627fc2643c1605baffacf365e0977fa75f6d1bc2eba66f" exitCode=60 Oct 30 17:15:03.819936 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:03.819741 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cf4621-29889675-vl2xw" event={"ID":"e13c30d1-8241-414d-b8fb-a31f796a805a","Type":"ContainerDied","Data":"6338d2fe6325c17a86627fc2643c1605baffacf365e0977fa75f6d1bc2eba66f"} Oct 30 17:15:03.820117 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:03.820099 4493 scope.go:122] "RemoveContainer" containerID="6338d2fe6325c17a86627fc2643c1605baffacf365e0977fa75f6d1bc2eba66f" Oct 30 17:15:03.822419 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:03.822395 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d5b798-29889675-hhqqx" event={"ID":"632128dd-dd61-4807-8e04-6c809d0e4711","Type":"ContainerStarted","Data":"643a4df46cd714b96a8d7cf8c227aa56603df52b921eb839ad5bc66e8c7c682f"} Oct 30 17:15:03.824595 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:03.824575 4493 generic.go:356] "Generic (PLEG): container finished" podID="b72976e0-c501-44b4-b42c-5dfa26192e87" containerID="151c09cc8938f2358b6fb660540937f0372b1012ce572b891507f4db0b148093" exitCode=60 Oct 30 17:15:03.824711 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:03.824676 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-13c495-29889675-79wjh" event={"ID":"b72976e0-c501-44b4-b42c-5dfa26192e87","Type":"ContainerDied","Data":"151c09cc8938f2358b6fb660540937f0372b1012ce572b891507f4db0b148093"} Oct 30 17:15:03.825582 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:03.825563 4493 scope.go:122] "RemoveContainer" containerID="151c09cc8938f2358b6fb660540937f0372b1012ce572b891507f4db0b148093" Oct 30 17:15:03.827013 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:03.826991 4493 generic.go:356] "Generic (PLEG): container finished" podID="c4086c1e-112f-4518-8ae2-3fa5526f9357" containerID="e4eab96730e6cfa8e66632eda11bdde18db1ff347d189bebf9da1d1419f79814" exitCode=60 Oct 30 17:15:03.827213 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:03.827194 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6acc78-29889675-664bj" event={"ID":"c4086c1e-112f-4518-8ae2-3fa5526f9357","Type":"ContainerDied","Data":"e4eab96730e6cfa8e66632eda11bdde18db1ff347d189bebf9da1d1419f79814"} Oct 30 17:15:03.827350 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:03.827293 4493 scope.go:122] "RemoveContainer" containerID="e4eab96730e6cfa8e66632eda11bdde18db1ff347d189bebf9da1d1419f79814" Oct 30 17:15:03.833187 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:03.833156 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-43197b-29889675-mm8kj" event={"ID":"5ff2bc89-4ecd-41d6-86ca-4111cba66bbd","Type":"ContainerStarted","Data":"7c1d5d05d2d9ee5d08a972ebd66919f59c33b7ea66da81df13944adc2072e3a6"} Oct 30 17:15:03.840238 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:03.840212 4493 generic.go:356] "Generic (PLEG): container finished" podID="68c22006-b1b5-4f15-b504-be63d753819c" containerID="5d90450533837efa9afd050f682dbcc0e679b7288d1f7daaa16b0c45078e5c24" exitCode=60 Oct 30 17:15:03.841197 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:03.841172 4493 scope.go:122] "RemoveContainer" containerID="5d90450533837efa9afd050f682dbcc0e679b7288d1f7daaa16b0c45078e5c24" Oct 30 17:15:03.841293 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:03.841219 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5758b7-29889675-hshgf" event={"ID":"68c22006-b1b5-4f15-b504-be63d753819c","Type":"ContainerDied","Data":"5d90450533837efa9afd050f682dbcc0e679b7288d1f7daaa16b0c45078e5c24"} Oct 30 17:15:03.846621 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:03.846591 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24acb2-29889675-wttj8" event={"ID":"49589f83-94f8-4e46-808f-cb013c6adfeb","Type":"ContainerStarted","Data":"166af153c2aa9a19e8e76963545d5c9d2c3d4deca87b406ed88b918221f88132"} Oct 30 17:15:03.850177 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:03.850153 4493 generic.go:356] "Generic (PLEG): container finished" podID="43be3031-1e16-4687-97f9-f7af506d98b5" containerID="bb01d16b169a5bf8346056bf7743e2241db9e0832e3d9175f1154a37b3ac7aed" exitCode=60 Oct 30 17:15:03.850266 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:03.850203 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889675-d8tcq" event={"ID":"43be3031-1e16-4687-97f9-f7af506d98b5","Type":"ContainerDied","Data":"bb01d16b169a5bf8346056bf7743e2241db9e0832e3d9175f1154a37b3ac7aed"} Oct 30 17:15:03.850639 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:03.850518 4493 scope.go:122] "RemoveContainer" containerID="bb01d16b169a5bf8346056bf7743e2241db9e0832e3d9175f1154a37b3ac7aed" Oct 30 17:15:03.857109 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:03.857067 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dd715e-29889675-sjdts" podStartSLOduration=3.857054025 podStartE2EDuration="3.857054025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:03.843846643 +0000 UTC m=+3131.071021764" watchObservedRunningTime="2026-10-30 17:15:03.857054025 +0000 UTC m=+3131.084229125" Oct 30 17:15:03.985964 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:03.985910 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-06e7f6-29889675-m9wc6" podStartSLOduration=3.98588931 podStartE2EDuration="3.98588931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:03.984879945 +0000 UTC m=+3131.212055043" watchObservedRunningTime="2026-10-30 17:15:03.98588931 +0000 UTC m=+3131.213064409" Oct 30 17:15:04.198483 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:04.198433 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-43197b-29889675-mm8kj" podStartSLOduration=4.198419803 podStartE2EDuration="4.198419803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:04.195175243 +0000 UTC m=+3131.422350343" watchObservedRunningTime="2026-10-30 17:15:04.198419803 +0000 UTC m=+3131.425594941" Oct 30 17:15:04.241102 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:04.241051 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d5b798-29889675-hhqqx" podStartSLOduration=4.241034749 podStartE2EDuration="4.241034749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:04.233639219 +0000 UTC m=+3131.460814320" watchObservedRunningTime="2026-10-30 17:15:04.241034749 +0000 UTC m=+3131.468209928" Oct 30 17:15:04.280567 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:04.280510 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889675-kxmbh" podStartSLOduration=4.280493082 podStartE2EDuration="4.280493082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:04.276336997 +0000 UTC m=+3131.503512095" watchObservedRunningTime="2026-10-30 17:15:04.280493082 +0000 UTC m=+3131.507668183" Oct 30 17:15:04.399834 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:04.399747 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3106d6-29889675-dtkjj" podStartSLOduration=4.399732315 podStartE2EDuration="4.399732315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:04.398401268 +0000 UTC m=+3131.625576380" watchObservedRunningTime="2026-10-30 17:15:04.399732315 +0000 UTC m=+3131.626907407" Oct 30 17:15:04.863136 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:04.863052 4493 generic.go:356] "Generic (PLEG): container finished" podID="9988e846-2f00-4c0a-b72e-b2943e79f01b" containerID="691b82e2c38c207e8fac733e08eaee4bff58961f187a0eba8c1e70feada3a7e0" exitCode=60 Oct 30 17:15:04.863136 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:04.863114 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889675-kxmbh" event={"ID":"9988e846-2f00-4c0a-b72e-b2943e79f01b","Type":"ContainerDied","Data":"691b82e2c38c207e8fac733e08eaee4bff58961f187a0eba8c1e70feada3a7e0"} Oct 30 17:15:04.863593 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:04.863404 4493 scope.go:122] "RemoveContainer" containerID="691b82e2c38c207e8fac733e08eaee4bff58961f187a0eba8c1e70feada3a7e0" Oct 30 17:15:04.865559 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:04.865536 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f94fb5-29889675-6w42p" event={"ID":"e0665578-a6c7-4b36-ba0f-5144760dce17","Type":"ContainerStarted","Data":"72c4a9aaf65528e7c93df69ae468e777be2575663c5145a59244ed0dec5138d4"} Oct 30 17:15:04.867425 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:04.867400 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8bff6e-29889675-q2zhc" event={"ID":"40a120b1-71f5-40ab-8faf-da2052ab0a56","Type":"ContainerStarted","Data":"4296a0bc73250f249eeeec21710d44c738013ab4cca1c62704824cb1ca200147"} Oct 30 17:15:04.869482 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:04.869462 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-abda73-29889675-8gqs6" event={"ID":"25dd3114-a3ba-4976-8f11-611923dc090f","Type":"ContainerStarted","Data":"0a88a17207a4f19c61bd883bc570847234c984d2b1f67a63fb331c7f230e8345"} Oct 30 17:15:04.871202 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:04.871178 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889675-nmmn8" event={"ID":"57140245-e2df-4ee9-9344-1de9618511d0","Type":"ContainerStarted","Data":"f03c209a9d5c5d5a3e1050cd82e1f837004c96405215457204da5460bb4d3268"} Oct 30 17:15:04.872745 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:04.872724 4493 generic.go:356] "Generic (PLEG): container finished" podID="7d2cd04d-1a67-45a1-8934-f9d4c9435523" containerID="3e48671cc4bc635875b9ed1a979af3856883b3097591540901a9806ced06a138" exitCode=60 Oct 30 17:15:04.872881 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:04.872801 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-06e7f6-29889675-m9wc6" event={"ID":"7d2cd04d-1a67-45a1-8934-f9d4c9435523","Type":"ContainerDied","Data":"3e48671cc4bc635875b9ed1a979af3856883b3097591540901a9806ced06a138"} Oct 30 17:15:04.873020 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:04.873007 4493 scope.go:122] "RemoveContainer" containerID="3e48671cc4bc635875b9ed1a979af3856883b3097591540901a9806ced06a138" Oct 30 17:15:04.874424 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:04.874403 4493 generic.go:356] "Generic (PLEG): container finished" podID="5c296662-aa00-4ee9-90e5-2829d66ce1be" containerID="04dfd24f4a1fe68bd17769ff7fe6c1006ed8cfe00911c2d7f801d93368e447cb" exitCode=60 Oct 30 17:15:04.874524 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:04.874451 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3106d6-29889675-dtkjj" event={"ID":"5c296662-aa00-4ee9-90e5-2829d66ce1be","Type":"ContainerDied","Data":"04dfd24f4a1fe68bd17769ff7fe6c1006ed8cfe00911c2d7f801d93368e447cb"} Oct 30 17:15:04.874641 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:04.874627 4493 scope.go:122] "RemoveContainer" containerID="04dfd24f4a1fe68bd17769ff7fe6c1006ed8cfe00911c2d7f801d93368e447cb" Oct 30 17:15:04.876562 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:04.876540 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2589b5-29889675-h7lhw" event={"ID":"045dd183-6322-48cb-85df-fa7cc872f8cd","Type":"ContainerStarted","Data":"a448f7e577825193408cedb18957c7733beb76591f5dbf56bc4927ac74242f7a"} Oct 30 17:15:04.879534 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:04.879513 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cb6251-29889675-tpztt" event={"ID":"53dc2f29-cd5a-4a6b-8e97-cc2d60925ade","Type":"ContainerStarted","Data":"24183ec2e18425ac4921177ac596eb5bee3005efcb5f1bd771c266685b9f9b53"} Oct 30 17:15:04.881832 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:04.881810 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cf4621-29889675-vl2xw" event={"ID":"e13c30d1-8241-414d-b8fb-a31f796a805a","Type":"ContainerStarted","Data":"42f282d9795a1f873ac8216a10f7d83d721088aec29b0d1034d5866298f8bf06"} Oct 30 17:15:04.884438 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:04.884418 4493 generic.go:356] "Generic (PLEG): container finished" podID="632128dd-dd61-4807-8e04-6c809d0e4711" containerID="643a4df46cd714b96a8d7cf8c227aa56603df52b921eb839ad5bc66e8c7c682f" exitCode=60 Oct 30 17:15:04.884527 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:04.884489 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d5b798-29889675-hhqqx" event={"ID":"632128dd-dd61-4807-8e04-6c809d0e4711","Type":"ContainerDied","Data":"643a4df46cd714b96a8d7cf8c227aa56603df52b921eb839ad5bc66e8c7c682f"} Oct 30 17:15:04.884678 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:04.884661 4493 scope.go:122] "RemoveContainer" containerID="643a4df46cd714b96a8d7cf8c227aa56603df52b921eb839ad5bc66e8c7c682f" Oct 30 17:15:04.887996 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:04.887470 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-13c495-29889675-79wjh" event={"ID":"b72976e0-c501-44b4-b42c-5dfa26192e87","Type":"ContainerStarted","Data":"b8ba8f34ce3c08e71f356f1929d6903b3d7fffe515309b6726e6f952641b5eb2"} Oct 30 17:15:04.890759 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:04.890713 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6acc78-29889675-664bj" event={"ID":"c4086c1e-112f-4518-8ae2-3fa5526f9357","Type":"ContainerStarted","Data":"82e4eaa41c79586864ec3d9b537aebdd262a9a6686ae8e56bf13382bbeeea4db"} Oct 30 17:15:04.896966 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:04.896912 4493 generic.go:356] "Generic (PLEG): container finished" podID="5ff2bc89-4ecd-41d6-86ca-4111cba66bbd" containerID="7c1d5d05d2d9ee5d08a972ebd66919f59c33b7ea66da81df13944adc2072e3a6" exitCode=60 Oct 30 17:15:04.897171 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:04.897154 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-43197b-29889675-mm8kj" event={"ID":"5ff2bc89-4ecd-41d6-86ca-4111cba66bbd","Type":"ContainerDied","Data":"7c1d5d05d2d9ee5d08a972ebd66919f59c33b7ea66da81df13944adc2072e3a6"} Oct 30 17:15:04.897299 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:04.897173 4493 scope.go:122] "RemoveContainer" containerID="7c1d5d05d2d9ee5d08a972ebd66919f59c33b7ea66da81df13944adc2072e3a6" Oct 30 17:15:04.900017 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:04.899997 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5758b7-29889675-hshgf" event={"ID":"68c22006-b1b5-4f15-b504-be63d753819c","Type":"ContainerStarted","Data":"f1c2ea551a6ba264d1929f69785d7ecb2ec903340cec93139511504c7ffee742"} Oct 30 17:15:04.919363 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:04.917825 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5c6b3c-29889675-zznjt" event={"ID":"89814242-301f-48ab-a666-b8574e1e8116","Type":"ContainerStarted","Data":"b8312e84383761fa67c32cfce6a80689f8a2210c344834da2094c88d5260aeaa"} Oct 30 17:15:04.926620 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:04.926568 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24acb2-29889675-wttj8" podStartSLOduration=4.926551484 podStartE2EDuration="4.926551484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:04.503069513 +0000 UTC m=+3131.730244613" watchObservedRunningTime="2026-10-30 17:15:04.926551484 +0000 UTC m=+3132.153726586" Oct 30 17:15:04.929290 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:04.929259 4493 generic.go:356] "Generic (PLEG): container finished" podID="49589f83-94f8-4e46-808f-cb013c6adfeb" containerID="166af153c2aa9a19e8e76963545d5c9d2c3d4deca87b406ed88b918221f88132" exitCode=60 Oct 30 17:15:04.929709 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:04.929689 4493 scope.go:122] "RemoveContainer" containerID="166af153c2aa9a19e8e76963545d5c9d2c3d4deca87b406ed88b918221f88132" Oct 30 17:15:04.937686 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:04.936993 4493 generic.go:356] "Generic (PLEG): container finished" podID="ac7c8bb8-9c6e-4b0e-bb84-2e70fc863d96" containerID="6ccb6efc2e22d38876475a47f9d341bdbf0ceeb298464627bc3492a278425340" exitCode=60 Oct 30 17:15:04.937686 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:04.937323 4493 scope.go:122] "RemoveContainer" containerID="6ccb6efc2e22d38876475a47f9d341bdbf0ceeb298464627bc3492a278425340" Oct 30 17:15:04.965187 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:04.965158 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24acb2-29889675-wttj8" event={"ID":"49589f83-94f8-4e46-808f-cb013c6adfeb","Type":"ContainerDied","Data":"166af153c2aa9a19e8e76963545d5c9d2c3d4deca87b406ed88b918221f88132"} Oct 30 17:15:04.965300 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:04.965208 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889675-d8tcq" event={"ID":"43be3031-1e16-4687-97f9-f7af506d98b5","Type":"ContainerStarted","Data":"0e4047b78a836a2d3ffda6955e8722e26319bd739d53229473308a90a531e673"} Oct 30 17:15:04.965300 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:04.965231 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dd715e-29889675-sjdts" event={"ID":"ac7c8bb8-9c6e-4b0e-bb84-2e70fc863d96","Type":"ContainerDied","Data":"6ccb6efc2e22d38876475a47f9d341bdbf0ceeb298464627bc3492a278425340"} Oct 30 17:15:04.991477 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:04.991431 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cb6251-29889675-tpztt" podStartSLOduration=4.991416383 podStartE2EDuration="4.991416383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:04.985890407 +0000 UTC m=+3132.213065508" watchObservedRunningTime="2026-10-30 17:15:04.991416383 +0000 UTC m=+3132.218591479" Oct 30 17:15:05.145618 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:05.145572 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f94fb5-29889675-6w42p" podStartSLOduration=5.14555905 podStartE2EDuration="5.14555905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:05.144030067 +0000 UTC m=+3132.371205189" watchObservedRunningTime="2026-10-30 17:15:05.14555905 +0000 UTC m=+3132.372734202" Oct 30 17:15:05.223558 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:05.223294 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889675-nmmn8" podStartSLOduration=5.22327484 podStartE2EDuration="5.22327484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:05.222764086 +0000 UTC m=+3132.449939186" watchObservedRunningTime="2026-10-30 17:15:05.22327484 +0000 UTC m=+3132.450449944" Oct 30 17:15:05.263516 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:05.263464 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8bff6e-29889675-q2zhc" podStartSLOduration=5.263450763 podStartE2EDuration="5.263450763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:05.261529526 +0000 UTC m=+3132.488704627" watchObservedRunningTime="2026-10-30 17:15:05.263450763 +0000 UTC m=+3132.490625863" Oct 30 17:15:05.354932 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:05.351876 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-abda73-29889675-8gqs6" podStartSLOduration=5.351857319 podStartE2EDuration="5.351857319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:05.344930251 +0000 UTC m=+3132.572105341" watchObservedRunningTime="2026-10-30 17:15:05.351857319 +0000 UTC m=+3132.579032420" Oct 30 17:15:05.558533 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:05.558478 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889675-d8tcq" podStartSLOduration=5.5584643830000005 podStartE2EDuration="5.558464383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:05.555566498 +0000 UTC m=+3132.782741594" watchObservedRunningTime="2026-10-30 17:15:05.558464383 +0000 UTC m=+3132.785639482" Oct 30 17:15:05.599428 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:05.599383 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5c6b3c-29889675-zznjt" podStartSLOduration=5.599370451 podStartE2EDuration="5.599370451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:05.594282329 +0000 UTC m=+3132.821457429" watchObservedRunningTime="2026-10-30 17:15:05.599370451 +0000 UTC m=+3132.826545551" Oct 30 17:15:05.949933 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:05.949883 4493 generic.go:356] "Generic (PLEG): container finished" podID="e0665578-a6c7-4b36-ba0f-5144760dce17" containerID="72c4a9aaf65528e7c93df69ae468e777be2575663c5145a59244ed0dec5138d4" exitCode=60 Oct 30 17:15:05.950346 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:05.949978 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f94fb5-29889675-6w42p" event={"ID":"e0665578-a6c7-4b36-ba0f-5144760dce17","Type":"ContainerDied","Data":"72c4a9aaf65528e7c93df69ae468e777be2575663c5145a59244ed0dec5138d4"} Oct 30 17:15:05.950346 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:05.950109 4493 scope.go:122] "RemoveContainer" containerID="62281486a085c3e18cccc01c2767e5066d5312a8bc1e3ea794e2331c7ff2cd5c" Oct 30 17:15:05.950472 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:05.950416 4493 scope.go:122] "RemoveContainer" containerID="72c4a9aaf65528e7c93df69ae468e777be2575663c5145a59244ed0dec5138d4" Oct 30 17:15:05.950852 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:15:05.950827 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-f94fb5-29889675-6w42p_odh-ai-gateway-infra(e0665578-a6c7-4b36-ba0f-5144760dce17)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f94fb5-29889675-6w42p" podUID="e0665578-a6c7-4b36-ba0f-5144760dce17" Oct 30 17:15:05.952200 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:05.952175 4493 generic.go:356] "Generic (PLEG): container finished" podID="40a120b1-71f5-40ab-8faf-da2052ab0a56" containerID="4296a0bc73250f249eeeec21710d44c738013ab4cca1c62704824cb1ca200147" exitCode=60 Oct 30 17:15:05.952310 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:05.952237 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8bff6e-29889675-q2zhc" event={"ID":"40a120b1-71f5-40ab-8faf-da2052ab0a56","Type":"ContainerDied","Data":"4296a0bc73250f249eeeec21710d44c738013ab4cca1c62704824cb1ca200147"} Oct 30 17:15:05.952611 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:05.952595 4493 scope.go:122] "RemoveContainer" containerID="4296a0bc73250f249eeeec21710d44c738013ab4cca1c62704824cb1ca200147" Oct 30 17:15:05.954175 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:05.954128 4493 generic.go:356] "Generic (PLEG): container finished" podID="25dd3114-a3ba-4976-8f11-611923dc090f" containerID="0a88a17207a4f19c61bd883bc570847234c984d2b1f67a63fb331c7f230e8345" exitCode=60 Oct 30 17:15:05.954175 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:05.954156 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-abda73-29889675-8gqs6" event={"ID":"25dd3114-a3ba-4976-8f11-611923dc090f","Type":"ContainerDied","Data":"0a88a17207a4f19c61bd883bc570847234c984d2b1f67a63fb331c7f230e8345"} Oct 30 17:15:05.954526 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:05.954507 4493 scope.go:122] "RemoveContainer" containerID="0a88a17207a4f19c61bd883bc570847234c984d2b1f67a63fb331c7f230e8345" Oct 30 17:15:05.956251 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:05.956227 4493 generic.go:356] "Generic (PLEG): container finished" podID="57140245-e2df-4ee9-9344-1de9618511d0" containerID="f03c209a9d5c5d5a3e1050cd82e1f837004c96405215457204da5460bb4d3268" exitCode=60 Oct 30 17:15:05.956363 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:05.956338 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889675-nmmn8" event={"ID":"57140245-e2df-4ee9-9344-1de9618511d0","Type":"ContainerDied","Data":"f03c209a9d5c5d5a3e1050cd82e1f837004c96405215457204da5460bb4d3268"} Oct 30 17:15:05.956567 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:05.956549 4493 scope.go:122] "RemoveContainer" containerID="f03c209a9d5c5d5a3e1050cd82e1f837004c96405215457204da5460bb4d3268" Oct 30 17:15:05.959989 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:05.959938 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3106d6-29889675-dtkjj" event={"ID":"5c296662-aa00-4ee9-90e5-2829d66ce1be","Type":"ContainerStarted","Data":"2c05a57129e4f6cd727a6c9b98f09581a20d11e75b97db527f438f18931fa287"} Oct 30 17:15:05.961949 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:05.961930 4493 generic.go:356] "Generic (PLEG): container finished" podID="045dd183-6322-48cb-85df-fa7cc872f8cd" containerID="a448f7e577825193408cedb18957c7733beb76591f5dbf56bc4927ac74242f7a" exitCode=60 Oct 30 17:15:05.962049 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:05.962002 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2589b5-29889675-h7lhw" event={"ID":"045dd183-6322-48cb-85df-fa7cc872f8cd","Type":"ContainerDied","Data":"a448f7e577825193408cedb18957c7733beb76591f5dbf56bc4927ac74242f7a"} Oct 30 17:15:05.962261 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:05.962245 4493 scope.go:122] "RemoveContainer" containerID="a448f7e577825193408cedb18957c7733beb76591f5dbf56bc4927ac74242f7a" Oct 30 17:15:05.962480 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:15:05.962456 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-2589b5-29889675-h7lhw_odh-ai-gateway-infra(045dd183-6322-48cb-85df-fa7cc872f8cd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2589b5-29889675-h7lhw" podUID="045dd183-6322-48cb-85df-fa7cc872f8cd" Oct 30 17:15:05.964097 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:05.964078 4493 generic.go:356] "Generic (PLEG): container finished" podID="53dc2f29-cd5a-4a6b-8e97-cc2d60925ade" containerID="24183ec2e18425ac4921177ac596eb5bee3005efcb5f1bd771c266685b9f9b53" exitCode=60 Oct 30 17:15:05.964187 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:05.964109 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cb6251-29889675-tpztt" event={"ID":"53dc2f29-cd5a-4a6b-8e97-cc2d60925ade","Type":"ContainerDied","Data":"24183ec2e18425ac4921177ac596eb5bee3005efcb5f1bd771c266685b9f9b53"} Oct 30 17:15:05.964408 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:05.964389 4493 scope.go:122] "RemoveContainer" containerID="24183ec2e18425ac4921177ac596eb5bee3005efcb5f1bd771c266685b9f9b53" Oct 30 17:15:05.964624 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:15:05.964607 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-cb6251-29889675-tpztt_odh-ai-gateway-infra(53dc2f29-cd5a-4a6b-8e97-cc2d60925ade)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cb6251-29889675-tpztt" podUID="53dc2f29-cd5a-4a6b-8e97-cc2d60925ade" Oct 30 17:15:05.968220 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:05.968186 4493 generic.go:356] "Generic (PLEG): container finished" podID="e13c30d1-8241-414d-b8fb-a31f796a805a" containerID="42f282d9795a1f873ac8216a10f7d83d721088aec29b0d1034d5866298f8bf06" exitCode=60 Oct 30 17:15:05.968324 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:05.968290 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cf4621-29889675-vl2xw" event={"ID":"e13c30d1-8241-414d-b8fb-a31f796a805a","Type":"ContainerDied","Data":"42f282d9795a1f873ac8216a10f7d83d721088aec29b0d1034d5866298f8bf06"} Oct 30 17:15:05.968491 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:05.968475 4493 scope.go:122] "RemoveContainer" containerID="42f282d9795a1f873ac8216a10f7d83d721088aec29b0d1034d5866298f8bf06" Oct 30 17:15:05.968714 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:15:05.968654 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-cf4621-29889675-vl2xw_odh-ai-gateway-infra(e13c30d1-8241-414d-b8fb-a31f796a805a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cf4621-29889675-vl2xw" podUID="e13c30d1-8241-414d-b8fb-a31f796a805a" Oct 30 17:15:05.970145 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:05.970128 4493 generic.go:356] "Generic (PLEG): container finished" podID="b72976e0-c501-44b4-b42c-5dfa26192e87" containerID="b8ba8f34ce3c08e71f356f1929d6903b3d7fffe515309b6726e6f952641b5eb2" exitCode=60 Oct 30 17:15:05.970242 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:05.970205 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-13c495-29889675-79wjh" event={"ID":"b72976e0-c501-44b4-b42c-5dfa26192e87","Type":"ContainerDied","Data":"b8ba8f34ce3c08e71f356f1929d6903b3d7fffe515309b6726e6f952641b5eb2"} Oct 30 17:15:05.970408 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:05.970392 4493 scope.go:122] "RemoveContainer" containerID="b8ba8f34ce3c08e71f356f1929d6903b3d7fffe515309b6726e6f952641b5eb2" Oct 30 17:15:05.970627 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:15:05.970610 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-13c495-29889675-79wjh_odh-ai-gateway-infra(b72976e0-c501-44b4-b42c-5dfa26192e87)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-13c495-29889675-79wjh" podUID="b72976e0-c501-44b4-b42c-5dfa26192e87" Oct 30 17:15:05.972050 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:05.972003 4493 scope.go:122] "RemoveContainer" containerID="9808e2a9895e7e7b3a958f63e24caa14f4b1d15f496362603e46cbbbacc8a7e8" Oct 30 17:15:05.972555 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:05.972533 4493 generic.go:356] "Generic (PLEG): container finished" podID="c4086c1e-112f-4518-8ae2-3fa5526f9357" containerID="82e4eaa41c79586864ec3d9b537aebdd262a9a6686ae8e56bf13382bbeeea4db" exitCode=60 Oct 30 17:15:05.972655 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:05.972614 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6acc78-29889675-664bj" event={"ID":"c4086c1e-112f-4518-8ae2-3fa5526f9357","Type":"ContainerDied","Data":"82e4eaa41c79586864ec3d9b537aebdd262a9a6686ae8e56bf13382bbeeea4db"} Oct 30 17:15:05.972931 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:05.972912 4493 scope.go:122] "RemoveContainer" containerID="82e4eaa41c79586864ec3d9b537aebdd262a9a6686ae8e56bf13382bbeeea4db" Oct 30 17:15:05.973215 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:15:05.973196 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-6acc78-29889675-664bj_odh-ai-gateway-infra(c4086c1e-112f-4518-8ae2-3fa5526f9357)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6acc78-29889675-664bj" podUID="c4086c1e-112f-4518-8ae2-3fa5526f9357" Oct 30 17:15:05.975220 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:05.975191 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-43197b-29889675-mm8kj" event={"ID":"5ff2bc89-4ecd-41d6-86ca-4111cba66bbd","Type":"ContainerStarted","Data":"117d3121d638e1e0b2ff3421d9f464c76ed248ce02c92088a6a7dd3e8655ad7b"} Oct 30 17:15:05.977214 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:05.977195 4493 generic.go:356] "Generic (PLEG): container finished" podID="68c22006-b1b5-4f15-b504-be63d753819c" containerID="f1c2ea551a6ba264d1929f69785d7ecb2ec903340cec93139511504c7ffee742" exitCode=60 Oct 30 17:15:05.977313 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:05.977265 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5758b7-29889675-hshgf" event={"ID":"68c22006-b1b5-4f15-b504-be63d753819c","Type":"ContainerDied","Data":"f1c2ea551a6ba264d1929f69785d7ecb2ec903340cec93139511504c7ffee742"} Oct 30 17:15:05.977454 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:05.977439 4493 scope.go:122] "RemoveContainer" containerID="f1c2ea551a6ba264d1929f69785d7ecb2ec903340cec93139511504c7ffee742" Oct 30 17:15:05.977668 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:15:05.977644 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-5758b7-29889675-hshgf_odh-ai-gateway-infra(68c22006-b1b5-4f15-b504-be63d753819c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5758b7-29889675-hshgf" podUID="68c22006-b1b5-4f15-b504-be63d753819c" Oct 30 17:15:05.978859 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:05.978836 4493 generic.go:356] "Generic (PLEG): container finished" podID="89814242-301f-48ab-a666-b8574e1e8116" containerID="b8312e84383761fa67c32cfce6a80689f8a2210c344834da2094c88d5260aeaa" exitCode=60 Oct 30 17:15:05.978961 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:05.978923 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5c6b3c-29889675-zznjt" event={"ID":"89814242-301f-48ab-a666-b8574e1e8116","Type":"ContainerDied","Data":"b8312e84383761fa67c32cfce6a80689f8a2210c344834da2094c88d5260aeaa"} Oct 30 17:15:05.979137 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:05.979115 4493 scope.go:122] "RemoveContainer" containerID="b8312e84383761fa67c32cfce6a80689f8a2210c344834da2094c88d5260aeaa" Oct 30 17:15:05.981128 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:05.981106 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24acb2-29889675-wttj8" event={"ID":"49589f83-94f8-4e46-808f-cb013c6adfeb","Type":"ContainerStarted","Data":"dcb3effadf2d9c1a067684e464f6ea7fd6273d0e9a81c74db8638f63fcab2d9e"} Oct 30 17:15:05.983071 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:05.983047 4493 generic.go:356] "Generic (PLEG): container finished" podID="43be3031-1e16-4687-97f9-f7af506d98b5" containerID="0e4047b78a836a2d3ffda6955e8722e26319bd739d53229473308a90a531e673" exitCode=60 Oct 30 17:15:05.983174 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:05.983094 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889675-d8tcq" event={"ID":"43be3031-1e16-4687-97f9-f7af506d98b5","Type":"ContainerDied","Data":"0e4047b78a836a2d3ffda6955e8722e26319bd739d53229473308a90a531e673"} Oct 30 17:15:05.983340 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:05.983323 4493 scope.go:122] "RemoveContainer" containerID="0e4047b78a836a2d3ffda6955e8722e26319bd739d53229473308a90a531e673" Oct 30 17:15:05.983539 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:15:05.983523 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f3503b-29889675-d8tcq_odh-ai-gateway-infra(43be3031-1e16-4687-97f9-f7af506d98b5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889675-d8tcq" podUID="43be3031-1e16-4687-97f9-f7af506d98b5" Oct 30 17:15:05.994800 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:05.994759 4493 scope.go:122] "RemoveContainer" containerID="96701ff6c65a04092dad8689a51e31a830d3d2853a5fe22b7a6fb6e66cb8b982" Oct 30 17:15:06.022565 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:06.022540 4493 scope.go:122] "RemoveContainer" containerID="6338d2fe6325c17a86627fc2643c1605baffacf365e0977fa75f6d1bc2eba66f" Oct 30 17:15:06.092926 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:06.092900 4493 scope.go:122] "RemoveContainer" containerID="151c09cc8938f2358b6fb660540937f0372b1012ce572b891507f4db0b148093" Oct 30 17:15:06.123614 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:06.123555 4493 scope.go:122] "RemoveContainer" containerID="e4eab96730e6cfa8e66632eda11bdde18db1ff347d189bebf9da1d1419f79814" Oct 30 17:15:06.140167 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:06.140147 4493 scope.go:122] "RemoveContainer" containerID="5d90450533837efa9afd050f682dbcc0e679b7288d1f7daaa16b0c45078e5c24" Oct 30 17:15:06.155627 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:06.155598 4493 scope.go:122] "RemoveContainer" containerID="bb01d16b169a5bf8346056bf7743e2241db9e0832e3d9175f1154a37b3ac7aed" Oct 30 17:15:07.008411 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:07.008381 4493 scope.go:122] "RemoveContainer" containerID="b8ba8f34ce3c08e71f356f1929d6903b3d7fffe515309b6726e6f952641b5eb2" Oct 30 17:15:07.010435 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:15:07.010404 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-13c495-29889675-79wjh_odh-ai-gateway-infra(b72976e0-c501-44b4-b42c-5dfa26192e87)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-13c495-29889675-79wjh" podUID="b72976e0-c501-44b4-b42c-5dfa26192e87" Oct 30 17:15:07.011325 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:07.011303 4493 scope.go:122] "RemoveContainer" containerID="82e4eaa41c79586864ec3d9b537aebdd262a9a6686ae8e56bf13382bbeeea4db" Oct 30 17:15:07.011667 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:15:07.011636 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-6acc78-29889675-664bj_odh-ai-gateway-infra(c4086c1e-112f-4518-8ae2-3fa5526f9357)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6acc78-29889675-664bj" podUID="c4086c1e-112f-4518-8ae2-3fa5526f9357" Oct 30 17:15:07.013036 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:07.013011 4493 generic.go:356] "Generic (PLEG): container finished" podID="5ff2bc89-4ecd-41d6-86ca-4111cba66bbd" containerID="117d3121d638e1e0b2ff3421d9f464c76ed248ce02c92088a6a7dd3e8655ad7b" exitCode=60 Oct 30 17:15:07.013240 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:07.013105 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-43197b-29889675-mm8kj" event={"ID":"5ff2bc89-4ecd-41d6-86ca-4111cba66bbd","Type":"ContainerDied","Data":"117d3121d638e1e0b2ff3421d9f464c76ed248ce02c92088a6a7dd3e8655ad7b"} Oct 30 17:15:07.013332 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:07.013319 4493 scope.go:122] "RemoveContainer" containerID="7c1d5d05d2d9ee5d08a972ebd66919f59c33b7ea66da81df13944adc2072e3a6" Oct 30 17:15:07.013532 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:07.013513 4493 scope.go:122] "RemoveContainer" containerID="117d3121d638e1e0b2ff3421d9f464c76ed248ce02c92088a6a7dd3e8655ad7b" Oct 30 17:15:07.013778 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:15:07.013760 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-43197b-29889675-mm8kj_odh-ai-gateway-infra(5ff2bc89-4ecd-41d6-86ca-4111cba66bbd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-43197b-29889675-mm8kj" podUID="5ff2bc89-4ecd-41d6-86ca-4111cba66bbd" Oct 30 17:15:07.016707 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:07.016687 4493 scope.go:122] "RemoveContainer" containerID="f1c2ea551a6ba264d1929f69785d7ecb2ec903340cec93139511504c7ffee742" Oct 30 17:15:07.017054 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:15:07.017027 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-5758b7-29889675-hshgf_odh-ai-gateway-infra(68c22006-b1b5-4f15-b504-be63d753819c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5758b7-29889675-hshgf" podUID="68c22006-b1b5-4f15-b504-be63d753819c" Oct 30 17:15:07.019536 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:07.019505 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5c6b3c-29889675-zznjt" event={"ID":"89814242-301f-48ab-a666-b8574e1e8116","Type":"ContainerStarted","Data":"1b5bb947a2471d3b91736074bc9437670af39e771580a8598e7375925ffc9daa"} Oct 30 17:15:07.022122 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:07.022099 4493 generic.go:356] "Generic (PLEG): container finished" podID="49589f83-94f8-4e46-808f-cb013c6adfeb" containerID="dcb3effadf2d9c1a067684e464f6ea7fd6273d0e9a81c74db8638f63fcab2d9e" exitCode=60 Oct 30 17:15:07.022225 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:07.022186 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24acb2-29889675-wttj8" event={"ID":"49589f83-94f8-4e46-808f-cb013c6adfeb","Type":"ContainerDied","Data":"dcb3effadf2d9c1a067684e464f6ea7fd6273d0e9a81c74db8638f63fcab2d9e"} Oct 30 17:15:07.022377 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:07.022361 4493 scope.go:122] "RemoveContainer" containerID="dcb3effadf2d9c1a067684e464f6ea7fd6273d0e9a81c74db8638f63fcab2d9e" Oct 30 17:15:07.022653 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:15:07.022629 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-24acb2-29889675-wttj8_odh-ai-gateway-infra(49589f83-94f8-4e46-808f-cb013c6adfeb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24acb2-29889675-wttj8" podUID="49589f83-94f8-4e46-808f-cb013c6adfeb" Oct 30 17:15:07.027097 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:07.027074 4493 scope.go:122] "RemoveContainer" containerID="0e4047b78a836a2d3ffda6955e8722e26319bd739d53229473308a90a531e673" Oct 30 17:15:07.027371 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:15:07.027346 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f3503b-29889675-d8tcq_odh-ai-gateway-infra(43be3031-1e16-4687-97f9-f7af506d98b5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889675-d8tcq" podUID="43be3031-1e16-4687-97f9-f7af506d98b5" Oct 30 17:15:07.029577 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:07.029558 4493 generic.go:356] "Generic (PLEG): container finished" podID="ac7c8bb8-9c6e-4b0e-bb84-2e70fc863d96" containerID="8135440523f5b61ef4609b0b90b12dd551c89e4796c26df265bacad440db2d0f" exitCode=60 Oct 30 17:15:07.029669 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:07.029632 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dd715e-29889675-sjdts" event={"ID":"ac7c8bb8-9c6e-4b0e-bb84-2e70fc863d96","Type":"ContainerDied","Data":"8135440523f5b61ef4609b0b90b12dd551c89e4796c26df265bacad440db2d0f"} Oct 30 17:15:07.029944 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:07.029896 4493 scope.go:122] "RemoveContainer" containerID="8135440523f5b61ef4609b0b90b12dd551c89e4796c26df265bacad440db2d0f" Oct 30 17:15:07.030195 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:15:07.030166 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-dd715e-29889675-sjdts_odh-ai-gateway-infra(ac7c8bb8-9c6e-4b0e-bb84-2e70fc863d96)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dd715e-29889675-sjdts" podUID="ac7c8bb8-9c6e-4b0e-bb84-2e70fc863d96" Oct 30 17:15:07.031988 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:07.031878 4493 generic.go:356] "Generic (PLEG): container finished" podID="9988e846-2f00-4c0a-b72e-b2943e79f01b" containerID="ddfe1e52d36ba0887edb669191e25a402e1f10e04a94c1d3b9a801b7fbc9e35d" exitCode=60 Oct 30 17:15:07.031988 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:07.031936 4493 scope.go:122] "RemoveContainer" containerID="166af153c2aa9a19e8e76963545d5c9d2c3d4deca87b406ed88b918221f88132" Oct 30 17:15:07.031988 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:07.031963 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889675-kxmbh" event={"ID":"9988e846-2f00-4c0a-b72e-b2943e79f01b","Type":"ContainerDied","Data":"ddfe1e52d36ba0887edb669191e25a402e1f10e04a94c1d3b9a801b7fbc9e35d"} Oct 30 17:15:07.032247 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:07.032227 4493 scope.go:122] "RemoveContainer" containerID="ddfe1e52d36ba0887edb669191e25a402e1f10e04a94c1d3b9a801b7fbc9e35d" Oct 30 17:15:07.032499 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:15:07.032478 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-18519c-29889675-kxmbh_odh-ai-gateway-infra(9988e846-2f00-4c0a-b72e-b2943e79f01b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889675-kxmbh" podUID="9988e846-2f00-4c0a-b72e-b2943e79f01b" Oct 30 17:15:07.034127 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:07.034109 4493 scope.go:122] "RemoveContainer" containerID="72c4a9aaf65528e7c93df69ae468e777be2575663c5145a59244ed0dec5138d4" Oct 30 17:15:07.034369 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:15:07.034348 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-f94fb5-29889675-6w42p_odh-ai-gateway-infra(e0665578-a6c7-4b36-ba0f-5144760dce17)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f94fb5-29889675-6w42p" podUID="e0665578-a6c7-4b36-ba0f-5144760dce17" Oct 30 17:15:07.036756 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:07.036737 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8bff6e-29889675-q2zhc" event={"ID":"40a120b1-71f5-40ab-8faf-da2052ab0a56","Type":"ContainerStarted","Data":"eb59297c61434910274d29c2ecfb4865e73ec1fe0336c44340b64d2056af607f"} Oct 30 17:15:07.038899 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:07.038874 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-abda73-29889675-8gqs6" event={"ID":"25dd3114-a3ba-4976-8f11-611923dc090f","Type":"ContainerStarted","Data":"2cd212eca9fae560c9a7e4f54513fea9738f6cb577d016f0fb536a68167e753c"} Oct 30 17:15:07.040905 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:07.040881 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889675-nmmn8" event={"ID":"57140245-e2df-4ee9-9344-1de9618511d0","Type":"ContainerStarted","Data":"4b0b33da715390c2a061dd467513c0ec8632eab41c1313ba22b83ba5302cdc5f"} Oct 30 17:15:07.043765 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:07.043742 4493 generic.go:356] "Generic (PLEG): container finished" podID="7d2cd04d-1a67-45a1-8934-f9d4c9435523" containerID="4fdb02b8970aefc7523bbb3228c96c15183f2b8ee327ee45d9f1ab78783e6bd4" exitCode=60 Oct 30 17:15:07.043895 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:07.043875 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-06e7f6-29889675-m9wc6" event={"ID":"7d2cd04d-1a67-45a1-8934-f9d4c9435523","Type":"ContainerDied","Data":"4fdb02b8970aefc7523bbb3228c96c15183f2b8ee327ee45d9f1ab78783e6bd4"} Oct 30 17:15:07.044110 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:07.044090 4493 scope.go:122] "RemoveContainer" containerID="4fdb02b8970aefc7523bbb3228c96c15183f2b8ee327ee45d9f1ab78783e6bd4" Oct 30 17:15:07.044490 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:15:07.044466 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-06e7f6-29889675-m9wc6_odh-ai-gateway-infra(7d2cd04d-1a67-45a1-8934-f9d4c9435523)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-06e7f6-29889675-m9wc6" podUID="7d2cd04d-1a67-45a1-8934-f9d4c9435523" Oct 30 17:15:07.045857 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:07.045833 4493 generic.go:356] "Generic (PLEG): container finished" podID="5c296662-aa00-4ee9-90e5-2829d66ce1be" containerID="2c05a57129e4f6cd727a6c9b98f09581a20d11e75b97db527f438f18931fa287" exitCode=60 Oct 30 17:15:07.045947 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:07.045937 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3106d6-29889675-dtkjj" event={"ID":"5c296662-aa00-4ee9-90e5-2829d66ce1be","Type":"ContainerDied","Data":"2c05a57129e4f6cd727a6c9b98f09581a20d11e75b97db527f438f18931fa287"} Oct 30 17:15:07.046300 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:07.046283 4493 scope.go:122] "RemoveContainer" containerID="2c05a57129e4f6cd727a6c9b98f09581a20d11e75b97db527f438f18931fa287" Oct 30 17:15:07.046547 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:15:07.046527 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-3106d6-29889675-dtkjj_odh-ai-gateway-infra(5c296662-aa00-4ee9-90e5-2829d66ce1be)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3106d6-29889675-dtkjj" podUID="5c296662-aa00-4ee9-90e5-2829d66ce1be" Oct 30 17:15:07.057013 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:07.056991 4493 scope.go:122] "RemoveContainer" containerID="6ccb6efc2e22d38876475a47f9d341bdbf0ceeb298464627bc3492a278425340" Oct 30 17:15:07.060527 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:07.060505 4493 scope.go:122] "RemoveContainer" containerID="24183ec2e18425ac4921177ac596eb5bee3005efcb5f1bd771c266685b9f9b53" Oct 30 17:15:07.060957 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:15:07.060931 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-cb6251-29889675-tpztt_odh-ai-gateway-infra(53dc2f29-cd5a-4a6b-8e97-cc2d60925ade)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cb6251-29889675-tpztt" podUID="53dc2f29-cd5a-4a6b-8e97-cc2d60925ade" Oct 30 17:15:07.062961 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:07.062941 4493 scope.go:122] "RemoveContainer" containerID="42f282d9795a1f873ac8216a10f7d83d721088aec29b0d1034d5866298f8bf06" Oct 30 17:15:07.064174 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:15:07.064142 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-cf4621-29889675-vl2xw_odh-ai-gateway-infra(e13c30d1-8241-414d-b8fb-a31f796a805a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cf4621-29889675-vl2xw" podUID="e13c30d1-8241-414d-b8fb-a31f796a805a" Oct 30 17:15:07.064967 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:07.064945 4493 generic.go:356] "Generic (PLEG): container finished" podID="632128dd-dd61-4807-8e04-6c809d0e4711" containerID="e12ef7b556a2a50db748b4eca28d3398ccc58af45cbbf96a6a642049b3de9454" exitCode=60 Oct 30 17:15:07.065050 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:07.065008 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d5b798-29889675-hhqqx" event={"ID":"632128dd-dd61-4807-8e04-6c809d0e4711","Type":"ContainerDied","Data":"e12ef7b556a2a50db748b4eca28d3398ccc58af45cbbf96a6a642049b3de9454"} Oct 30 17:15:07.065404 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:07.065387 4493 scope.go:122] "RemoveContainer" containerID="e12ef7b556a2a50db748b4eca28d3398ccc58af45cbbf96a6a642049b3de9454" Oct 30 17:15:07.065673 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:15:07.065656 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-d5b798-29889675-hhqqx_odh-ai-gateway-infra(632128dd-dd61-4807-8e04-6c809d0e4711)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d5b798-29889675-hhqqx" podUID="632128dd-dd61-4807-8e04-6c809d0e4711" Oct 30 17:15:07.074351 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:07.074327 4493 scope.go:122] "RemoveContainer" containerID="691b82e2c38c207e8fac733e08eaee4bff58961f187a0eba8c1e70feada3a7e0" Oct 30 17:15:07.089914 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:07.089859 4493 scope.go:122] "RemoveContainer" containerID="3e48671cc4bc635875b9ed1a979af3856883b3097591540901a9806ced06a138" Oct 30 17:15:07.109956 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:07.109925 4493 scope.go:122] "RemoveContainer" containerID="04dfd24f4a1fe68bd17769ff7fe6c1006ed8cfe00911c2d7f801d93368e447cb" Oct 30 17:15:07.127848 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:07.127826 4493 scope.go:122] "RemoveContainer" containerID="643a4df46cd714b96a8d7cf8c227aa56603df52b921eb839ad5bc66e8c7c682f" Oct 30 17:15:07.909435 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:07.909399 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-24acb2/test-model-24acb2-kserve-946ff6bb7-h8cjs" Oct 30 17:15:07.922122 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:07.922092 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-24acb2/test-model-24acb2-kserve-946ff6bb7-h8cjs" Oct 30 17:15:08.040261 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:08.040226 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70ea3790-kw8mh"] Oct 30 17:15:08.049317 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:08.049274 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f32ebb4-f4cwf"] Oct 30 17:15:08.055483 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:08.055448 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70ea3790-kw8mh"] Oct 30 17:15:08.061112 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:08.061085 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f32ebb4-f4cwf"] Oct 30 17:15:08.076919 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:08.076893 4493 scope.go:122] "RemoveContainer" containerID="117d3121d638e1e0b2ff3421d9f464c76ed248ce02c92088a6a7dd3e8655ad7b" Oct 30 17:15:08.077320 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:15:08.077284 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-43197b-29889675-mm8kj_odh-ai-gateway-infra(5ff2bc89-4ecd-41d6-86ca-4111cba66bbd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-43197b-29889675-mm8kj" podUID="5ff2bc89-4ecd-41d6-86ca-4111cba66bbd" Oct 30 17:15:08.078921 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:08.078894 4493 generic.go:356] "Generic (PLEG): container finished" podID="89814242-301f-48ab-a666-b8574e1e8116" containerID="1b5bb947a2471d3b91736074bc9437670af39e771580a8598e7375925ffc9daa" exitCode=60 Oct 30 17:15:08.079040 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:08.078976 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5c6b3c-29889675-zznjt" event={"ID":"89814242-301f-48ab-a666-b8574e1e8116","Type":"ContainerDied","Data":"1b5bb947a2471d3b91736074bc9437670af39e771580a8598e7375925ffc9daa"} Oct 30 17:15:08.079040 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:08.079034 4493 scope.go:122] "RemoveContainer" containerID="b8312e84383761fa67c32cfce6a80689f8a2210c344834da2094c88d5260aeaa" Oct 30 17:15:08.079293 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:08.079267 4493 scope.go:122] "RemoveContainer" containerID="1b5bb947a2471d3b91736074bc9437670af39e771580a8598e7375925ffc9daa" Oct 30 17:15:08.079666 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:15:08.079640 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-5c6b3c-29889675-zznjt_odh-ai-gateway-infra(89814242-301f-48ab-a666-b8574e1e8116)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5c6b3c-29889675-zznjt" podUID="89814242-301f-48ab-a666-b8574e1e8116" Oct 30 17:15:08.081320 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:08.081303 4493 scope.go:122] "RemoveContainer" containerID="dcb3effadf2d9c1a067684e464f6ea7fd6273d0e9a81c74db8638f63fcab2d9e" Oct 30 17:15:08.081629 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:15:08.081610 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-24acb2-29889675-wttj8_odh-ai-gateway-infra(49589f83-94f8-4e46-808f-cb013c6adfeb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24acb2-29889675-wttj8" podUID="49589f83-94f8-4e46-808f-cb013c6adfeb" Oct 30 17:15:08.083059 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:08.083042 4493 scope.go:122] "RemoveContainer" containerID="8135440523f5b61ef4609b0b90b12dd551c89e4796c26df265bacad440db2d0f" Oct 30 17:15:08.083320 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:15:08.083297 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-dd715e-29889675-sjdts_odh-ai-gateway-infra(ac7c8bb8-9c6e-4b0e-bb84-2e70fc863d96)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dd715e-29889675-sjdts" podUID="ac7c8bb8-9c6e-4b0e-bb84-2e70fc863d96" Oct 30 17:15:08.084762 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:08.084745 4493 scope.go:122] "RemoveContainer" containerID="ddfe1e52d36ba0887edb669191e25a402e1f10e04a94c1d3b9a801b7fbc9e35d" Oct 30 17:15:08.085053 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:15:08.085024 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-18519c-29889675-kxmbh_odh-ai-gateway-infra(9988e846-2f00-4c0a-b72e-b2943e79f01b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889675-kxmbh" podUID="9988e846-2f00-4c0a-b72e-b2943e79f01b" Oct 30 17:15:08.086627 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:08.086603 4493 generic.go:356] "Generic (PLEG): container finished" podID="40a120b1-71f5-40ab-8faf-da2052ab0a56" containerID="eb59297c61434910274d29c2ecfb4865e73ec1fe0336c44340b64d2056af607f" exitCode=60 Oct 30 17:15:08.086717 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:08.086688 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8bff6e-29889675-q2zhc" event={"ID":"40a120b1-71f5-40ab-8faf-da2052ab0a56","Type":"ContainerDied","Data":"eb59297c61434910274d29c2ecfb4865e73ec1fe0336c44340b64d2056af607f"} Oct 30 17:15:08.086984 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:08.086963 4493 scope.go:122] "RemoveContainer" containerID="eb59297c61434910274d29c2ecfb4865e73ec1fe0336c44340b64d2056af607f" Oct 30 17:15:08.087269 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:15:08.087248 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-8bff6e-29889675-q2zhc_odh-ai-gateway-infra(40a120b1-71f5-40ab-8faf-da2052ab0a56)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8bff6e-29889675-q2zhc" podUID="40a120b1-71f5-40ab-8faf-da2052ab0a56" Oct 30 17:15:08.088671 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:08.088651 4493 generic.go:356] "Generic (PLEG): container finished" podID="25dd3114-a3ba-4976-8f11-611923dc090f" containerID="2cd212eca9fae560c9a7e4f54513fea9738f6cb577d016f0fb536a68167e753c" exitCode=60 Oct 30 17:15:08.088763 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:08.088695 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-abda73-29889675-8gqs6" event={"ID":"25dd3114-a3ba-4976-8f11-611923dc090f","Type":"ContainerDied","Data":"2cd212eca9fae560c9a7e4f54513fea9738f6cb577d016f0fb536a68167e753c"} Oct 30 17:15:08.088960 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:08.088944 4493 scope.go:122] "RemoveContainer" containerID="2cd212eca9fae560c9a7e4f54513fea9738f6cb577d016f0fb536a68167e753c" Oct 30 17:15:08.089180 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:15:08.089160 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-abda73-29889675-8gqs6_odh-ai-gateway-infra(25dd3114-a3ba-4976-8f11-611923dc090f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-abda73-29889675-8gqs6" podUID="25dd3114-a3ba-4976-8f11-611923dc090f" Oct 30 17:15:08.090879 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:08.090859 4493 generic.go:356] "Generic (PLEG): container finished" podID="57140245-e2df-4ee9-9344-1de9618511d0" containerID="4b0b33da715390c2a061dd467513c0ec8632eab41c1313ba22b83ba5302cdc5f" exitCode=60 Oct 30 17:15:08.090981 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:08.090948 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889675-nmmn8" event={"ID":"57140245-e2df-4ee9-9344-1de9618511d0","Type":"ContainerDied","Data":"4b0b33da715390c2a061dd467513c0ec8632eab41c1313ba22b83ba5302cdc5f"} Oct 30 17:15:08.091144 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:08.091128 4493 scope.go:122] "RemoveContainer" containerID="4b0b33da715390c2a061dd467513c0ec8632eab41c1313ba22b83ba5302cdc5f" Oct 30 17:15:08.091375 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:15:08.091356 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-2d365c-29889675-nmmn8_odh-ai-gateway-infra(57140245-e2df-4ee9-9344-1de9618511d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889675-nmmn8" podUID="57140245-e2df-4ee9-9344-1de9618511d0" Oct 30 17:15:08.092938 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:08.092922 4493 scope.go:122] "RemoveContainer" containerID="4fdb02b8970aefc7523bbb3228c96c15183f2b8ee327ee45d9f1ab78783e6bd4" Oct 30 17:15:08.093179 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:15:08.093157 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-06e7f6-29889675-m9wc6_odh-ai-gateway-infra(7d2cd04d-1a67-45a1-8934-f9d4c9435523)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-06e7f6-29889675-m9wc6" podUID="7d2cd04d-1a67-45a1-8934-f9d4c9435523" Oct 30 17:15:08.095176 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:08.095151 4493 scope.go:122] "RemoveContainer" containerID="2c05a57129e4f6cd727a6c9b98f09581a20d11e75b97db527f438f18931fa287" Oct 30 17:15:08.095402 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:15:08.095378 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-3106d6-29889675-dtkjj_odh-ai-gateway-infra(5c296662-aa00-4ee9-90e5-2829d66ce1be)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3106d6-29889675-dtkjj" podUID="5c296662-aa00-4ee9-90e5-2829d66ce1be" Oct 30 17:15:08.097046 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:08.097028 4493 scope.go:122] "RemoveContainer" containerID="4296a0bc73250f249eeeec21710d44c738013ab4cca1c62704824cb1ca200147" Oct 30 17:15:08.097267 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:08.097246 4493 scope.go:122] "RemoveContainer" containerID="e12ef7b556a2a50db748b4eca28d3398ccc58af45cbbf96a6a642049b3de9454" Oct 30 17:15:08.097546 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:15:08.097510 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-d5b798-29889675-hhqqx_odh-ai-gateway-infra(632128dd-dd61-4807-8e04-6c809d0e4711)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d5b798-29889675-hhqqx" podUID="632128dd-dd61-4807-8e04-6c809d0e4711" Oct 30 17:15:08.113237 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:08.113204 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-24acb2/test-model-24acb2-kserve-946ff6bb7-h8cjs" Oct 30 17:15:08.115852 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:08.115824 4493 scope.go:122] "RemoveContainer" containerID="0a88a17207a4f19c61bd883bc570847234c984d2b1f67a63fb331c7f230e8345" Oct 30 17:15:08.139099 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:08.138764 4493 scope.go:122] "RemoveContainer" containerID="f03c209a9d5c5d5a3e1050cd82e1f837004c96405215457204da5460bb4d3268" Oct 30 17:15:08.938714 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:08.938678 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a41054e2-8a2b-4ccf-a336-1fc8aa76f378" path="/var/lib/kubelet/pods/a41054e2-8a2b-4ccf-a336-1fc8aa76f378/volumes" Oct 30 17:15:08.939237 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:08.939204 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9f9cccb-f1e1-43e6-861b-eca7d198b1de" path="/var/lib/kubelet/pods/c9f9cccb-f1e1-43e6-861b-eca7d198b1de/volumes" Oct 30 17:15:09.039113 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:09.039015 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fda72a4f-j869l"] Oct 30 17:15:09.048334 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:09.048295 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fda72a4f-j869l"] Oct 30 17:15:09.112269 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:09.112238 4493 scope.go:122] "RemoveContainer" containerID="eb59297c61434910274d29c2ecfb4865e73ec1fe0336c44340b64d2056af607f" Oct 30 17:15:09.113209 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:15:09.113168 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-8bff6e-29889675-q2zhc_odh-ai-gateway-infra(40a120b1-71f5-40ab-8faf-da2052ab0a56)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8bff6e-29889675-q2zhc" podUID="40a120b1-71f5-40ab-8faf-da2052ab0a56" Oct 30 17:15:09.114323 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:09.114297 4493 scope.go:122] "RemoveContainer" containerID="2cd212eca9fae560c9a7e4f54513fea9738f6cb577d016f0fb536a68167e753c" Oct 30 17:15:09.114727 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:15:09.114692 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-abda73-29889675-8gqs6_odh-ai-gateway-infra(25dd3114-a3ba-4976-8f11-611923dc090f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-abda73-29889675-8gqs6" podUID="25dd3114-a3ba-4976-8f11-611923dc090f" Oct 30 17:15:09.116038 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:09.116017 4493 scope.go:122] "RemoveContainer" containerID="4b0b33da715390c2a061dd467513c0ec8632eab41c1313ba22b83ba5302cdc5f" Oct 30 17:15:09.116380 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:15:09.116356 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-2d365c-29889675-nmmn8_odh-ai-gateway-infra(57140245-e2df-4ee9-9344-1de9618511d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889675-nmmn8" podUID="57140245-e2df-4ee9-9344-1de9618511d0" Oct 30 17:15:09.118302 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:09.118281 4493 scope.go:122] "RemoveContainer" containerID="1b5bb947a2471d3b91736074bc9437670af39e771580a8598e7375925ffc9daa" Oct 30 17:15:09.118529 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:15:09.118508 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-5c6b3c-29889675-zznjt_odh-ai-gateway-infra(89814242-301f-48ab-a666-b8574e1e8116)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5c6b3c-29889675-zznjt" podUID="89814242-301f-48ab-a666-b8574e1e8116" Oct 30 17:15:10.935250 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:10.935214 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b46b93f9-fb79-4f47-b25d-549cd661e9a5" path="/var/lib/kubelet/pods/b46b93f9-fb79-4f47-b25d-549cd661e9a5/volumes" Oct 30 17:15:13.041216 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:13.041182 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70d329be-27kfd"] Oct 30 17:15:13.053288 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:13.053258 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70d329be-27kfd"] Oct 30 17:15:14.939293 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:14.939259 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cead726f-c70c-4c0a-acb0-c3381f720710" path="/var/lib/kubelet/pods/cead726f-c70c-4c0a-acb0-c3381f720710/volumes" Oct 30 17:15:15.929318 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:15.929295 4493 scope.go:122] "RemoveContainer" containerID="0e4047b78a836a2d3ffda6955e8722e26319bd739d53229473308a90a531e673" Oct 30 17:15:16.939950 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:16.939930 4493 scope.go:122] "RemoveContainer" containerID="42f282d9795a1f873ac8216a10f7d83d721088aec29b0d1034d5866298f8bf06" Oct 30 17:15:17.241337 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:17.241299 4493 generic.go:356] "Generic (PLEG): container finished" podID="43be3031-1e16-4687-97f9-f7af506d98b5" containerID="c643f824c46a7588f75ac4c0129ecfb4dc38ec58c7e5ce703c746476cfd46021" exitCode=60 Oct 30 17:15:17.241505 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:17.241370 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889675-d8tcq" event={"ID":"43be3031-1e16-4687-97f9-f7af506d98b5","Type":"ContainerDied","Data":"c643f824c46a7588f75ac4c0129ecfb4dc38ec58c7e5ce703c746476cfd46021"} Oct 30 17:15:17.241505 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:17.241434 4493 scope.go:122] "RemoveContainer" containerID="0e4047b78a836a2d3ffda6955e8722e26319bd739d53229473308a90a531e673" Oct 30 17:15:17.241955 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:17.241933 4493 scope.go:122] "RemoveContainer" containerID="c643f824c46a7588f75ac4c0129ecfb4dc38ec58c7e5ce703c746476cfd46021" Oct 30 17:15:17.242373 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:15:17.242348 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f3503b-29889675-d8tcq_odh-ai-gateway-infra(43be3031-1e16-4687-97f9-f7af506d98b5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889675-d8tcq" podUID="43be3031-1e16-4687-97f9-f7af506d98b5" Oct 30 17:15:17.925971 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:17.925938 4493 scope.go:122] "RemoveContainer" containerID="a448f7e577825193408cedb18957c7733beb76591f5dbf56bc4927ac74242f7a" Oct 30 17:15:17.926291 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:17.926043 4493 scope.go:122] "RemoveContainer" containerID="2cd212eca9fae560c9a7e4f54513fea9738f6cb577d016f0fb536a68167e753c" Oct 30 17:15:17.926291 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:17.926250 4493 scope.go:122] "RemoveContainer" containerID="eb59297c61434910274d29c2ecfb4865e73ec1fe0336c44340b64d2056af607f" Oct 30 17:15:17.927199 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:17.926777 4493 scope.go:122] "RemoveContainer" containerID="e12ef7b556a2a50db748b4eca28d3398ccc58af45cbbf96a6a642049b3de9454" Oct 30 17:15:18.255020 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:18.254991 4493 generic.go:356] "Generic (PLEG): container finished" podID="e13c30d1-8241-414d-b8fb-a31f796a805a" containerID="f397b024710d714a48689594d41190314559be0cfda5d97d325864f76e20f47b" exitCode=60 Oct 30 17:15:18.255429 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:18.255066 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cf4621-29889675-vl2xw" event={"ID":"e13c30d1-8241-414d-b8fb-a31f796a805a","Type":"ContainerDied","Data":"f397b024710d714a48689594d41190314559be0cfda5d97d325864f76e20f47b"} Oct 30 17:15:18.255429 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:18.255127 4493 scope.go:122] "RemoveContainer" containerID="42f282d9795a1f873ac8216a10f7d83d721088aec29b0d1034d5866298f8bf06" Oct 30 17:15:18.255602 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:18.255583 4493 scope.go:122] "RemoveContainer" containerID="f397b024710d714a48689594d41190314559be0cfda5d97d325864f76e20f47b" Oct 30 17:15:18.256043 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:15:18.256021 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-cf4621-29889675-vl2xw_odh-ai-gateway-infra(e13c30d1-8241-414d-b8fb-a31f796a805a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cf4621-29889675-vl2xw" podUID="e13c30d1-8241-414d-b8fb-a31f796a805a" Oct 30 17:15:18.295744 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:18.295719 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889675-d8tcq"] Oct 30 17:15:18.558861 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:18.558839 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889675-d8tcq" Oct 30 17:15:18.698854 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:18.698821 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43be3031-1e16-4687-97f9-f7af506d98b5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43be3031-1e16-4687-97f9-f7af506d98b5-maas-api-service-ca\") pod \"43be3031-1e16-4687-97f9-f7af506d98b5\" (UID: \"43be3031-1e16-4687-97f9-f7af506d98b5\") " Oct 30 17:15:18.699068 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:18.698865 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43be3031-1e16-4687-97f9-f7af506d98b5-kube-api-access-sdzqc\" (UniqueName: \"kubernetes.io/projected/43be3031-1e16-4687-97f9-f7af506d98b5-kube-api-access-sdzqc\") pod \"43be3031-1e16-4687-97f9-f7af506d98b5\" (UID: \"43be3031-1e16-4687-97f9-f7af506d98b5\") " Oct 30 17:15:18.699213 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:18.699186 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43be3031-1e16-4687-97f9-f7af506d98b5-maas-api-service-ca" pod "43be3031-1e16-4687-97f9-f7af506d98b5" (UID: "43be3031-1e16-4687-97f9-f7af506d98b5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:18.699528 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:18.699501 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43be3031-1e16-4687-97f9-f7af506d98b5-maas-api-service-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:15:18.702277 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:18.702255 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43be3031-1e16-4687-97f9-f7af506d98b5-kube-api-access-sdzqc" pod "43be3031-1e16-4687-97f9-f7af506d98b5" (UID: "43be3031-1e16-4687-97f9-f7af506d98b5"). InnerVolumeSpecName "kube-api-access-sdzqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:18.801179 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:18.801144 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdzqc\" (UniqueName: \"kubernetes.io/projected/43be3031-1e16-4687-97f9-f7af506d98b5-kube-api-access-sdzqc\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:15:18.937582 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:18.937548 4493 scope.go:122] "RemoveContainer" containerID="24183ec2e18425ac4921177ac596eb5bee3005efcb5f1bd771c266685b9f9b53" Oct 30 17:15:18.938037 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:18.938010 4493 scope.go:122] "RemoveContainer" containerID="b8ba8f34ce3c08e71f356f1929d6903b3d7fffe515309b6726e6f952641b5eb2" Oct 30 17:15:18.938459 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:18.938440 4493 scope.go:122] "RemoveContainer" containerID="82e4eaa41c79586864ec3d9b537aebdd262a9a6686ae8e56bf13382bbeeea4db" Oct 30 17:15:18.938740 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:18.938694 4493 scope.go:122] "RemoveContainer" containerID="117d3121d638e1e0b2ff3421d9f464c76ed248ce02c92088a6a7dd3e8655ad7b" Oct 30 17:15:18.939159 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:18.939142 4493 scope.go:122] "RemoveContainer" containerID="4b0b33da715390c2a061dd467513c0ec8632eab41c1313ba22b83ba5302cdc5f" Oct 30 17:15:18.939318 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:18.939253 4493 scope.go:122] "RemoveContainer" containerID="72c4a9aaf65528e7c93df69ae468e777be2575663c5145a59244ed0dec5138d4" Oct 30 17:15:18.939592 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:18.939569 4493 scope.go:122] "RemoveContainer" containerID="2c05a57129e4f6cd727a6c9b98f09581a20d11e75b97db527f438f18931fa287" Oct 30 17:15:18.939698 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:18.939675 4493 scope.go:122] "RemoveContainer" containerID="1b5bb947a2471d3b91736074bc9437670af39e771580a8598e7375925ffc9daa" Oct 30 17:15:19.039730 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:19.039703 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c03254f-9rlgl"] Oct 30 17:15:19.049813 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:19.049768 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c03254f-9rlgl"] Oct 30 17:15:19.268055 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:19.267973 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8bff6e-29889675-q2zhc" event={"ID":"40a120b1-71f5-40ab-8faf-da2052ab0a56","Type":"ContainerStarted","Data":"b8b059f37f256802cf2a02f5652a9b4133ba913eb13f8f6f9082a269b689c074"} Oct 30 17:15:19.270438 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:19.270408 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-abda73-29889675-8gqs6" event={"ID":"25dd3114-a3ba-4976-8f11-611923dc090f","Type":"ContainerStarted","Data":"d261ba8ed04c10026978853db64fbbdf1e39144fc9847472d69f358cd9f177eb"} Oct 30 17:15:19.272800 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:19.272766 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2589b5-29889675-h7lhw" event={"ID":"045dd183-6322-48cb-85df-fa7cc872f8cd","Type":"ContainerStarted","Data":"80cd318e58a1f4c6f897b1d236b8f0ad128df7707453523320c43c400544cdef"} Oct 30 17:15:19.276386 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:19.276363 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d5b798-29889675-hhqqx" event={"ID":"632128dd-dd61-4807-8e04-6c809d0e4711","Type":"ContainerStarted","Data":"49dac82fea46abca46c9c3d0d276a1d916d48cb6dbf82d44927dfd0e9f451584"} Oct 30 17:15:19.277730 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:19.277715 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889675-d8tcq" Oct 30 17:15:19.277823 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:19.277752 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889675-d8tcq" event={"ID":"43be3031-1e16-4687-97f9-f7af506d98b5","Type":"ContainerDied","Data":"8943b60b629e463b0685c7666bde22517bdf7d22f79cc483e1d2ca3cccacbed2"} Oct 30 17:15:19.277874 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:19.277843 4493 scope.go:122] "RemoveContainer" containerID="c643f824c46a7588f75ac4c0129ecfb4dc38ec58c7e5ce703c746476cfd46021" Oct 30 17:15:19.320843 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:19.320765 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cf4621-29889675-vl2xw"] Oct 30 17:15:19.434868 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:19.434838 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889675-d8tcq"] Oct 30 17:15:19.440376 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:19.439580 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f3503b-29889675-d8tcq"] Oct 30 17:15:19.684017 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:19.683978 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cf4621-29889675-vl2xw" Oct 30 17:15:19.758571 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:19.758539 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e13c30d1-8241-414d-b8fb-a31f796a805a-kube-api-access-m4n8n\" (UniqueName: \"kubernetes.io/projected/e13c30d1-8241-414d-b8fb-a31f796a805a-kube-api-access-m4n8n\") pod \"e13c30d1-8241-414d-b8fb-a31f796a805a\" (UID: \"e13c30d1-8241-414d-b8fb-a31f796a805a\") " Oct 30 17:15:19.759092 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:19.759065 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e13c30d1-8241-414d-b8fb-a31f796a805a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e13c30d1-8241-414d-b8fb-a31f796a805a-maas-api-service-ca\") pod \"e13c30d1-8241-414d-b8fb-a31f796a805a\" (UID: \"e13c30d1-8241-414d-b8fb-a31f796a805a\") " Oct 30 17:15:19.759858 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:19.759541 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e13c30d1-8241-414d-b8fb-a31f796a805a-maas-api-service-ca" pod "e13c30d1-8241-414d-b8fb-a31f796a805a" (UID: "e13c30d1-8241-414d-b8fb-a31f796a805a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:19.760293 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:19.760275 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e13c30d1-8241-414d-b8fb-a31f796a805a-maas-api-service-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:15:19.761018 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:19.760990 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e13c30d1-8241-414d-b8fb-a31f796a805a-kube-api-access-m4n8n" pod "e13c30d1-8241-414d-b8fb-a31f796a805a" (UID: "e13c30d1-8241-414d-b8fb-a31f796a805a"). InnerVolumeSpecName "kube-api-access-m4n8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:19.861462 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:19.861431 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4n8n\" (UniqueName: \"kubernetes.io/projected/e13c30d1-8241-414d-b8fb-a31f796a805a-kube-api-access-m4n8n\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:15:19.926997 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:19.926245 4493 scope.go:122] "RemoveContainer" containerID="ddfe1e52d36ba0887edb669191e25a402e1f10e04a94c1d3b9a801b7fbc9e35d" Oct 30 17:15:19.926997 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:19.926455 4493 scope.go:122] "RemoveContainer" containerID="f1c2ea551a6ba264d1929f69785d7ecb2ec903340cec93139511504c7ffee742" Oct 30 17:15:19.926997 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:19.926673 4493 scope.go:122] "RemoveContainer" containerID="8135440523f5b61ef4609b0b90b12dd551c89e4796c26df265bacad440db2d0f" Oct 30 17:15:19.927480 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:19.927399 4493 scope.go:122] "RemoveContainer" containerID="dcb3effadf2d9c1a067684e464f6ea7fd6273d0e9a81c74db8638f63fcab2d9e" Oct 30 17:15:20.041557 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:20.041519 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82d4a543-x4slv"] Oct 30 17:15:20.059574 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:20.059537 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82d4a543-x4slv"] Oct 30 17:15:20.288732 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:20.288696 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5c6b3c-29889675-zznjt" event={"ID":"89814242-301f-48ab-a666-b8574e1e8116","Type":"ContainerStarted","Data":"350abc87b73911ff0b51d6e39512908153add30cf811ddede3b234818ae1c235"} Oct 30 17:15:20.291846 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:20.291818 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f94fb5-29889675-6w42p" event={"ID":"e0665578-a6c7-4b36-ba0f-5144760dce17","Type":"ContainerStarted","Data":"20722bbe4572ddabdf19be37ebf67dcee7efae2e985540a07c63d548459984b0"} Oct 30 17:15:20.293703 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:20.293680 4493 generic.go:356] "Generic (PLEG): container finished" podID="40a120b1-71f5-40ab-8faf-da2052ab0a56" containerID="b8b059f37f256802cf2a02f5652a9b4133ba913eb13f8f6f9082a269b689c074" exitCode=60 Oct 30 17:15:20.293851 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:20.293755 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8bff6e-29889675-q2zhc" event={"ID":"40a120b1-71f5-40ab-8faf-da2052ab0a56","Type":"ContainerDied","Data":"b8b059f37f256802cf2a02f5652a9b4133ba913eb13f8f6f9082a269b689c074"} Oct 30 17:15:20.293851 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:20.293833 4493 scope.go:122] "RemoveContainer" containerID="eb59297c61434910274d29c2ecfb4865e73ec1fe0336c44340b64d2056af607f" Oct 30 17:15:20.294222 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:20.294206 4493 scope.go:122] "RemoveContainer" containerID="b8b059f37f256802cf2a02f5652a9b4133ba913eb13f8f6f9082a269b689c074" Oct 30 17:15:20.294592 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:15:20.294566 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-8bff6e-29889675-q2zhc_odh-ai-gateway-infra(40a120b1-71f5-40ab-8faf-da2052ab0a56)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8bff6e-29889675-q2zhc" podUID="40a120b1-71f5-40ab-8faf-da2052ab0a56" Oct 30 17:15:20.295828 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:20.295808 4493 generic.go:356] "Generic (PLEG): container finished" podID="25dd3114-a3ba-4976-8f11-611923dc090f" containerID="d261ba8ed04c10026978853db64fbbdf1e39144fc9847472d69f358cd9f177eb" exitCode=60 Oct 30 17:15:20.295941 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:20.295870 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-abda73-29889675-8gqs6" event={"ID":"25dd3114-a3ba-4976-8f11-611923dc090f","Type":"ContainerDied","Data":"d261ba8ed04c10026978853db64fbbdf1e39144fc9847472d69f358cd9f177eb"} Oct 30 17:15:20.296251 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:20.296235 4493 scope.go:122] "RemoveContainer" containerID="d261ba8ed04c10026978853db64fbbdf1e39144fc9847472d69f358cd9f177eb" Oct 30 17:15:20.296553 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:15:20.296535 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-abda73-29889675-8gqs6_odh-ai-gateway-infra(25dd3114-a3ba-4976-8f11-611923dc090f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-abda73-29889675-8gqs6" podUID="25dd3114-a3ba-4976-8f11-611923dc090f" Oct 30 17:15:20.297988 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:20.297969 4493 generic.go:356] "Generic (PLEG): container finished" podID="57140245-e2df-4ee9-9344-1de9618511d0" containerID="56c376ba2995118cc1f7f3f0dd0926842244a4bea27e7089f298f75c7fa8c95e" exitCode=60 Oct 30 17:15:20.298090 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:20.298043 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889675-nmmn8" event={"ID":"57140245-e2df-4ee9-9344-1de9618511d0","Type":"ContainerDied","Data":"56c376ba2995118cc1f7f3f0dd0926842244a4bea27e7089f298f75c7fa8c95e"} Oct 30 17:15:20.298378 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:20.298361 4493 scope.go:122] "RemoveContainer" containerID="56c376ba2995118cc1f7f3f0dd0926842244a4bea27e7089f298f75c7fa8c95e" Oct 30 17:15:20.298666 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:15:20.298642 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-2d365c-29889675-nmmn8_odh-ai-gateway-infra(57140245-e2df-4ee9-9344-1de9618511d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889675-nmmn8" podUID="57140245-e2df-4ee9-9344-1de9618511d0" Oct 30 17:15:20.300061 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:20.300039 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3106d6-29889675-dtkjj" event={"ID":"5c296662-aa00-4ee9-90e5-2829d66ce1be","Type":"ContainerStarted","Data":"20b750c07652a27b019b0ad166633efc953c337ae0652996274e6c28ac35eed5"} Oct 30 17:15:20.302070 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:20.302051 4493 generic.go:356] "Generic (PLEG): container finished" podID="045dd183-6322-48cb-85df-fa7cc872f8cd" containerID="80cd318e58a1f4c6f897b1d236b8f0ad128df7707453523320c43c400544cdef" exitCode=60 Oct 30 17:15:20.302158 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:20.302120 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2589b5-29889675-h7lhw" event={"ID":"045dd183-6322-48cb-85df-fa7cc872f8cd","Type":"ContainerDied","Data":"80cd318e58a1f4c6f897b1d236b8f0ad128df7707453523320c43c400544cdef"} Oct 30 17:15:20.302431 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:20.302418 4493 scope.go:122] "RemoveContainer" containerID="80cd318e58a1f4c6f897b1d236b8f0ad128df7707453523320c43c400544cdef" Oct 30 17:15:20.302663 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:15:20.302645 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-2589b5-29889675-h7lhw_odh-ai-gateway-infra(045dd183-6322-48cb-85df-fa7cc872f8cd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2589b5-29889675-h7lhw" podUID="045dd183-6322-48cb-85df-fa7cc872f8cd" Oct 30 17:15:20.304153 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:20.304128 4493 generic.go:356] "Generic (PLEG): container finished" podID="53dc2f29-cd5a-4a6b-8e97-cc2d60925ade" containerID="3e4a8d2a8c1000091e2b1e2c5e1dff63c9c042ccde82e2c0cd54862a795e9324" exitCode=60 Oct 30 17:15:20.304240 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:20.304204 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cb6251-29889675-tpztt" event={"ID":"53dc2f29-cd5a-4a6b-8e97-cc2d60925ade","Type":"ContainerDied","Data":"3e4a8d2a8c1000091e2b1e2c5e1dff63c9c042ccde82e2c0cd54862a795e9324"} Oct 30 17:15:20.304658 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:20.304612 4493 scope.go:122] "RemoveContainer" containerID="3e4a8d2a8c1000091e2b1e2c5e1dff63c9c042ccde82e2c0cd54862a795e9324" Oct 30 17:15:20.304939 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:15:20.304916 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-cb6251-29889675-tpztt_odh-ai-gateway-infra(53dc2f29-cd5a-4a6b-8e97-cc2d60925ade)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cb6251-29889675-tpztt" podUID="53dc2f29-cd5a-4a6b-8e97-cc2d60925ade" Oct 30 17:15:20.306848 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:20.306179 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cf4621-29889675-vl2xw" Oct 30 17:15:20.306848 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:20.306191 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cf4621-29889675-vl2xw" event={"ID":"e13c30d1-8241-414d-b8fb-a31f796a805a","Type":"ContainerDied","Data":"02892a1e40d0072539a1bbd1f0adfa80655f7affdbcf45410e302f480cb33793"} Oct 30 17:15:20.316841 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:20.314440 4493 generic.go:356] "Generic (PLEG): container finished" podID="632128dd-dd61-4807-8e04-6c809d0e4711" containerID="49dac82fea46abca46c9c3d0d276a1d916d48cb6dbf82d44927dfd0e9f451584" exitCode=60 Oct 30 17:15:20.316841 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:20.314561 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d5b798-29889675-hhqqx" event={"ID":"632128dd-dd61-4807-8e04-6c809d0e4711","Type":"ContainerDied","Data":"49dac82fea46abca46c9c3d0d276a1d916d48cb6dbf82d44927dfd0e9f451584"} Oct 30 17:15:20.316841 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:20.315016 4493 scope.go:122] "RemoveContainer" containerID="49dac82fea46abca46c9c3d0d276a1d916d48cb6dbf82d44927dfd0e9f451584" Oct 30 17:15:20.316841 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:15:20.315330 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-d5b798-29889675-hhqqx_odh-ai-gateway-infra(632128dd-dd61-4807-8e04-6c809d0e4711)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d5b798-29889675-hhqqx" podUID="632128dd-dd61-4807-8e04-6c809d0e4711" Oct 30 17:15:20.319329 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:20.319304 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-13c495-29889675-79wjh" event={"ID":"b72976e0-c501-44b4-b42c-5dfa26192e87","Type":"ContainerStarted","Data":"abdd9705b0b1a24d742d1f510d3dc3563417ea22443dabefecf20ba1a37df37f"} Oct 30 17:15:20.321968 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:20.321941 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6acc78-29889675-664bj" event={"ID":"c4086c1e-112f-4518-8ae2-3fa5526f9357","Type":"ContainerStarted","Data":"e467effa1264a238d1ae0195f0b60ac618c76a123cfc4db81aaa5049f9134433"} Oct 30 17:15:20.325696 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:20.325667 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-43197b-29889675-mm8kj" event={"ID":"5ff2bc89-4ecd-41d6-86ca-4111cba66bbd","Type":"ContainerStarted","Data":"4fa29b5d7868ce7ae51cfb86fdf611f6731ea5e406ed6f1e8e5e4521891bbf47"} Oct 30 17:15:20.329278 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:20.329251 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8bff6e-29889675-q2zhc"] Oct 30 17:15:20.365292 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:20.365265 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2589b5-29889675-h7lhw"] Oct 30 17:15:20.404054 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:20.404022 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-abda73-29889675-8gqs6"] Oct 30 17:15:20.485547 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:20.485514 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d5b798-29889675-hhqqx"] Oct 30 17:15:20.820058 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:20.820033 4493 scope.go:122] "RemoveContainer" containerID="2cd212eca9fae560c9a7e4f54513fea9738f6cb577d016f0fb536a68167e753c" Oct 30 17:15:20.853667 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:20.853637 4493 scope.go:122] "RemoveContainer" containerID="4b0b33da715390c2a061dd467513c0ec8632eab41c1313ba22b83ba5302cdc5f" Oct 30 17:15:20.871124 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:20.871098 4493 scope.go:122] "RemoveContainer" containerID="a448f7e577825193408cedb18957c7733beb76591f5dbf56bc4927ac74242f7a" Oct 30 17:15:20.887227 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:20.887193 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cf4621-29889675-vl2xw"] Oct 30 17:15:20.895317 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:20.895288 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cf4621-29889675-vl2xw"] Oct 30 17:15:20.895898 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:20.895875 4493 scope.go:122] "RemoveContainer" containerID="24183ec2e18425ac4921177ac596eb5bee3005efcb5f1bd771c266685b9f9b53" Oct 30 17:15:20.916050 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:20.916027 4493 scope.go:122] "RemoveContainer" containerID="f397b024710d714a48689594d41190314559be0cfda5d97d325864f76e20f47b" Oct 30 17:15:20.952755 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:20.952678 4493 scope.go:122] "RemoveContainer" containerID="4fdb02b8970aefc7523bbb3228c96c15183f2b8ee327ee45d9f1ab78783e6bd4" Oct 30 17:15:20.959103 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:20.959082 4493 scope.go:122] "RemoveContainer" containerID="e12ef7b556a2a50db748b4eca28d3398ccc58af45cbbf96a6a642049b3de9454" Oct 30 17:15:20.966243 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:20.966216 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43be3031-1e16-4687-97f9-f7af506d98b5" path="/var/lib/kubelet/pods/43be3031-1e16-4687-97f9-f7af506d98b5/volumes" Oct 30 17:15:20.966838 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:20.966815 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8abd1354-46f4-40cc-b5cc-c86321cfb197" path="/var/lib/kubelet/pods/8abd1354-46f4-40cc-b5cc-c86321cfb197/volumes" Oct 30 17:15:20.967300 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:20.967278 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e13c30d1-8241-414d-b8fb-a31f796a805a" path="/var/lib/kubelet/pods/e13c30d1-8241-414d-b8fb-a31f796a805a/volumes" Oct 30 17:15:20.968209 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:20.968189 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4075789-9024-4cb3-bf1b-fd540609361f" path="/var/lib/kubelet/pods/f4075789-9024-4cb3-bf1b-fd540609361f/volumes" Oct 30 17:15:21.341975 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:21.341938 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5c6b3c-29889675-zznjt"] Oct 30 17:15:21.352846 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:21.352253 4493 generic.go:356] "Generic (PLEG): container finished" podID="49589f83-94f8-4e46-808f-cb013c6adfeb" containerID="617d8bff48447f0d212ca81b3af05fc84072371d723b3861b4587730dff5b171" exitCode=60 Oct 30 17:15:21.352846 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:21.352334 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24acb2-29889675-wttj8" event={"ID":"49589f83-94f8-4e46-808f-cb013c6adfeb","Type":"ContainerDied","Data":"617d8bff48447f0d212ca81b3af05fc84072371d723b3861b4587730dff5b171"} Oct 30 17:15:21.352846 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:21.352399 4493 scope.go:122] "RemoveContainer" containerID="dcb3effadf2d9c1a067684e464f6ea7fd6273d0e9a81c74db8638f63fcab2d9e" Oct 30 17:15:21.352846 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:21.352810 4493 scope.go:122] "RemoveContainer" containerID="617d8bff48447f0d212ca81b3af05fc84072371d723b3861b4587730dff5b171" Oct 30 17:15:21.353217 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:15:21.353096 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-24acb2-29889675-wttj8_odh-ai-gateway-infra(49589f83-94f8-4e46-808f-cb013c6adfeb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24acb2-29889675-wttj8" podUID="49589f83-94f8-4e46-808f-cb013c6adfeb" Oct 30 17:15:21.355766 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:21.355744 4493 generic.go:356] "Generic (PLEG): container finished" podID="ac7c8bb8-9c6e-4b0e-bb84-2e70fc863d96" containerID="b6fb93f9b51e5bbe29b73d564dea34dec564db08297b2c944d9c3c39a13dff98" exitCode=60 Oct 30 17:15:21.355909 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:21.355841 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dd715e-29889675-sjdts" event={"ID":"ac7c8bb8-9c6e-4b0e-bb84-2e70fc863d96","Type":"ContainerDied","Data":"b6fb93f9b51e5bbe29b73d564dea34dec564db08297b2c944d9c3c39a13dff98"} Oct 30 17:15:21.356242 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:21.356227 4493 scope.go:122] "RemoveContainer" containerID="b6fb93f9b51e5bbe29b73d564dea34dec564db08297b2c944d9c3c39a13dff98" Oct 30 17:15:21.356612 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:15:21.356575 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-dd715e-29889675-sjdts_odh-ai-gateway-infra(ac7c8bb8-9c6e-4b0e-bb84-2e70fc863d96)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dd715e-29889675-sjdts" podUID="ac7c8bb8-9c6e-4b0e-bb84-2e70fc863d96" Oct 30 17:15:21.358969 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:21.358945 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889675-kxmbh" event={"ID":"9988e846-2f00-4c0a-b72e-b2943e79f01b","Type":"ContainerStarted","Data":"d00a09919fb68ead2a1397e5c2481706964fb017f094d5f0c500770996e4e5cf"} Oct 30 17:15:21.361465 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:21.361443 4493 generic.go:356] "Generic (PLEG): container finished" podID="e0665578-a6c7-4b36-ba0f-5144760dce17" containerID="20722bbe4572ddabdf19be37ebf67dcee7efae2e985540a07c63d548459984b0" exitCode=60 Oct 30 17:15:21.361578 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:21.361520 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f94fb5-29889675-6w42p" event={"ID":"e0665578-a6c7-4b36-ba0f-5144760dce17","Type":"ContainerDied","Data":"20722bbe4572ddabdf19be37ebf67dcee7efae2e985540a07c63d548459984b0"} Oct 30 17:15:21.361923 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:21.361894 4493 scope.go:122] "RemoveContainer" containerID="20722bbe4572ddabdf19be37ebf67dcee7efae2e985540a07c63d548459984b0" Oct 30 17:15:21.362746 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:15:21.362209 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-f94fb5-29889675-6w42p_odh-ai-gateway-infra(e0665578-a6c7-4b36-ba0f-5144760dce17)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f94fb5-29889675-6w42p" podUID="e0665578-a6c7-4b36-ba0f-5144760dce17" Oct 30 17:15:21.373673 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:21.373645 4493 scope.go:122] "RemoveContainer" containerID="8135440523f5b61ef4609b0b90b12dd551c89e4796c26df265bacad440db2d0f" Oct 30 17:15:21.386080 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:21.386055 4493 generic.go:356] "Generic (PLEG): container finished" podID="5c296662-aa00-4ee9-90e5-2829d66ce1be" containerID="20b750c07652a27b019b0ad166633efc953c337ae0652996274e6c28ac35eed5" exitCode=60 Oct 30 17:15:21.386209 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:21.386122 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3106d6-29889675-dtkjj" event={"ID":"5c296662-aa00-4ee9-90e5-2829d66ce1be","Type":"ContainerDied","Data":"20b750c07652a27b019b0ad166633efc953c337ae0652996274e6c28ac35eed5"} Oct 30 17:15:21.386559 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:21.386537 4493 scope.go:122] "RemoveContainer" containerID="20b750c07652a27b019b0ad166633efc953c337ae0652996274e6c28ac35eed5" Oct 30 17:15:21.386902 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:15:21.386881 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-3106d6-29889675-dtkjj_odh-ai-gateway-infra(5c296662-aa00-4ee9-90e5-2829d66ce1be)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3106d6-29889675-dtkjj" podUID="5c296662-aa00-4ee9-90e5-2829d66ce1be" Oct 30 17:15:21.396584 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:21.396496 4493 generic.go:356] "Generic (PLEG): container finished" podID="b72976e0-c501-44b4-b42c-5dfa26192e87" containerID="abdd9705b0b1a24d742d1f510d3dc3563417ea22443dabefecf20ba1a37df37f" exitCode=60 Oct 30 17:15:21.396584 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:21.396569 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-13c495-29889675-79wjh" event={"ID":"b72976e0-c501-44b4-b42c-5dfa26192e87","Type":"ContainerDied","Data":"abdd9705b0b1a24d742d1f510d3dc3563417ea22443dabefecf20ba1a37df37f"} Oct 30 17:15:21.396717 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:21.396591 4493 scope.go:122] "RemoveContainer" containerID="72c4a9aaf65528e7c93df69ae468e777be2575663c5145a59244ed0dec5138d4" Oct 30 17:15:21.397008 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:21.396992 4493 scope.go:122] "RemoveContainer" containerID="abdd9705b0b1a24d742d1f510d3dc3563417ea22443dabefecf20ba1a37df37f" Oct 30 17:15:21.397299 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:15:21.397280 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-13c495-29889675-79wjh_odh-ai-gateway-infra(b72976e0-c501-44b4-b42c-5dfa26192e87)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-13c495-29889675-79wjh" podUID="b72976e0-c501-44b4-b42c-5dfa26192e87" Oct 30 17:15:21.399448 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:21.399426 4493 generic.go:356] "Generic (PLEG): container finished" podID="c4086c1e-112f-4518-8ae2-3fa5526f9357" containerID="e467effa1264a238d1ae0195f0b60ac618c76a123cfc4db81aaa5049f9134433" exitCode=60 Oct 30 17:15:21.399542 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:21.399489 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6acc78-29889675-664bj" event={"ID":"c4086c1e-112f-4518-8ae2-3fa5526f9357","Type":"ContainerDied","Data":"e467effa1264a238d1ae0195f0b60ac618c76a123cfc4db81aaa5049f9134433"} Oct 30 17:15:21.399881 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:21.399864 4493 scope.go:122] "RemoveContainer" containerID="e467effa1264a238d1ae0195f0b60ac618c76a123cfc4db81aaa5049f9134433" Oct 30 17:15:21.400172 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:15:21.400152 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-6acc78-29889675-664bj_odh-ai-gateway-infra(c4086c1e-112f-4518-8ae2-3fa5526f9357)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6acc78-29889675-664bj" podUID="c4086c1e-112f-4518-8ae2-3fa5526f9357" Oct 30 17:15:21.401304 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:21.401285 4493 generic.go:356] "Generic (PLEG): container finished" podID="5ff2bc89-4ecd-41d6-86ca-4111cba66bbd" containerID="4fa29b5d7868ce7ae51cfb86fdf611f6731ea5e406ed6f1e8e5e4521891bbf47" exitCode=60 Oct 30 17:15:21.401387 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:21.401352 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-43197b-29889675-mm8kj" event={"ID":"5ff2bc89-4ecd-41d6-86ca-4111cba66bbd","Type":"ContainerDied","Data":"4fa29b5d7868ce7ae51cfb86fdf611f6731ea5e406ed6f1e8e5e4521891bbf47"} Oct 30 17:15:21.401704 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:21.401687 4493 scope.go:122] "RemoveContainer" containerID="4fa29b5d7868ce7ae51cfb86fdf611f6731ea5e406ed6f1e8e5e4521891bbf47" Oct 30 17:15:21.402037 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:15:21.402017 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-43197b-29889675-mm8kj_odh-ai-gateway-infra(5ff2bc89-4ecd-41d6-86ca-4111cba66bbd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-43197b-29889675-mm8kj" podUID="5ff2bc89-4ecd-41d6-86ca-4111cba66bbd" Oct 30 17:15:21.403239 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:21.403219 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5758b7-29889675-hshgf" event={"ID":"68c22006-b1b5-4f15-b504-be63d753819c","Type":"ContainerStarted","Data":"2b7d41806ab0837bc6019e5abde366c5e86f15d6b9f22a0863c851b413bdb001"} Oct 30 17:15:21.409849 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:21.408999 4493 generic.go:356] "Generic (PLEG): container finished" podID="89814242-301f-48ab-a666-b8574e1e8116" containerID="350abc87b73911ff0b51d6e39512908153add30cf811ddede3b234818ae1c235" exitCode=60 Oct 30 17:15:21.409849 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:21.409030 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5c6b3c-29889675-zznjt" event={"ID":"89814242-301f-48ab-a666-b8574e1e8116","Type":"ContainerDied","Data":"350abc87b73911ff0b51d6e39512908153add30cf811ddede3b234818ae1c235"} Oct 30 17:15:21.415754 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:21.415729 4493 scope.go:122] "RemoveContainer" containerID="2c05a57129e4f6cd727a6c9b98f09581a20d11e75b97db527f438f18931fa287" Oct 30 17:15:21.430152 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:21.430116 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-13c495-29889675-79wjh"] Oct 30 17:15:21.505177 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:21.500982 4493 scope.go:122] "RemoveContainer" containerID="b8ba8f34ce3c08e71f356f1929d6903b3d7fffe515309b6726e6f952641b5eb2" Oct 30 17:15:21.624403 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:21.624372 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6acc78-29889675-664bj"] Oct 30 17:15:21.670820 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:21.669427 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cb6251-29889675-tpztt"] Oct 30 17:15:21.685331 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:21.685226 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-43197b-29889675-mm8kj"] Oct 30 17:15:21.734294 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:21.731847 4493 scope.go:122] "RemoveContainer" containerID="82e4eaa41c79586864ec3d9b537aebdd262a9a6686ae8e56bf13382bbeeea4db" Oct 30 17:15:21.761087 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:21.759466 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f94fb5-29889675-6w42p"] Oct 30 17:15:21.819052 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:21.819021 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3106d6-29889675-dtkjj"] Oct 30 17:15:21.830545 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:21.830519 4493 scope.go:122] "RemoveContainer" containerID="117d3121d638e1e0b2ff3421d9f464c76ed248ce02c92088a6a7dd3e8655ad7b" Oct 30 17:15:21.863378 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:21.863340 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889675-nmmn8"] Oct 30 17:15:21.880114 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:21.879983 4493 scope.go:122] "RemoveContainer" containerID="1b5bb947a2471d3b91736074bc9437670af39e771580a8598e7375925ffc9daa" Oct 30 17:15:21.973486 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:21.973122 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-abda73-29889675-8gqs6" Oct 30 17:15:22.134156 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.134129 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25dd3114-a3ba-4976-8f11-611923dc090f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25dd3114-a3ba-4976-8f11-611923dc090f-maas-api-service-ca\") pod \"25dd3114-a3ba-4976-8f11-611923dc090f\" (UID: \"25dd3114-a3ba-4976-8f11-611923dc090f\") " Oct 30 17:15:22.135592 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.134433 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25dd3114-a3ba-4976-8f11-611923dc090f-kube-api-access-q2597\" (UniqueName: \"kubernetes.io/projected/25dd3114-a3ba-4976-8f11-611923dc090f-kube-api-access-q2597\") pod \"25dd3114-a3ba-4976-8f11-611923dc090f\" (UID: \"25dd3114-a3ba-4976-8f11-611923dc090f\") " Oct 30 17:15:22.135592 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.134735 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25dd3114-a3ba-4976-8f11-611923dc090f-maas-api-service-ca" pod "25dd3114-a3ba-4976-8f11-611923dc090f" (UID: "25dd3114-a3ba-4976-8f11-611923dc090f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:22.135592 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.135247 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25dd3114-a3ba-4976-8f11-611923dc090f-maas-api-service-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:15:22.140073 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.140054 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25dd3114-a3ba-4976-8f11-611923dc090f-kube-api-access-q2597" pod "25dd3114-a3ba-4976-8f11-611923dc090f" (UID: "25dd3114-a3ba-4976-8f11-611923dc090f"). InnerVolumeSpecName "kube-api-access-q2597". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:22.213755 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.212324 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d5b798-29889675-hhqqx" Oct 30 17:15:22.220620 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.220593 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8bff6e-29889675-q2zhc" Oct 30 17:15:22.224471 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.224445 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5c6b3c-29889675-zznjt" Oct 30 17:15:22.229225 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.229202 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2589b5-29889675-h7lhw" Oct 30 17:15:22.233525 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.233504 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cb6251-29889675-tpztt" Oct 30 17:15:22.236718 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.236691 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2597\" (UniqueName: \"kubernetes.io/projected/25dd3114-a3ba-4976-8f11-611923dc090f-kube-api-access-q2597\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:15:22.239130 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.239109 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889675-nmmn8" Oct 30 17:15:22.338314 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.337690 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/045dd183-6322-48cb-85df-fa7cc872f8cd-kube-api-access-6dk4t\" (UniqueName: \"kubernetes.io/projected/045dd183-6322-48cb-85df-fa7cc872f8cd-kube-api-access-6dk4t\") pod \"045dd183-6322-48cb-85df-fa7cc872f8cd\" (UID: \"045dd183-6322-48cb-85df-fa7cc872f8cd\") " Oct 30 17:15:22.338633 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.338612 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89814242-301f-48ab-a666-b8574e1e8116-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89814242-301f-48ab-a666-b8574e1e8116-maas-api-service-ca\") pod \"89814242-301f-48ab-a666-b8574e1e8116\" (UID: \"89814242-301f-48ab-a666-b8574e1e8116\") " Oct 30 17:15:22.338808 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.338778 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/632128dd-dd61-4807-8e04-6c809d0e4711-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/632128dd-dd61-4807-8e04-6c809d0e4711-maas-api-service-ca\") pod \"632128dd-dd61-4807-8e04-6c809d0e4711\" (UID: \"632128dd-dd61-4807-8e04-6c809d0e4711\") " Oct 30 17:15:22.338977 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.338962 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53dc2f29-cd5a-4a6b-8e97-cc2d60925ade-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53dc2f29-cd5a-4a6b-8e97-cc2d60925ade-maas-api-service-ca\") pod \"53dc2f29-cd5a-4a6b-8e97-cc2d60925ade\" (UID: \"53dc2f29-cd5a-4a6b-8e97-cc2d60925ade\") " Oct 30 17:15:22.339190 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.339156 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/632128dd-dd61-4807-8e04-6c809d0e4711-kube-api-access-zkc7m\" (UniqueName: \"kubernetes.io/projected/632128dd-dd61-4807-8e04-6c809d0e4711-kube-api-access-zkc7m\") pod \"632128dd-dd61-4807-8e04-6c809d0e4711\" (UID: \"632128dd-dd61-4807-8e04-6c809d0e4711\") " Oct 30 17:15:22.339357 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.339333 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57140245-e2df-4ee9-9344-1de9618511d0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57140245-e2df-4ee9-9344-1de9618511d0-maas-api-service-ca\") pod \"57140245-e2df-4ee9-9344-1de9618511d0\" (UID: \"57140245-e2df-4ee9-9344-1de9618511d0\") " Oct 30 17:15:22.340001 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.339492 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89814242-301f-48ab-a666-b8574e1e8116-kube-api-access-8x7zk\" (UniqueName: \"kubernetes.io/projected/89814242-301f-48ab-a666-b8574e1e8116-kube-api-access-8x7zk\") pod \"89814242-301f-48ab-a666-b8574e1e8116\" (UID: \"89814242-301f-48ab-a666-b8574e1e8116\") " Oct 30 17:15:22.340252 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.340233 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40a120b1-71f5-40ab-8faf-da2052ab0a56-kube-api-access-h2gh4\" (UniqueName: \"kubernetes.io/projected/40a120b1-71f5-40ab-8faf-da2052ab0a56-kube-api-access-h2gh4\") pod \"40a120b1-71f5-40ab-8faf-da2052ab0a56\" (UID: \"40a120b1-71f5-40ab-8faf-da2052ab0a56\") " Oct 30 17:15:22.340609 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.340591 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57140245-e2df-4ee9-9344-1de9618511d0-kube-api-access-6lppp\" (UniqueName: \"kubernetes.io/projected/57140245-e2df-4ee9-9344-1de9618511d0-kube-api-access-6lppp\") pod \"57140245-e2df-4ee9-9344-1de9618511d0\" (UID: \"57140245-e2df-4ee9-9344-1de9618511d0\") " Oct 30 17:15:22.340731 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.340718 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40a120b1-71f5-40ab-8faf-da2052ab0a56-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40a120b1-71f5-40ab-8faf-da2052ab0a56-maas-api-service-ca\") pod \"40a120b1-71f5-40ab-8faf-da2052ab0a56\" (UID: \"40a120b1-71f5-40ab-8faf-da2052ab0a56\") " Oct 30 17:15:22.341186 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.339186 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/632128dd-dd61-4807-8e04-6c809d0e4711-maas-api-service-ca" pod "632128dd-dd61-4807-8e04-6c809d0e4711" (UID: "632128dd-dd61-4807-8e04-6c809d0e4711"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:22.341186 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.339465 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53dc2f29-cd5a-4a6b-8e97-cc2d60925ade-maas-api-service-ca" pod "53dc2f29-cd5a-4a6b-8e97-cc2d60925ade" (UID: "53dc2f29-cd5a-4a6b-8e97-cc2d60925ade"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:22.341186 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.339862 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89814242-301f-48ab-a666-b8574e1e8116-maas-api-service-ca" pod "89814242-301f-48ab-a666-b8574e1e8116" (UID: "89814242-301f-48ab-a666-b8574e1e8116"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:22.341186 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.340665 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57140245-e2df-4ee9-9344-1de9618511d0-maas-api-service-ca" pod "57140245-e2df-4ee9-9344-1de9618511d0" (UID: "57140245-e2df-4ee9-9344-1de9618511d0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:22.341186 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.340955 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/045dd183-6322-48cb-85df-fa7cc872f8cd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/045dd183-6322-48cb-85df-fa7cc872f8cd-maas-api-service-ca\") pod \"045dd183-6322-48cb-85df-fa7cc872f8cd\" (UID: \"045dd183-6322-48cb-85df-fa7cc872f8cd\") " Oct 30 17:15:22.345625 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.341522 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/045dd183-6322-48cb-85df-fa7cc872f8cd-maas-api-service-ca" pod "045dd183-6322-48cb-85df-fa7cc872f8cd" (UID: "045dd183-6322-48cb-85df-fa7cc872f8cd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:22.345625 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.341586 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53dc2f29-cd5a-4a6b-8e97-cc2d60925ade-kube-api-access-lhspq\" (UniqueName: \"kubernetes.io/projected/53dc2f29-cd5a-4a6b-8e97-cc2d60925ade-kube-api-access-lhspq\") pod \"53dc2f29-cd5a-4a6b-8e97-cc2d60925ade\" (UID: \"53dc2f29-cd5a-4a6b-8e97-cc2d60925ade\") " Oct 30 17:15:22.345625 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.342833 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/045dd183-6322-48cb-85df-fa7cc872f8cd-maas-api-service-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:15:22.345625 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.342857 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89814242-301f-48ab-a666-b8574e1e8116-maas-api-service-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:15:22.345625 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.342871 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/632128dd-dd61-4807-8e04-6c809d0e4711-maas-api-service-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:15:22.345625 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.342886 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53dc2f29-cd5a-4a6b-8e97-cc2d60925ade-maas-api-service-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:15:22.345625 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.342901 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57140245-e2df-4ee9-9344-1de9618511d0-maas-api-service-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:15:22.346376 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.345927 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57140245-e2df-4ee9-9344-1de9618511d0-kube-api-access-6lppp" pod "57140245-e2df-4ee9-9344-1de9618511d0" (UID: "57140245-e2df-4ee9-9344-1de9618511d0"). InnerVolumeSpecName "kube-api-access-6lppp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:22.346654 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.346626 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40a120b1-71f5-40ab-8faf-da2052ab0a56-maas-api-service-ca" pod "40a120b1-71f5-40ab-8faf-da2052ab0a56" (UID: "40a120b1-71f5-40ab-8faf-da2052ab0a56"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:22.347195 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.347170 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89814242-301f-48ab-a666-b8574e1e8116-kube-api-access-8x7zk" pod "89814242-301f-48ab-a666-b8574e1e8116" (UID: "89814242-301f-48ab-a666-b8574e1e8116"). InnerVolumeSpecName "kube-api-access-8x7zk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:22.347295 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.347208 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40a120b1-71f5-40ab-8faf-da2052ab0a56-kube-api-access-h2gh4" pod "40a120b1-71f5-40ab-8faf-da2052ab0a56" (UID: "40a120b1-71f5-40ab-8faf-da2052ab0a56"). InnerVolumeSpecName "kube-api-access-h2gh4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:22.347909 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.347875 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/632128dd-dd61-4807-8e04-6c809d0e4711-kube-api-access-zkc7m" pod "632128dd-dd61-4807-8e04-6c809d0e4711" (UID: "632128dd-dd61-4807-8e04-6c809d0e4711"). InnerVolumeSpecName "kube-api-access-zkc7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:22.349451 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.349432 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53dc2f29-cd5a-4a6b-8e97-cc2d60925ade-kube-api-access-lhspq" pod "53dc2f29-cd5a-4a6b-8e97-cc2d60925ade" (UID: "53dc2f29-cd5a-4a6b-8e97-cc2d60925ade"). InnerVolumeSpecName "kube-api-access-lhspq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:22.360293 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.360261 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/045dd183-6322-48cb-85df-fa7cc872f8cd-kube-api-access-6dk4t" pod "045dd183-6322-48cb-85df-fa7cc872f8cd" (UID: "045dd183-6322-48cb-85df-fa7cc872f8cd"). InnerVolumeSpecName "kube-api-access-6dk4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:22.409760 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.409724 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24acb2-29889675-wttj8"] Oct 30 17:15:22.428162 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.428111 4493 generic.go:356] "Generic (PLEG): container finished" podID="7d2cd04d-1a67-45a1-8934-f9d4c9435523" containerID="bb0760cc4c248837f4f83f8d1ea8e19c68f4cff4a36580ed68dc38b944c4311e" exitCode=60 Oct 30 17:15:22.428473 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.428452 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-06e7f6-29889675-m9wc6" event={"ID":"7d2cd04d-1a67-45a1-8934-f9d4c9435523","Type":"ContainerDied","Data":"bb0760cc4c248837f4f83f8d1ea8e19c68f4cff4a36580ed68dc38b944c4311e"} Oct 30 17:15:22.428640 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.428626 4493 scope.go:122] "RemoveContainer" containerID="4fdb02b8970aefc7523bbb3228c96c15183f2b8ee327ee45d9f1ab78783e6bd4" Oct 30 17:15:22.429271 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.429248 4493 scope.go:122] "RemoveContainer" containerID="bb0760cc4c248837f4f83f8d1ea8e19c68f4cff4a36580ed68dc38b944c4311e" Oct 30 17:15:22.430244 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:15:22.429690 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-06e7f6-29889675-m9wc6_odh-ai-gateway-infra(7d2cd04d-1a67-45a1-8934-f9d4c9435523)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-06e7f6-29889675-m9wc6" podUID="7d2cd04d-1a67-45a1-8934-f9d4c9435523" Oct 30 17:15:22.434016 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.433991 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2589b5-29889675-h7lhw" event={"ID":"045dd183-6322-48cb-85df-fa7cc872f8cd","Type":"ContainerDied","Data":"a2bac27902e2208992516933f74af78c8b95e7fdaecc86b75e3be7b6c638364b"} Oct 30 17:15:22.434126 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.434059 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2589b5-29889675-h7lhw" Oct 30 17:15:22.440536 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.440210 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cb6251-29889675-tpztt" Oct 30 17:15:22.440536 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.440509 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cb6251-29889675-tpztt" event={"ID":"53dc2f29-cd5a-4a6b-8e97-cc2d60925ade","Type":"ContainerDied","Data":"afea8a23d9d8b0e0dbec5de6341670189e36813a2a5335afff3d0e00824a3b88"} Oct 30 17:15:22.444430 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.444309 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2gh4\" (UniqueName: \"kubernetes.io/projected/40a120b1-71f5-40ab-8faf-da2052ab0a56-kube-api-access-h2gh4\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:15:22.444430 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.444340 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lppp\" (UniqueName: \"kubernetes.io/projected/57140245-e2df-4ee9-9344-1de9618511d0-kube-api-access-6lppp\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:15:22.444430 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.444354 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40a120b1-71f5-40ab-8faf-da2052ab0a56-maas-api-service-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:15:22.444430 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.444368 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhspq\" (UniqueName: \"kubernetes.io/projected/53dc2f29-cd5a-4a6b-8e97-cc2d60925ade-kube-api-access-lhspq\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:15:22.444430 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.444382 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dk4t\" (UniqueName: \"kubernetes.io/projected/045dd183-6322-48cb-85df-fa7cc872f8cd-kube-api-access-6dk4t\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:15:22.444430 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.444395 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkc7m\" (UniqueName: \"kubernetes.io/projected/632128dd-dd61-4807-8e04-6c809d0e4711-kube-api-access-zkc7m\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:15:22.444430 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.444408 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8x7zk\" (UniqueName: \"kubernetes.io/projected/89814242-301f-48ab-a666-b8574e1e8116-kube-api-access-8x7zk\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:15:22.472530 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.470109 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d5b798-29889675-hhqqx" event={"ID":"632128dd-dd61-4807-8e04-6c809d0e4711","Type":"ContainerDied","Data":"a8930bd07c2f8af414ac1c4ab3f69edf444fe172c7b61b5c247f4ee051e5e24c"} Oct 30 17:15:22.472530 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.470204 4493 scope.go:122] "RemoveContainer" containerID="80cd318e58a1f4c6f897b1d236b8f0ad128df7707453523320c43c400544cdef" Oct 30 17:15:22.472530 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.470310 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d5b798-29889675-hhqqx" Oct 30 17:15:22.493870 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.492833 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5758b7-29889675-hshgf" event={"ID":"68c22006-b1b5-4f15-b504-be63d753819c","Type":"ContainerDied","Data":"2b7d41806ab0837bc6019e5abde366c5e86f15d6b9f22a0863c851b413bdb001"} Oct 30 17:15:22.493870 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.493268 4493 scope.go:122] "RemoveContainer" containerID="2b7d41806ab0837bc6019e5abde366c5e86f15d6b9f22a0863c851b413bdb001" Oct 30 17:15:22.493870 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:15:22.493722 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-5758b7-29889675-hshgf_odh-ai-gateway-infra(68c22006-b1b5-4f15-b504-be63d753819c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5758b7-29889675-hshgf" podUID="68c22006-b1b5-4f15-b504-be63d753819c" Oct 30 17:15:22.495033 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.494587 4493 generic.go:356] "Generic (PLEG): container finished" podID="68c22006-b1b5-4f15-b504-be63d753819c" containerID="2b7d41806ab0837bc6019e5abde366c5e86f15d6b9f22a0863c851b413bdb001" exitCode=60 Oct 30 17:15:22.498564 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.496897 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5c6b3c-29889675-zznjt" event={"ID":"89814242-301f-48ab-a666-b8574e1e8116","Type":"ContainerDied","Data":"f8780fd09918d0ec00f49628fc24d972f1587be7eef0141326344387f431a8d2"} Oct 30 17:15:22.498564 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.496993 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5c6b3c-29889675-zznjt" Oct 30 17:15:22.506853 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.506764 4493 scope.go:122] "RemoveContainer" containerID="3e4a8d2a8c1000091e2b1e2c5e1dff63c9c042ccde82e2c0cd54862a795e9324" Oct 30 17:15:22.516689 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.514762 4493 generic.go:356] "Generic (PLEG): container finished" podID="9988e846-2f00-4c0a-b72e-b2943e79f01b" containerID="d00a09919fb68ead2a1397e5c2481706964fb017f094d5f0c500770996e4e5cf" exitCode=60 Oct 30 17:15:22.516689 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.514868 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889675-kxmbh" event={"ID":"9988e846-2f00-4c0a-b72e-b2943e79f01b","Type":"ContainerDied","Data":"d00a09919fb68ead2a1397e5c2481706964fb017f094d5f0c500770996e4e5cf"} Oct 30 17:15:22.516689 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.515335 4493 scope.go:122] "RemoveContainer" containerID="d00a09919fb68ead2a1397e5c2481706964fb017f094d5f0c500770996e4e5cf" Oct 30 17:15:22.516689 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:15:22.515758 4493 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-18519c-29889675-kxmbh_odh-ai-gateway-infra(9988e846-2f00-4c0a-b72e-b2943e79f01b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889675-kxmbh" podUID="9988e846-2f00-4c0a-b72e-b2943e79f01b" Oct 30 17:15:22.534890 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.521865 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8bff6e-29889675-q2zhc" event={"ID":"40a120b1-71f5-40ab-8faf-da2052ab0a56","Type":"ContainerDied","Data":"69aa4f2f7fc330138ca66573efcb5932b9b1592bdee1f5f3dd98cde4ab05c380"} Oct 30 17:15:22.534890 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.521985 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8bff6e-29889675-q2zhc" Oct 30 17:15:22.561913 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.559435 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-abda73-29889675-8gqs6" event={"ID":"25dd3114-a3ba-4976-8f11-611923dc090f","Type":"ContainerDied","Data":"6059761abf4f1f5f2c4ca6507eaec0a54e6ac1cb8c9752daa2add833da49e916"} Oct 30 17:15:22.561913 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.559482 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-abda73-29889675-8gqs6" Oct 30 17:15:22.585925 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.585007 4493 scope.go:122] "RemoveContainer" containerID="49dac82fea46abca46c9c3d0d276a1d916d48cb6dbf82d44927dfd0e9f451584" Oct 30 17:15:22.602925 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.588164 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889675-nmmn8" event={"ID":"57140245-e2df-4ee9-9344-1de9618511d0","Type":"ContainerDied","Data":"a6d8ec3d8ae273dddb8a9c88cbb2b05f75bc2047959d7dc62d43dc2a7b74937d"} Oct 30 17:15:22.602925 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.588335 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889675-nmmn8" Oct 30 17:15:22.642390 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.634292 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dd715e-29889675-sjdts"] Oct 30 17:15:22.669771 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.669743 4493 scope.go:122] "RemoveContainer" containerID="f1c2ea551a6ba264d1929f69785d7ecb2ec903340cec93139511504c7ffee742" Oct 30 17:15:22.674948 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.674605 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8bff6e-29889675-q2zhc"] Oct 30 17:15:22.682895 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.682861 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8bff6e-29889675-q2zhc"] Oct 30 17:15:22.722092 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.717974 4493 scope.go:122] "RemoveContainer" containerID="350abc87b73911ff0b51d6e39512908153add30cf811ddede3b234818ae1c235" Oct 30 17:15:22.757717 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.757197 4493 scope.go:122] "RemoveContainer" containerID="ddfe1e52d36ba0887edb669191e25a402e1f10e04a94c1d3b9a801b7fbc9e35d" Oct 30 17:15:22.763544 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.763509 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-abda73-29889675-8gqs6"] Oct 30 17:15:22.772061 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.772002 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889675-kxmbh"] Oct 30 17:15:22.779892 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.779627 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-abda73-29889675-8gqs6"] Oct 30 17:15:22.792923 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.792895 4493 scope.go:122] "RemoveContainer" containerID="b8b059f37f256802cf2a02f5652a9b4133ba913eb13f8f6f9082a269b689c074" Oct 30 17:15:22.803848 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.803396 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3106d6-29889675-dtkjj" Oct 30 17:15:22.819072 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.818971 4493 scope.go:122] "RemoveContainer" containerID="d261ba8ed04c10026978853db64fbbdf1e39144fc9847472d69f358cd9f177eb" Oct 30 17:15:22.844059 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.842868 4493 scope.go:122] "RemoveContainer" containerID="56c376ba2995118cc1f7f3f0dd0926842244a4bea27e7089f298f75c7fa8c95e" Oct 30 17:15:22.874118 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.874092 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6acc78-29889675-664bj" Oct 30 17:15:22.879478 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.879410 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-13c495-29889675-79wjh" Oct 30 17:15:22.915358 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.915324 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5758b7-29889675-hshgf"] Oct 30 17:15:22.960032 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.959991 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c296662-aa00-4ee9-90e5-2829d66ce1be-kube-api-access-tcxv2\" (UniqueName: \"kubernetes.io/projected/5c296662-aa00-4ee9-90e5-2829d66ce1be-kube-api-access-tcxv2\") pod \"5c296662-aa00-4ee9-90e5-2829d66ce1be\" (UID: \"5c296662-aa00-4ee9-90e5-2829d66ce1be\") " Oct 30 17:15:22.960228 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.960059 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c296662-aa00-4ee9-90e5-2829d66ce1be-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c296662-aa00-4ee9-90e5-2829d66ce1be-maas-api-service-ca\") pod \"5c296662-aa00-4ee9-90e5-2829d66ce1be\" (UID: \"5c296662-aa00-4ee9-90e5-2829d66ce1be\") " Oct 30 17:15:22.961213 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.961183 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c296662-aa00-4ee9-90e5-2829d66ce1be-maas-api-service-ca" pod "5c296662-aa00-4ee9-90e5-2829d66ce1be" (UID: "5c296662-aa00-4ee9-90e5-2829d66ce1be"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:22.971122 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.971098 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25dd3114-a3ba-4976-8f11-611923dc090f" path="/var/lib/kubelet/pods/25dd3114-a3ba-4976-8f11-611923dc090f/volumes" Oct 30 17:15:22.971817 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.971795 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40a120b1-71f5-40ab-8faf-da2052ab0a56" path="/var/lib/kubelet/pods/40a120b1-71f5-40ab-8faf-da2052ab0a56/volumes" Oct 30 17:15:22.999512 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.999487 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5c6b3c-29889675-zznjt"] Oct 30 17:15:22.999669 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.999526 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5c6b3c-29889675-zznjt"] Oct 30 17:15:22.999669 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.999545 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cb6251-29889675-tpztt"] Oct 30 17:15:22.999669 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:22.999558 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cb6251-29889675-tpztt"] Oct 30 17:15:23.002024 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.001979 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c296662-aa00-4ee9-90e5-2829d66ce1be-kube-api-access-tcxv2" pod "5c296662-aa00-4ee9-90e5-2829d66ce1be" (UID: "5c296662-aa00-4ee9-90e5-2829d66ce1be"). InnerVolumeSpecName "kube-api-access-tcxv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:23.010028 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.009011 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-43197b-29889675-mm8kj" Oct 30 17:15:23.020426 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.020404 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f94fb5-29889675-6w42p" Oct 30 17:15:23.064667 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.064632 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b72976e0-c501-44b4-b42c-5dfa26192e87-kube-api-access-x7wsh\" (UniqueName: \"kubernetes.io/projected/b72976e0-c501-44b4-b42c-5dfa26192e87-kube-api-access-x7wsh\") pod \"b72976e0-c501-44b4-b42c-5dfa26192e87\" (UID: \"b72976e0-c501-44b4-b42c-5dfa26192e87\") " Oct 30 17:15:23.065136 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.065110 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b72976e0-c501-44b4-b42c-5dfa26192e87-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b72976e0-c501-44b4-b42c-5dfa26192e87-maas-api-service-ca\") pod \"b72976e0-c501-44b4-b42c-5dfa26192e87\" (UID: \"b72976e0-c501-44b4-b42c-5dfa26192e87\") " Oct 30 17:15:23.065315 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.065298 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4086c1e-112f-4518-8ae2-3fa5526f9357-kube-api-access-65k76\" (UniqueName: \"kubernetes.io/projected/c4086c1e-112f-4518-8ae2-3fa5526f9357-kube-api-access-65k76\") pod \"c4086c1e-112f-4518-8ae2-3fa5526f9357\" (UID: \"c4086c1e-112f-4518-8ae2-3fa5526f9357\") " Oct 30 17:15:23.065704 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.065683 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4086c1e-112f-4518-8ae2-3fa5526f9357-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4086c1e-112f-4518-8ae2-3fa5526f9357-maas-api-service-ca\") pod \"c4086c1e-112f-4518-8ae2-3fa5526f9357\" (UID: \"c4086c1e-112f-4518-8ae2-3fa5526f9357\") " Oct 30 17:15:23.066260 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.066230 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b72976e0-c501-44b4-b42c-5dfa26192e87-maas-api-service-ca" pod "b72976e0-c501-44b4-b42c-5dfa26192e87" (UID: "b72976e0-c501-44b4-b42c-5dfa26192e87"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:23.066942 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.066919 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b72976e0-c501-44b4-b42c-5dfa26192e87-maas-api-service-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:15:23.067138 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.067124 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcxv2\" (UniqueName: \"kubernetes.io/projected/5c296662-aa00-4ee9-90e5-2829d66ce1be-kube-api-access-tcxv2\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:15:23.067254 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.067241 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c296662-aa00-4ee9-90e5-2829d66ce1be-maas-api-service-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:15:23.067837 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.067275 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4086c1e-112f-4518-8ae2-3fa5526f9357-maas-api-service-ca" pod "c4086c1e-112f-4518-8ae2-3fa5526f9357" (UID: "c4086c1e-112f-4518-8ae2-3fa5526f9357"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:23.069302 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.068945 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4086c1e-112f-4518-8ae2-3fa5526f9357-kube-api-access-65k76" pod "c4086c1e-112f-4518-8ae2-3fa5526f9357" (UID: "c4086c1e-112f-4518-8ae2-3fa5526f9357"). InnerVolumeSpecName "kube-api-access-65k76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:23.069302 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.068984 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b72976e0-c501-44b4-b42c-5dfa26192e87-kube-api-access-x7wsh" pod "b72976e0-c501-44b4-b42c-5dfa26192e87" (UID: "b72976e0-c501-44b4-b42c-5dfa26192e87"). InnerVolumeSpecName "kube-api-access-x7wsh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:23.077576 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.077547 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d5b798-29889675-hhqqx"] Oct 30 17:15:23.082522 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.082494 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d5b798-29889675-hhqqx"] Oct 30 17:15:23.097240 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.097217 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dd715e-29889675-sjdts" Oct 30 17:15:23.105947 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.105925 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24acb2-29889675-wttj8" Oct 30 17:15:23.149593 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.149561 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2589b5-29889675-h7lhw"] Oct 30 17:15:23.154651 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.154626 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2589b5-29889675-h7lhw"] Oct 30 17:15:23.167992 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.167964 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac7c8bb8-9c6e-4b0e-bb84-2e70fc863d96-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac7c8bb8-9c6e-4b0e-bb84-2e70fc863d96-maas-api-service-ca\") pod \"ac7c8bb8-9c6e-4b0e-bb84-2e70fc863d96\" (UID: \"ac7c8bb8-9c6e-4b0e-bb84-2e70fc863d96\") " Oct 30 17:15:23.168147 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.168098 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49589f83-94f8-4e46-808f-cb013c6adfeb-kube-api-access-9gccm\" (UniqueName: \"kubernetes.io/projected/49589f83-94f8-4e46-808f-cb013c6adfeb-kube-api-access-9gccm\") pod \"49589f83-94f8-4e46-808f-cb013c6adfeb\" (UID: \"49589f83-94f8-4e46-808f-cb013c6adfeb\") " Oct 30 17:15:23.168147 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.168134 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ff2bc89-4ecd-41d6-86ca-4111cba66bbd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ff2bc89-4ecd-41d6-86ca-4111cba66bbd-maas-api-service-ca\") pod \"5ff2bc89-4ecd-41d6-86ca-4111cba66bbd\" (UID: \"5ff2bc89-4ecd-41d6-86ca-4111cba66bbd\") " Oct 30 17:15:23.168291 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.168182 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49589f83-94f8-4e46-808f-cb013c6adfeb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49589f83-94f8-4e46-808f-cb013c6adfeb-maas-api-service-ca\") pod \"49589f83-94f8-4e46-808f-cb013c6adfeb\" (UID: \"49589f83-94f8-4e46-808f-cb013c6adfeb\") " Oct 30 17:15:23.168493 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.168399 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac7c8bb8-9c6e-4b0e-bb84-2e70fc863d96-maas-api-service-ca" pod "ac7c8bb8-9c6e-4b0e-bb84-2e70fc863d96" (UID: "ac7c8bb8-9c6e-4b0e-bb84-2e70fc863d96"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:23.168565 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.168493 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ff2bc89-4ecd-41d6-86ca-4111cba66bbd-maas-api-service-ca" pod "5ff2bc89-4ecd-41d6-86ca-4111cba66bbd" (UID: "5ff2bc89-4ecd-41d6-86ca-4111cba66bbd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:23.168565 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.168527 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac7c8bb8-9c6e-4b0e-bb84-2e70fc863d96-kube-api-access-zrs54\" (UniqueName: \"kubernetes.io/projected/ac7c8bb8-9c6e-4b0e-bb84-2e70fc863d96-kube-api-access-zrs54\") pod \"ac7c8bb8-9c6e-4b0e-bb84-2e70fc863d96\" (UID: \"ac7c8bb8-9c6e-4b0e-bb84-2e70fc863d96\") " Oct 30 17:15:23.168565 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.168532 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49589f83-94f8-4e46-808f-cb013c6adfeb-maas-api-service-ca" pod "49589f83-94f8-4e46-808f-cb013c6adfeb" (UID: "49589f83-94f8-4e46-808f-cb013c6adfeb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:23.168735 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.168652 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0665578-a6c7-4b36-ba0f-5144760dce17-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0665578-a6c7-4b36-ba0f-5144760dce17-maas-api-service-ca\") pod \"e0665578-a6c7-4b36-ba0f-5144760dce17\" (UID: \"e0665578-a6c7-4b36-ba0f-5144760dce17\") " Oct 30 17:15:23.168735 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.168686 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ff2bc89-4ecd-41d6-86ca-4111cba66bbd-kube-api-access-zvhrz\" (UniqueName: \"kubernetes.io/projected/5ff2bc89-4ecd-41d6-86ca-4111cba66bbd-kube-api-access-zvhrz\") pod \"5ff2bc89-4ecd-41d6-86ca-4111cba66bbd\" (UID: \"5ff2bc89-4ecd-41d6-86ca-4111cba66bbd\") " Oct 30 17:15:23.168865 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.168763 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0665578-a6c7-4b36-ba0f-5144760dce17-kube-api-access-d97qs\" (UniqueName: \"kubernetes.io/projected/e0665578-a6c7-4b36-ba0f-5144760dce17-kube-api-access-d97qs\") pod \"e0665578-a6c7-4b36-ba0f-5144760dce17\" (UID: \"e0665578-a6c7-4b36-ba0f-5144760dce17\") " Oct 30 17:15:23.169568 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.169495 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0665578-a6c7-4b36-ba0f-5144760dce17-maas-api-service-ca" pod "e0665578-a6c7-4b36-ba0f-5144760dce17" (UID: "e0665578-a6c7-4b36-ba0f-5144760dce17"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:23.169831 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.169807 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4086c1e-112f-4518-8ae2-3fa5526f9357-maas-api-service-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:15:23.169942 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.169837 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac7c8bb8-9c6e-4b0e-bb84-2e70fc863d96-maas-api-service-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:15:23.169942 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.169854 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7wsh\" (UniqueName: \"kubernetes.io/projected/b72976e0-c501-44b4-b42c-5dfa26192e87-kube-api-access-x7wsh\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:15:23.169942 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.169868 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ff2bc89-4ecd-41d6-86ca-4111cba66bbd-maas-api-service-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:15:23.169942 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.169882 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49589f83-94f8-4e46-808f-cb013c6adfeb-maas-api-service-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:15:23.169942 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.169895 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65k76\" (UniqueName: \"kubernetes.io/projected/c4086c1e-112f-4518-8ae2-3fa5526f9357-kube-api-access-65k76\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:15:23.170828 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.170777 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49589f83-94f8-4e46-808f-cb013c6adfeb-kube-api-access-9gccm" pod "49589f83-94f8-4e46-808f-cb013c6adfeb" (UID: "49589f83-94f8-4e46-808f-cb013c6adfeb"). InnerVolumeSpecName "kube-api-access-9gccm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:23.171547 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.171513 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ff2bc89-4ecd-41d6-86ca-4111cba66bbd-kube-api-access-zvhrz" pod "5ff2bc89-4ecd-41d6-86ca-4111cba66bbd" (UID: "5ff2bc89-4ecd-41d6-86ca-4111cba66bbd"). InnerVolumeSpecName "kube-api-access-zvhrz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:23.171730 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.171707 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0665578-a6c7-4b36-ba0f-5144760dce17-kube-api-access-d97qs" pod "e0665578-a6c7-4b36-ba0f-5144760dce17" (UID: "e0665578-a6c7-4b36-ba0f-5144760dce17"). InnerVolumeSpecName "kube-api-access-d97qs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:23.171994 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.171976 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac7c8bb8-9c6e-4b0e-bb84-2e70fc863d96-kube-api-access-zrs54" pod "ac7c8bb8-9c6e-4b0e-bb84-2e70fc863d96" (UID: "ac7c8bb8-9c6e-4b0e-bb84-2e70fc863d96"). InnerVolumeSpecName "kube-api-access-zrs54". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:23.227491 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.227418 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889675-nmmn8"] Oct 30 17:15:23.233047 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.233019 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2d365c-29889675-nmmn8"] Oct 30 17:15:23.271484 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.271444 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrs54\" (UniqueName: \"kubernetes.io/projected/ac7c8bb8-9c6e-4b0e-bb84-2e70fc863d96-kube-api-access-zrs54\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:15:23.271484 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.271486 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0665578-a6c7-4b36-ba0f-5144760dce17-maas-api-service-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:15:23.271754 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.271502 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvhrz\" (UniqueName: \"kubernetes.io/projected/5ff2bc89-4ecd-41d6-86ca-4111cba66bbd-kube-api-access-zvhrz\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:15:23.271754 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.271515 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d97qs\" (UniqueName: \"kubernetes.io/projected/e0665578-a6c7-4b36-ba0f-5144760dce17-kube-api-access-d97qs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:15:23.271754 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.271529 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gccm\" (UniqueName: \"kubernetes.io/projected/49589f83-94f8-4e46-808f-cb013c6adfeb-kube-api-access-9gccm\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:15:23.613626 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.613395 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3106d6-29889675-dtkjj" event={"ID":"5c296662-aa00-4ee9-90e5-2829d66ce1be","Type":"ContainerDied","Data":"2769a2bd72d663fd14194b0f6d3153024d44441192903add7b095bed3f838f7a"} Oct 30 17:15:23.613626 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.613490 4493 scope.go:122] "RemoveContainer" containerID="20b750c07652a27b019b0ad166633efc953c337ae0652996274e6c28ac35eed5" Oct 30 17:15:23.624996 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.624936 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3106d6-29889675-dtkjj" Oct 30 17:15:23.626291 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.626264 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-13c495-29889675-79wjh" event={"ID":"b72976e0-c501-44b4-b42c-5dfa26192e87","Type":"ContainerDied","Data":"80baf4e07396d3da64b406253cbc1f2c5aca71b159b921b1f69f94d8cfb4c6fe"} Oct 30 17:15:23.626447 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.626322 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-06e7f6-29889675-m9wc6"] Oct 30 17:15:23.626447 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.626347 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-13c495-29889675-79wjh" Oct 30 17:15:23.628457 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.628433 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6acc78-29889675-664bj" event={"ID":"c4086c1e-112f-4518-8ae2-3fa5526f9357","Type":"ContainerDied","Data":"49220f40f51b730842615758b0fa20f2e9cd17f4672239c2ef3549f3c92a9ac5"} Oct 30 17:15:23.628720 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.628701 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6acc78-29889675-664bj" Oct 30 17:15:23.631189 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.631153 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-43197b-29889675-mm8kj" Oct 30 17:15:23.631923 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.631151 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-43197b-29889675-mm8kj" event={"ID":"5ff2bc89-4ecd-41d6-86ca-4111cba66bbd","Type":"ContainerDied","Data":"af1c9a919fe9b6872ec410a19db9b2897f40d92b98ca854402a878b02d67944b"} Oct 30 17:15:23.636923 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.636731 4493 scope.go:122] "RemoveContainer" containerID="abdd9705b0b1a24d742d1f510d3dc3563417ea22443dabefecf20ba1a37df37f" Oct 30 17:15:23.649647 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.646193 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24acb2-29889675-wttj8" event={"ID":"49589f83-94f8-4e46-808f-cb013c6adfeb","Type":"ContainerDied","Data":"9e8519c338871447d8b1578d2570b403fcd9b46157cb52e30bc191d8ac7cbee6"} Oct 30 17:15:23.649647 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.646236 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24acb2-29889675-wttj8" Oct 30 17:15:23.696957 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.696177 4493 scope.go:122] "RemoveContainer" containerID="e467effa1264a238d1ae0195f0b60ac618c76a123cfc4db81aaa5049f9134433" Oct 30 17:15:23.699419 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.699387 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dd715e-29889675-sjdts" event={"ID":"ac7c8bb8-9c6e-4b0e-bb84-2e70fc863d96","Type":"ContainerDied","Data":"155b7e3fc2f6d2da48c61bd027aaa1759647f783605103ecc483c2b57d9e6518"} Oct 30 17:15:23.699806 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.699749 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dd715e-29889675-sjdts" Oct 30 17:15:23.713811 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.710049 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3106d6-29889675-dtkjj"] Oct 30 17:15:23.727057 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.721382 4493 scope.go:122] "RemoveContainer" containerID="4fa29b5d7868ce7ae51cfb86fdf611f6731ea5e406ed6f1e8e5e4521891bbf47" Oct 30 17:15:23.727057 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.723267 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f94fb5-29889675-6w42p" event={"ID":"e0665578-a6c7-4b36-ba0f-5144760dce17","Type":"ContainerDied","Data":"286b223fdaa8f36f362c17760172878c07dd1d97fd223bc4f787848283aa8903"} Oct 30 17:15:23.727057 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.723727 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f94fb5-29889675-6w42p" Oct 30 17:15:23.762872 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.762841 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3106d6-29889675-dtkjj"] Oct 30 17:15:23.773522 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.766013 4493 scope.go:122] "RemoveContainer" containerID="617d8bff48447f0d212ca81b3af05fc84072371d723b3861b4587730dff5b171" Oct 30 17:15:23.793880 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.793815 4493 scope.go:122] "RemoveContainer" containerID="b6fb93f9b51e5bbe29b73d564dea34dec564db08297b2c944d9c3c39a13dff98" Oct 30 17:15:23.810683 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.810658 4493 scope.go:122] "RemoveContainer" containerID="20722bbe4572ddabdf19be37ebf67dcee7efae2e985540a07c63d548459984b0" Oct 30 17:15:23.875044 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.874959 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-06e7f6-29889675-m9wc6" Oct 30 17:15:23.884367 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.884343 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-13c495-29889675-79wjh"] Oct 30 17:15:23.889064 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.889031 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-13c495-29889675-79wjh"] Oct 30 17:15:23.895385 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.894677 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d2cd04d-1a67-45a1-8934-f9d4c9435523-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d2cd04d-1a67-45a1-8934-f9d4c9435523-maas-api-service-ca\") pod \"7d2cd04d-1a67-45a1-8934-f9d4c9435523\" (UID: \"7d2cd04d-1a67-45a1-8934-f9d4c9435523\") " Oct 30 17:15:23.895385 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.894997 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d2cd04d-1a67-45a1-8934-f9d4c9435523-kube-api-access-mg6lv\" (UniqueName: \"kubernetes.io/projected/7d2cd04d-1a67-45a1-8934-f9d4c9435523-kube-api-access-mg6lv\") pod \"7d2cd04d-1a67-45a1-8934-f9d4c9435523\" (UID: \"7d2cd04d-1a67-45a1-8934-f9d4c9435523\") " Oct 30 17:15:23.896006 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.895976 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d2cd04d-1a67-45a1-8934-f9d4c9435523-maas-api-service-ca" pod "7d2cd04d-1a67-45a1-8934-f9d4c9435523" (UID: "7d2cd04d-1a67-45a1-8934-f9d4c9435523"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:23.896438 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.896413 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d2cd04d-1a67-45a1-8934-f9d4c9435523-maas-api-service-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:15:23.929808 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.925988 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d2cd04d-1a67-45a1-8934-f9d4c9435523-kube-api-access-mg6lv" pod "7d2cd04d-1a67-45a1-8934-f9d4c9435523" (UID: "7d2cd04d-1a67-45a1-8934-f9d4c9435523"). InnerVolumeSpecName "kube-api-access-mg6lv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:23.941013 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.940950 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24acb2-29889675-wttj8"] Oct 30 17:15:23.948440 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.948412 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-24acb2-29889675-wttj8"] Oct 30 17:15:23.954386 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.954365 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5758b7-29889675-hshgf" Oct 30 17:15:23.970437 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.969554 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6acc78-29889675-664bj"] Oct 30 17:15:23.975706 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.975685 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889675-kxmbh" Oct 30 17:15:23.982853 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.982831 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6acc78-29889675-664bj"] Oct 30 17:15:23.996955 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.996933 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68c22006-b1b5-4f15-b504-be63d753819c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68c22006-b1b5-4f15-b504-be63d753819c-maas-api-service-ca\") pod \"68c22006-b1b5-4f15-b504-be63d753819c\" (UID: \"68c22006-b1b5-4f15-b504-be63d753819c\") " Oct 30 17:15:23.997047 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.996970 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68c22006-b1b5-4f15-b504-be63d753819c-kube-api-access-vhkfk\" (UniqueName: \"kubernetes.io/projected/68c22006-b1b5-4f15-b504-be63d753819c-kube-api-access-vhkfk\") pod \"68c22006-b1b5-4f15-b504-be63d753819c\" (UID: \"68c22006-b1b5-4f15-b504-be63d753819c\") " Oct 30 17:15:23.997086 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.997058 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9988e846-2f00-4c0a-b72e-b2943e79f01b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9988e846-2f00-4c0a-b72e-b2943e79f01b-maas-api-service-ca\") pod \"9988e846-2f00-4c0a-b72e-b2943e79f01b\" (UID: \"9988e846-2f00-4c0a-b72e-b2943e79f01b\") " Oct 30 17:15:23.997201 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.997185 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9988e846-2f00-4c0a-b72e-b2943e79f01b-kube-api-access-7kzfv\" (UniqueName: \"kubernetes.io/projected/9988e846-2f00-4c0a-b72e-b2943e79f01b-kube-api-access-7kzfv\") pod \"9988e846-2f00-4c0a-b72e-b2943e79f01b\" (UID: \"9988e846-2f00-4c0a-b72e-b2943e79f01b\") " Oct 30 17:15:23.997380 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.997358 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68c22006-b1b5-4f15-b504-be63d753819c-maas-api-service-ca" pod "68c22006-b1b5-4f15-b504-be63d753819c" (UID: "68c22006-b1b5-4f15-b504-be63d753819c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:23.997464 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.997403 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9988e846-2f00-4c0a-b72e-b2943e79f01b-maas-api-service-ca" pod "9988e846-2f00-4c0a-b72e-b2943e79f01b" (UID: "9988e846-2f00-4c0a-b72e-b2943e79f01b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:23.997937 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.997906 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9988e846-2f00-4c0a-b72e-b2943e79f01b-maas-api-service-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:15:23.997937 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.997927 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mg6lv\" (UniqueName: \"kubernetes.io/projected/7d2cd04d-1a67-45a1-8934-f9d4c9435523-kube-api-access-mg6lv\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:15:23.998091 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.997956 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68c22006-b1b5-4f15-b504-be63d753819c-maas-api-service-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:15:23.999512 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.999488 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9988e846-2f00-4c0a-b72e-b2943e79f01b-kube-api-access-7kzfv" pod "9988e846-2f00-4c0a-b72e-b2943e79f01b" (UID: "9988e846-2f00-4c0a-b72e-b2943e79f01b"). InnerVolumeSpecName "kube-api-access-7kzfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:23.999652 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:23.999526 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68c22006-b1b5-4f15-b504-be63d753819c-kube-api-access-vhkfk" pod "68c22006-b1b5-4f15-b504-be63d753819c" (UID: "68c22006-b1b5-4f15-b504-be63d753819c"). InnerVolumeSpecName "kube-api-access-vhkfk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:24.006673 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.006653 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-43197b-29889675-mm8kj"] Oct 30 17:15:24.011029 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.011011 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-43197b-29889675-mm8kj"] Oct 30 17:15:24.066811 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.066753 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67607d5a-hnkqk"] Oct 30 17:15:24.068427 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.068399 4493 cpu_manager.go:414] "containerMap: removing container" podUID="57140245-e2df-4ee9-9344-1de9618511d0" containerName="cleanup" Oct 30 17:15:24.068603 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.068590 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57140245-e2df-4ee9-9344-1de9618511d0" containerName="cleanup" Oct 30 17:15:24.068696 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.068687 4493 cpu_manager.go:414] "containerMap: removing container" podUID="ac7c8bb8-9c6e-4b0e-bb84-2e70fc863d96" containerName="cleanup" Oct 30 17:15:24.068766 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.068756 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac7c8bb8-9c6e-4b0e-bb84-2e70fc863d96" containerName="cleanup" Oct 30 17:15:24.068879 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.068868 4493 cpu_manager.go:414] "containerMap: removing container" podUID="57140245-e2df-4ee9-9344-1de9618511d0" containerName="cleanup" Oct 30 17:15:24.068961 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.068942 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57140245-e2df-4ee9-9344-1de9618511d0" containerName="cleanup" Oct 30 17:15:24.069099 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.068973 4493 cpu_manager.go:414] "containerMap: removing container" podUID="7d2cd04d-1a67-45a1-8934-f9d4c9435523" containerName="cleanup" Oct 30 17:15:24.069099 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.068980 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d2cd04d-1a67-45a1-8934-f9d4c9435523" containerName="cleanup" Oct 30 17:15:24.069099 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069009 4493 cpu_manager.go:414] "containerMap: removing container" podUID="045dd183-6322-48cb-85df-fa7cc872f8cd" containerName="cleanup" Oct 30 17:15:24.069099 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069016 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="045dd183-6322-48cb-85df-fa7cc872f8cd" containerName="cleanup" Oct 30 17:15:24.069099 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069025 4493 cpu_manager.go:414] "containerMap: removing container" podUID="43be3031-1e16-4687-97f9-f7af506d98b5" containerName="cleanup" Oct 30 17:15:24.069099 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069032 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43be3031-1e16-4687-97f9-f7af506d98b5" containerName="cleanup" Oct 30 17:15:24.069099 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069054 4493 cpu_manager.go:414] "containerMap: removing container" podUID="e13c30d1-8241-414d-b8fb-a31f796a805a" containerName="cleanup" Oct 30 17:15:24.069099 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069062 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e13c30d1-8241-414d-b8fb-a31f796a805a" containerName="cleanup" Oct 30 17:15:24.069099 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069072 4493 cpu_manager.go:414] "containerMap: removing container" podUID="40a120b1-71f5-40ab-8faf-da2052ab0a56" containerName="cleanup" Oct 30 17:15:24.069099 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069080 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40a120b1-71f5-40ab-8faf-da2052ab0a56" containerName="cleanup" Oct 30 17:15:24.069099 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069092 4493 cpu_manager.go:414] "containerMap: removing container" podUID="68c22006-b1b5-4f15-b504-be63d753819c" containerName="cleanup" Oct 30 17:15:24.069099 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069100 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68c22006-b1b5-4f15-b504-be63d753819c" containerName="cleanup" Oct 30 17:15:24.069677 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069116 4493 cpu_manager.go:414] "containerMap: removing container" podUID="5c296662-aa00-4ee9-90e5-2829d66ce1be" containerName="cleanup" Oct 30 17:15:24.069677 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069124 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c296662-aa00-4ee9-90e5-2829d66ce1be" containerName="cleanup" Oct 30 17:15:24.069677 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069133 4493 cpu_manager.go:414] "containerMap: removing container" podUID="632128dd-dd61-4807-8e04-6c809d0e4711" containerName="cleanup" Oct 30 17:15:24.069677 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069139 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="632128dd-dd61-4807-8e04-6c809d0e4711" containerName="cleanup" Oct 30 17:15:24.069677 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069147 4493 cpu_manager.go:414] "containerMap: removing container" podUID="40a120b1-71f5-40ab-8faf-da2052ab0a56" containerName="cleanup" Oct 30 17:15:24.069677 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069154 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40a120b1-71f5-40ab-8faf-da2052ab0a56" containerName="cleanup" Oct 30 17:15:24.069677 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069176 4493 cpu_manager.go:414] "containerMap: removing container" podUID="b72976e0-c501-44b4-b42c-5dfa26192e87" containerName="cleanup" Oct 30 17:15:24.069677 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069182 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b72976e0-c501-44b4-b42c-5dfa26192e87" containerName="cleanup" Oct 30 17:15:24.069677 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069201 4493 cpu_manager.go:414] "containerMap: removing container" podUID="68c22006-b1b5-4f15-b504-be63d753819c" containerName="cleanup" Oct 30 17:15:24.069677 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069208 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68c22006-b1b5-4f15-b504-be63d753819c" containerName="cleanup" Oct 30 17:15:24.069677 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069226 4493 cpu_manager.go:414] "containerMap: removing container" podUID="89814242-301f-48ab-a666-b8574e1e8116" containerName="cleanup" Oct 30 17:15:24.069677 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069233 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89814242-301f-48ab-a666-b8574e1e8116" containerName="cleanup" Oct 30 17:15:24.069677 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069242 4493 cpu_manager.go:414] "containerMap: removing container" podUID="ac7c8bb8-9c6e-4b0e-bb84-2e70fc863d96" containerName="cleanup" Oct 30 17:15:24.069677 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069249 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac7c8bb8-9c6e-4b0e-bb84-2e70fc863d96" containerName="cleanup" Oct 30 17:15:24.069677 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069262 4493 cpu_manager.go:414] "containerMap: removing container" podUID="c4086c1e-112f-4518-8ae2-3fa5526f9357" containerName="cleanup" Oct 30 17:15:24.069677 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069275 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4086c1e-112f-4518-8ae2-3fa5526f9357" containerName="cleanup" Oct 30 17:15:24.069677 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069303 4493 cpu_manager.go:414] "containerMap: removing container" podUID="89814242-301f-48ab-a666-b8574e1e8116" containerName="cleanup" Oct 30 17:15:24.069677 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069311 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89814242-301f-48ab-a666-b8574e1e8116" containerName="cleanup" Oct 30 17:15:24.069677 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069329 4493 cpu_manager.go:414] "containerMap: removing container" podUID="53dc2f29-cd5a-4a6b-8e97-cc2d60925ade" containerName="cleanup" Oct 30 17:15:24.069677 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069336 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53dc2f29-cd5a-4a6b-8e97-cc2d60925ade" containerName="cleanup" Oct 30 17:15:24.069677 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069345 4493 cpu_manager.go:414] "containerMap: removing container" podUID="49589f83-94f8-4e46-808f-cb013c6adfeb" containerName="cleanup" Oct 30 17:15:24.069677 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069351 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49589f83-94f8-4e46-808f-cb013c6adfeb" containerName="cleanup" Oct 30 17:15:24.069677 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069369 4493 cpu_manager.go:414] "containerMap: removing container" podUID="49589f83-94f8-4e46-808f-cb013c6adfeb" containerName="cleanup" Oct 30 17:15:24.069677 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069376 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49589f83-94f8-4e46-808f-cb013c6adfeb" containerName="cleanup" Oct 30 17:15:24.069677 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069386 4493 cpu_manager.go:414] "containerMap: removing container" podUID="7d2cd04d-1a67-45a1-8934-f9d4c9435523" containerName="cleanup" Oct 30 17:15:24.069677 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069393 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d2cd04d-1a67-45a1-8934-f9d4c9435523" containerName="cleanup" Oct 30 17:15:24.069677 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069412 4493 cpu_manager.go:414] "containerMap: removing container" podUID="e0665578-a6c7-4b36-ba0f-5144760dce17" containerName="cleanup" Oct 30 17:15:24.069677 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069419 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0665578-a6c7-4b36-ba0f-5144760dce17" containerName="cleanup" Oct 30 17:15:24.069677 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069438 4493 cpu_manager.go:414] "containerMap: removing container" podUID="25dd3114-a3ba-4976-8f11-611923dc090f" containerName="cleanup" Oct 30 17:15:24.069677 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069445 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25dd3114-a3ba-4976-8f11-611923dc090f" containerName="cleanup" Oct 30 17:15:24.069677 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069465 4493 cpu_manager.go:414] "containerMap: removing container" podUID="e13c30d1-8241-414d-b8fb-a31f796a805a" containerName="cleanup" Oct 30 17:15:24.069677 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069472 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e13c30d1-8241-414d-b8fb-a31f796a805a" containerName="cleanup" Oct 30 17:15:24.069677 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069490 4493 cpu_manager.go:414] "containerMap: removing container" podUID="9988e846-2f00-4c0a-b72e-b2943e79f01b" containerName="cleanup" Oct 30 17:15:24.069677 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069497 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9988e846-2f00-4c0a-b72e-b2943e79f01b" containerName="cleanup" Oct 30 17:15:24.069677 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069515 4493 cpu_manager.go:414] "containerMap: removing container" podUID="43be3031-1e16-4687-97f9-f7af506d98b5" containerName="cleanup" Oct 30 17:15:24.069677 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069522 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43be3031-1e16-4687-97f9-f7af506d98b5" containerName="cleanup" Oct 30 17:15:24.069677 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069531 4493 cpu_manager.go:414] "containerMap: removing container" podUID="632128dd-dd61-4807-8e04-6c809d0e4711" containerName="cleanup" Oct 30 17:15:24.069677 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069538 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="632128dd-dd61-4807-8e04-6c809d0e4711" containerName="cleanup" Oct 30 17:15:24.069677 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069550 4493 cpu_manager.go:414] "containerMap: removing container" podUID="25dd3114-a3ba-4976-8f11-611923dc090f" containerName="cleanup" Oct 30 17:15:24.069677 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069557 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25dd3114-a3ba-4976-8f11-611923dc090f" containerName="cleanup" Oct 30 17:15:24.069677 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069572 4493 cpu_manager.go:414] "containerMap: removing container" podUID="e13c30d1-8241-414d-b8fb-a31f796a805a" containerName="cleanup" Oct 30 17:15:24.069677 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069579 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e13c30d1-8241-414d-b8fb-a31f796a805a" containerName="cleanup" Oct 30 17:15:24.069677 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069599 4493 cpu_manager.go:414] "containerMap: removing container" podUID="9988e846-2f00-4c0a-b72e-b2943e79f01b" containerName="cleanup" Oct 30 17:15:24.069677 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069606 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9988e846-2f00-4c0a-b72e-b2943e79f01b" containerName="cleanup" Oct 30 17:15:24.069677 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069622 4493 cpu_manager.go:414] "containerMap: removing container" podUID="045dd183-6322-48cb-85df-fa7cc872f8cd" containerName="cleanup" Oct 30 17:15:24.069677 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069629 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="045dd183-6322-48cb-85df-fa7cc872f8cd" containerName="cleanup" Oct 30 17:15:24.069677 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069653 4493 cpu_manager.go:414] "containerMap: removing container" podUID="5c296662-aa00-4ee9-90e5-2829d66ce1be" containerName="cleanup" Oct 30 17:15:24.069677 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069659 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c296662-aa00-4ee9-90e5-2829d66ce1be" containerName="cleanup" Oct 30 17:15:24.069677 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069670 4493 cpu_manager.go:414] "containerMap: removing container" podUID="5ff2bc89-4ecd-41d6-86ca-4111cba66bbd" containerName="cleanup" Oct 30 17:15:24.069677 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069678 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ff2bc89-4ecd-41d6-86ca-4111cba66bbd" containerName="cleanup" Oct 30 17:15:24.069677 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069705 4493 cpu_manager.go:414] "containerMap: removing container" podUID="b72976e0-c501-44b4-b42c-5dfa26192e87" containerName="cleanup" Oct 30 17:15:24.071604 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069717 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b72976e0-c501-44b4-b42c-5dfa26192e87" containerName="cleanup" Oct 30 17:15:24.071604 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069727 4493 cpu_manager.go:414] "containerMap: removing container" podUID="7d2cd04d-1a67-45a1-8934-f9d4c9435523" containerName="cleanup" Oct 30 17:15:24.071604 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069734 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d2cd04d-1a67-45a1-8934-f9d4c9435523" containerName="cleanup" Oct 30 17:15:24.071604 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069765 4493 cpu_manager.go:414] "containerMap: removing container" podUID="5c296662-aa00-4ee9-90e5-2829d66ce1be" containerName="cleanup" Oct 30 17:15:24.071604 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069772 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c296662-aa00-4ee9-90e5-2829d66ce1be" containerName="cleanup" Oct 30 17:15:24.071604 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069820 4493 cpu_manager.go:414] "containerMap: removing container" podUID="045dd183-6322-48cb-85df-fa7cc872f8cd" containerName="cleanup" Oct 30 17:15:24.071604 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069828 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="045dd183-6322-48cb-85df-fa7cc872f8cd" containerName="cleanup" Oct 30 17:15:24.071604 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069845 4493 cpu_manager.go:414] "containerMap: removing container" podUID="9988e846-2f00-4c0a-b72e-b2943e79f01b" containerName="cleanup" Oct 30 17:15:24.071604 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069852 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9988e846-2f00-4c0a-b72e-b2943e79f01b" containerName="cleanup" Oct 30 17:15:24.071604 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069871 4493 cpu_manager.go:414] "containerMap: removing container" podUID="5ff2bc89-4ecd-41d6-86ca-4111cba66bbd" containerName="cleanup" Oct 30 17:15:24.071604 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069878 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ff2bc89-4ecd-41d6-86ca-4111cba66bbd" containerName="cleanup" Oct 30 17:15:24.071604 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069895 4493 cpu_manager.go:414] "containerMap: removing container" podUID="53dc2f29-cd5a-4a6b-8e97-cc2d60925ade" containerName="cleanup" Oct 30 17:15:24.071604 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069909 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53dc2f29-cd5a-4a6b-8e97-cc2d60925ade" containerName="cleanup" Oct 30 17:15:24.071604 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069928 4493 cpu_manager.go:414] "containerMap: removing container" podUID="5ff2bc89-4ecd-41d6-86ca-4111cba66bbd" containerName="cleanup" Oct 30 17:15:24.071604 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069936 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ff2bc89-4ecd-41d6-86ca-4111cba66bbd" containerName="cleanup" Oct 30 17:15:24.071604 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069952 4493 cpu_manager.go:414] "containerMap: removing container" podUID="e0665578-a6c7-4b36-ba0f-5144760dce17" containerName="cleanup" Oct 30 17:15:24.071604 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069959 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0665578-a6c7-4b36-ba0f-5144760dce17" containerName="cleanup" Oct 30 17:15:24.071604 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069973 4493 cpu_manager.go:414] "containerMap: removing container" podUID="c4086c1e-112f-4518-8ae2-3fa5526f9357" containerName="cleanup" Oct 30 17:15:24.071604 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069979 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4086c1e-112f-4518-8ae2-3fa5526f9357" containerName="cleanup" Oct 30 17:15:24.071604 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.069996 4493 cpu_manager.go:414] "containerMap: removing container" podUID="53dc2f29-cd5a-4a6b-8e97-cc2d60925ade" containerName="cleanup" Oct 30 17:15:24.071604 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.070004 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53dc2f29-cd5a-4a6b-8e97-cc2d60925ade" containerName="cleanup" Oct 30 17:15:24.071604 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.070283 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="b72976e0-c501-44b4-b42c-5dfa26192e87" containerName="cleanup" Oct 30 17:15:24.071604 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.070304 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d2cd04d-1a67-45a1-8934-f9d4c9435523" containerName="cleanup" Oct 30 17:15:24.071604 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.070315 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="89814242-301f-48ab-a666-b8574e1e8116" containerName="cleanup" Oct 30 17:15:24.071604 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.070327 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c296662-aa00-4ee9-90e5-2829d66ce1be" containerName="cleanup" Oct 30 17:15:24.071604 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.070344 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="045dd183-6322-48cb-85df-fa7cc872f8cd" containerName="cleanup" Oct 30 17:15:24.071604 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.070357 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="9988e846-2f00-4c0a-b72e-b2943e79f01b" containerName="cleanup" Oct 30 17:15:24.071604 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.070368 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4086c1e-112f-4518-8ae2-3fa5526f9357" containerName="cleanup" Oct 30 17:15:24.071604 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.070373 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ff2bc89-4ecd-41d6-86ca-4111cba66bbd" containerName="cleanup" Oct 30 17:15:24.071604 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.070383 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="53dc2f29-cd5a-4a6b-8e97-cc2d60925ade" containerName="cleanup" Oct 30 17:15:24.071604 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.070389 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="40a120b1-71f5-40ab-8faf-da2052ab0a56" containerName="cleanup" Oct 30 17:15:24.071604 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.070394 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ff2bc89-4ecd-41d6-86ca-4111cba66bbd" containerName="cleanup" Oct 30 17:15:24.071604 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.070403 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="49589f83-94f8-4e46-808f-cb013c6adfeb" containerName="cleanup" Oct 30 17:15:24.071604 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.070408 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0665578-a6c7-4b36-ba0f-5144760dce17" containerName="cleanup" Oct 30 17:15:24.071604 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.070419 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4086c1e-112f-4518-8ae2-3fa5526f9357" containerName="cleanup" Oct 30 17:15:24.071604 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.070425 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="68c22006-b1b5-4f15-b504-be63d753819c" containerName="cleanup" Oct 30 17:15:24.071604 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.070436 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="53dc2f29-cd5a-4a6b-8e97-cc2d60925ade" containerName="cleanup" Oct 30 17:15:24.071604 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.070451 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="25dd3114-a3ba-4976-8f11-611923dc090f" containerName="cleanup" Oct 30 17:15:24.071604 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.070456 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="632128dd-dd61-4807-8e04-6c809d0e4711" containerName="cleanup" Oct 30 17:15:24.071604 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.070466 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="57140245-e2df-4ee9-9344-1de9618511d0" containerName="cleanup" Oct 30 17:15:24.071604 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.070472 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac7c8bb8-9c6e-4b0e-bb84-2e70fc863d96" containerName="cleanup" Oct 30 17:15:24.071604 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.070480 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="57140245-e2df-4ee9-9344-1de9618511d0" containerName="cleanup" Oct 30 17:15:24.071604 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.070489 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="045dd183-6322-48cb-85df-fa7cc872f8cd" containerName="cleanup" Oct 30 17:15:24.071604 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.070493 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="43be3031-1e16-4687-97f9-f7af506d98b5" containerName="cleanup" Oct 30 17:15:24.071604 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.070499 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="e13c30d1-8241-414d-b8fb-a31f796a805a" containerName="cleanup" Oct 30 17:15:24.071604 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.070508 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="68c22006-b1b5-4f15-b504-be63d753819c" containerName="cleanup" Oct 30 17:15:24.071604 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.070518 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="632128dd-dd61-4807-8e04-6c809d0e4711" containerName="cleanup" Oct 30 17:15:24.071604 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.070528 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="40a120b1-71f5-40ab-8faf-da2052ab0a56" containerName="cleanup" Oct 30 17:15:24.071604 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.070533 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="b72976e0-c501-44b4-b42c-5dfa26192e87" containerName="cleanup" Oct 30 17:15:24.071604 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.070539 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="68c22006-b1b5-4f15-b504-be63d753819c" containerName="cleanup" Oct 30 17:15:24.071604 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.070549 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac7c8bb8-9c6e-4b0e-bb84-2e70fc863d96" containerName="cleanup" Oct 30 17:15:24.071604 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.070556 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="89814242-301f-48ab-a666-b8574e1e8116" containerName="cleanup" Oct 30 17:15:24.071604 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.070566 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="49589f83-94f8-4e46-808f-cb013c6adfeb" containerName="cleanup" Oct 30 17:15:24.071604 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.070572 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="49589f83-94f8-4e46-808f-cb013c6adfeb" containerName="cleanup" Oct 30 17:15:24.071604 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.070581 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d2cd04d-1a67-45a1-8934-f9d4c9435523" containerName="cleanup" Oct 30 17:15:24.071604 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.070586 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0665578-a6c7-4b36-ba0f-5144760dce17" containerName="cleanup" Oct 30 17:15:24.071604 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.070592 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="43be3031-1e16-4687-97f9-f7af506d98b5" containerName="cleanup" Oct 30 17:15:24.071604 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.070597 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="25dd3114-a3ba-4976-8f11-611923dc090f" containerName="cleanup" Oct 30 17:15:24.071604 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.070606 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="e13c30d1-8241-414d-b8fb-a31f796a805a" containerName="cleanup" Oct 30 17:15:24.071604 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.070612 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="9988e846-2f00-4c0a-b72e-b2943e79f01b" containerName="cleanup" Oct 30 17:15:24.071604 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.070620 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="57140245-e2df-4ee9-9344-1de9618511d0" containerName="cleanup" Oct 30 17:15:24.071604 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.070625 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c296662-aa00-4ee9-90e5-2829d66ce1be" containerName="cleanup" Oct 30 17:15:24.075024 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.075003 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67607d5a-hnkqk" Oct 30 17:15:24.084897 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.084875 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67607d5a-hnkqk"] Oct 30 17:15:24.090363 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.090334 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:15:24.090927 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.090893 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f94fb5-29889675-6w42p"] Oct 30 17:15:24.097254 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.097230 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-f94fb5-29889675-6w42p"] Oct 30 17:15:24.099612 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.099582 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05b43e44-eef6-4fd8-99d3-8ffebc0c7370-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67607d5a-hnkqk\" (UID: \"05b43e44-eef6-4fd8-99d3-8ffebc0c7370\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67607d5a-hnkqk" Oct 30 17:15:24.099893 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.099869 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfcfq\" (UniqueName: \"kubernetes.io/projected/05b43e44-eef6-4fd8-99d3-8ffebc0c7370-kube-api-access-nfcfq\") pod \"maas-api-revoke-sub-67607d5a-hnkqk\" (UID: \"05b43e44-eef6-4fd8-99d3-8ffebc0c7370\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67607d5a-hnkqk" Oct 30 17:15:24.100052 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.100033 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05b43e44-eef6-4fd8-99d3-8ffebc0c7370-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67607d5a-hnkqk\" (UID: \"05b43e44-eef6-4fd8-99d3-8ffebc0c7370\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67607d5a-hnkqk" Oct 30 17:15:24.100265 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.100245 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhkfk\" (UniqueName: \"kubernetes.io/projected/68c22006-b1b5-4f15-b504-be63d753819c-kube-api-access-vhkfk\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:15:24.100335 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.100272 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kzfv\" (UniqueName: \"kubernetes.io/projected/9988e846-2f00-4c0a-b72e-b2943e79f01b-kube-api-access-7kzfv\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:15:24.129936 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.129869 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dd715e-29889675-sjdts"] Oct 30 17:15:24.136030 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.136012 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-dd715e-29889675-sjdts"] Oct 30 17:15:24.201430 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.201399 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05b43e44-eef6-4fd8-99d3-8ffebc0c7370-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67607d5a-hnkqk\" (UID: \"05b43e44-eef6-4fd8-99d3-8ffebc0c7370\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67607d5a-hnkqk" Oct 30 17:15:24.201601 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.201553 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfcfq\" (UniqueName: \"kubernetes.io/projected/05b43e44-eef6-4fd8-99d3-8ffebc0c7370-kube-api-access-nfcfq\") pod \"maas-api-revoke-sub-67607d5a-hnkqk\" (UID: \"05b43e44-eef6-4fd8-99d3-8ffebc0c7370\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67607d5a-hnkqk" Oct 30 17:15:24.201688 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.201630 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05b43e44-eef6-4fd8-99d3-8ffebc0c7370-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67607d5a-hnkqk\" (UID: \"05b43e44-eef6-4fd8-99d3-8ffebc0c7370\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67607d5a-hnkqk" Oct 30 17:15:24.202318 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.202296 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05b43e44-eef6-4fd8-99d3-8ffebc0c7370-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67607d5a-hnkqk\" (UID: \"05b43e44-eef6-4fd8-99d3-8ffebc0c7370\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67607d5a-hnkqk" Oct 30 17:15:24.204112 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.204088 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05b43e44-eef6-4fd8-99d3-8ffebc0c7370-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67607d5a-hnkqk\" (UID: \"05b43e44-eef6-4fd8-99d3-8ffebc0c7370\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67607d5a-hnkqk" Oct 30 17:15:24.225659 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.225631 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfcfq\" (UniqueName: \"kubernetes.io/projected/05b43e44-eef6-4fd8-99d3-8ffebc0c7370-kube-api-access-nfcfq\") pod \"maas-api-revoke-sub-67607d5a-hnkqk\" (UID: \"05b43e44-eef6-4fd8-99d3-8ffebc0c7370\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67607d5a-hnkqk" Oct 30 17:15:24.393762 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.393733 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67607d5a-hnkqk" Oct 30 17:15:24.703616 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:15:24.703589 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05b43e44_eef6_4fd8_99d3_8ffebc0c7370.slice/crio-4056d3c784523d3ce678148d5b7a682026798bc4fb7095eaf55971daf335e0e1 WatchSource:0}: Error finding container 4056d3c784523d3ce678148d5b7a682026798bc4fb7095eaf55971daf335e0e1: Status 404 returned error can't find the container with id 4056d3c784523d3ce678148d5b7a682026798bc4fb7095eaf55971daf335e0e1 Oct 30 17:15:24.704190 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.704164 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67607d5a-hnkqk"] Oct 30 17:15:24.789644 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.789613 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-06e7f6-29889675-m9wc6" event={"ID":"7d2cd04d-1a67-45a1-8934-f9d4c9435523","Type":"ContainerDied","Data":"7979981d2aa15242228c3746654b0ba39d1e39e416f588e4c5be3a8909d33da8"} Oct 30 17:15:24.789778 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.789623 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-06e7f6-29889675-m9wc6" Oct 30 17:15:24.789778 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.789669 4493 scope.go:122] "RemoveContainer" containerID="bb0760cc4c248837f4f83f8d1ea8e19c68f4cff4a36580ed68dc38b944c4311e" Oct 30 17:15:24.792401 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.792374 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67607d5a-hnkqk" event={"ID":"05b43e44-eef6-4fd8-99d3-8ffebc0c7370","Type":"ContainerStarted","Data":"4056d3c784523d3ce678148d5b7a682026798bc4fb7095eaf55971daf335e0e1"} Oct 30 17:15:24.798116 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.798096 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5758b7-29889675-hshgf" Oct 30 17:15:24.798308 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.798107 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5758b7-29889675-hshgf" event={"ID":"68c22006-b1b5-4f15-b504-be63d753819c","Type":"ContainerDied","Data":"f58e55958ca3cb4a9ef60cce1c455976558cfbead0006bcb347f963d4dd75a4e"} Oct 30 17:15:24.802337 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.802313 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889675-kxmbh" Oct 30 17:15:24.802435 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.802315 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889675-kxmbh" event={"ID":"9988e846-2f00-4c0a-b72e-b2943e79f01b","Type":"ContainerDied","Data":"2bc4d6d3e595e9fdea11817fb7f184e02732341271cf5c6401125fc2d98ff2e9"} Oct 30 17:15:24.807567 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.807543 4493 scope.go:122] "RemoveContainer" containerID="2b7d41806ab0837bc6019e5abde366c5e86f15d6b9f22a0863c851b413bdb001" Oct 30 17:15:24.823601 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.823580 4493 scope.go:122] "RemoveContainer" containerID="d00a09919fb68ead2a1397e5c2481706964fb017f094d5f0c500770996e4e5cf" Oct 30 17:15:24.879136 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.879107 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-06e7f6-29889675-m9wc6"] Oct 30 17:15:24.884738 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.884714 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-06e7f6-29889675-m9wc6"] Oct 30 17:15:24.916179 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.916152 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5758b7-29889675-hshgf"] Oct 30 17:15:24.920439 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.920417 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5758b7-29889675-hshgf"] Oct 30 17:15:24.934337 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.934311 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="045dd183-6322-48cb-85df-fa7cc872f8cd" path="/var/lib/kubelet/pods/045dd183-6322-48cb-85df-fa7cc872f8cd/volumes" Oct 30 17:15:24.934762 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.934748 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49589f83-94f8-4e46-808f-cb013c6adfeb" path="/var/lib/kubelet/pods/49589f83-94f8-4e46-808f-cb013c6adfeb/volumes" Oct 30 17:15:24.935200 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.935186 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53dc2f29-cd5a-4a6b-8e97-cc2d60925ade" path="/var/lib/kubelet/pods/53dc2f29-cd5a-4a6b-8e97-cc2d60925ade/volumes" Oct 30 17:15:24.935887 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.935872 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57140245-e2df-4ee9-9344-1de9618511d0" path="/var/lib/kubelet/pods/57140245-e2df-4ee9-9344-1de9618511d0/volumes" Oct 30 17:15:24.936262 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.936250 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c296662-aa00-4ee9-90e5-2829d66ce1be" path="/var/lib/kubelet/pods/5c296662-aa00-4ee9-90e5-2829d66ce1be/volumes" Oct 30 17:15:24.936626 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.936614 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ff2bc89-4ecd-41d6-86ca-4111cba66bbd" path="/var/lib/kubelet/pods/5ff2bc89-4ecd-41d6-86ca-4111cba66bbd/volumes" Oct 30 17:15:24.937208 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.937192 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="632128dd-dd61-4807-8e04-6c809d0e4711" path="/var/lib/kubelet/pods/632128dd-dd61-4807-8e04-6c809d0e4711/volumes" Oct 30 17:15:24.937583 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.937570 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68c22006-b1b5-4f15-b504-be63d753819c" path="/var/lib/kubelet/pods/68c22006-b1b5-4f15-b504-be63d753819c/volumes" Oct 30 17:15:24.938010 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.937997 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d2cd04d-1a67-45a1-8934-f9d4c9435523" path="/var/lib/kubelet/pods/7d2cd04d-1a67-45a1-8934-f9d4c9435523/volumes" Oct 30 17:15:24.938662 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.938647 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89814242-301f-48ab-a666-b8574e1e8116" path="/var/lib/kubelet/pods/89814242-301f-48ab-a666-b8574e1e8116/volumes" Oct 30 17:15:24.939074 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.939060 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac7c8bb8-9c6e-4b0e-bb84-2e70fc863d96" path="/var/lib/kubelet/pods/ac7c8bb8-9c6e-4b0e-bb84-2e70fc863d96/volumes" Oct 30 17:15:24.939438 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.939425 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b72976e0-c501-44b4-b42c-5dfa26192e87" path="/var/lib/kubelet/pods/b72976e0-c501-44b4-b42c-5dfa26192e87/volumes" Oct 30 17:15:24.940111 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.940093 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4086c1e-112f-4518-8ae2-3fa5526f9357" path="/var/lib/kubelet/pods/c4086c1e-112f-4518-8ae2-3fa5526f9357/volumes" Oct 30 17:15:24.940535 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.940516 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0665578-a6c7-4b36-ba0f-5144760dce17" path="/var/lib/kubelet/pods/e0665578-a6c7-4b36-ba0f-5144760dce17/volumes" Oct 30 17:15:24.954187 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.954124 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889675-kxmbh"] Oct 30 17:15:24.958292 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:24.958270 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-18519c-29889675-kxmbh"] Oct 30 17:15:25.823412 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:25.823375 4493 generic.go:356] "Generic (PLEG): container finished" podID="05b43e44-eef6-4fd8-99d3-8ffebc0c7370" containerID="7da8f2e9769248f89ce9b3553929211b3d55be421968d6dc744c421b35e28df2" exitCode=0 Oct 30 17:15:25.823953 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:25.823457 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67607d5a-hnkqk" event={"ID":"05b43e44-eef6-4fd8-99d3-8ffebc0c7370","Type":"ContainerDied","Data":"7da8f2e9769248f89ce9b3553929211b3d55be421968d6dc744c421b35e28df2"} Oct 30 17:15:27.011865 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:27.000101 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9988e846-2f00-4c0a-b72e-b2943e79f01b" path="/var/lib/kubelet/pods/9988e846-2f00-4c0a-b72e-b2943e79f01b/volumes" Oct 30 17:15:27.085266 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:27.085213 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67607d5a-hnkqk" Oct 30 17:15:27.152986 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:27.152943 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05b43e44-eef6-4fd8-99d3-8ffebc0c7370-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05b43e44-eef6-4fd8-99d3-8ffebc0c7370-maas-api-xks-ca\") pod \"05b43e44-eef6-4fd8-99d3-8ffebc0c7370\" (UID: \"05b43e44-eef6-4fd8-99d3-8ffebc0c7370\") " Oct 30 17:15:27.153196 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:27.153126 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05b43e44-eef6-4fd8-99d3-8ffebc0c7370-kube-api-access-nfcfq\" (UniqueName: \"kubernetes.io/projected/05b43e44-eef6-4fd8-99d3-8ffebc0c7370-kube-api-access-nfcfq\") pod \"05b43e44-eef6-4fd8-99d3-8ffebc0c7370\" (UID: \"05b43e44-eef6-4fd8-99d3-8ffebc0c7370\") " Oct 30 17:15:27.153280 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:27.153193 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05b43e44-eef6-4fd8-99d3-8ffebc0c7370-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05b43e44-eef6-4fd8-99d3-8ffebc0c7370-maas-api-openshift-ca\") pod \"05b43e44-eef6-4fd8-99d3-8ffebc0c7370\" (UID: \"05b43e44-eef6-4fd8-99d3-8ffebc0c7370\") " Oct 30 17:15:27.154238 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:27.153827 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05b43e44-eef6-4fd8-99d3-8ffebc0c7370-maas-api-openshift-ca" pod "05b43e44-eef6-4fd8-99d3-8ffebc0c7370" (UID: "05b43e44-eef6-4fd8-99d3-8ffebc0c7370"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:27.154238 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:27.154194 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05b43e44-eef6-4fd8-99d3-8ffebc0c7370-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:15:27.156119 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:27.156094 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05b43e44-eef6-4fd8-99d3-8ffebc0c7370-kube-api-access-nfcfq" pod "05b43e44-eef6-4fd8-99d3-8ffebc0c7370" (UID: "05b43e44-eef6-4fd8-99d3-8ffebc0c7370"). InnerVolumeSpecName "kube-api-access-nfcfq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:27.156223 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:27.156194 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05b43e44-eef6-4fd8-99d3-8ffebc0c7370-maas-api-xks-ca" pod "05b43e44-eef6-4fd8-99d3-8ffebc0c7370" (UID: "05b43e44-eef6-4fd8-99d3-8ffebc0c7370"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:15:27.255711 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:27.255673 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05b43e44-eef6-4fd8-99d3-8ffebc0c7370-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:15:27.255711 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:27.255708 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfcfq\" (UniqueName: \"kubernetes.io/projected/05b43e44-eef6-4fd8-99d3-8ffebc0c7370-kube-api-access-nfcfq\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:15:27.844059 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:27.844031 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67607d5a-hnkqk" Oct 30 17:15:27.844251 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:27.844061 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67607d5a-hnkqk" event={"ID":"05b43e44-eef6-4fd8-99d3-8ffebc0c7370","Type":"ContainerDied","Data":"4056d3c784523d3ce678148d5b7a682026798bc4fb7095eaf55971daf335e0e1"} Oct 30 17:15:27.844251 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:27.844115 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4056d3c784523d3ce678148d5b7a682026798bc4fb7095eaf55971daf335e0e1" Oct 30 17:15:30.342945 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:15:30.342902 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 17:15:36.670461 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:36.670422 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71a240d6-npfcw"] Oct 30 17:15:36.671339 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:36.671319 4493 cpu_manager.go:414] "containerMap: removing container" podUID="40a120b1-71f5-40ab-8faf-da2052ab0a56" containerName="cleanup" Oct 30 17:15:36.671413 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:36.671340 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40a120b1-71f5-40ab-8faf-da2052ab0a56" containerName="cleanup" Oct 30 17:15:36.671413 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:36.671355 4493 cpu_manager.go:414] "containerMap: removing container" podUID="68c22006-b1b5-4f15-b504-be63d753819c" containerName="cleanup" Oct 30 17:15:36.671413 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:36.671359 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68c22006-b1b5-4f15-b504-be63d753819c" containerName="cleanup" Oct 30 17:15:36.671413 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:36.671376 4493 cpu_manager.go:414] "containerMap: removing container" podUID="89814242-301f-48ab-a666-b8574e1e8116" containerName="cleanup" Oct 30 17:15:36.671413 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:36.671381 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89814242-301f-48ab-a666-b8574e1e8116" containerName="cleanup" Oct 30 17:15:36.671413 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:36.671388 4493 cpu_manager.go:414] "containerMap: removing container" podUID="49589f83-94f8-4e46-808f-cb013c6adfeb" containerName="cleanup" Oct 30 17:15:36.671413 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:36.671393 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49589f83-94f8-4e46-808f-cb013c6adfeb" containerName="cleanup" Oct 30 17:15:36.671413 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:36.671405 4493 cpu_manager.go:414] "containerMap: removing container" podUID="43be3031-1e16-4687-97f9-f7af506d98b5" containerName="cleanup" Oct 30 17:15:36.671413 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:36.671409 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43be3031-1e16-4687-97f9-f7af506d98b5" containerName="cleanup" Oct 30 17:15:36.671413 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:36.671415 4493 cpu_manager.go:414] "containerMap: removing container" podUID="632128dd-dd61-4807-8e04-6c809d0e4711" containerName="cleanup" Oct 30 17:15:36.671713 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:36.671420 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="632128dd-dd61-4807-8e04-6c809d0e4711" containerName="cleanup" Oct 30 17:15:36.671713 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:36.671428 4493 cpu_manager.go:414] "containerMap: removing container" podUID="25dd3114-a3ba-4976-8f11-611923dc090f" containerName="cleanup" Oct 30 17:15:36.671713 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:36.671433 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25dd3114-a3ba-4976-8f11-611923dc090f" containerName="cleanup" Oct 30 17:15:36.671713 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:36.671449 4493 cpu_manager.go:414] "containerMap: removing container" podUID="b72976e0-c501-44b4-b42c-5dfa26192e87" containerName="cleanup" Oct 30 17:15:36.671713 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:36.671453 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b72976e0-c501-44b4-b42c-5dfa26192e87" containerName="cleanup" Oct 30 17:15:36.671713 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:36.671463 4493 cpu_manager.go:414] "containerMap: removing container" podUID="05b43e44-eef6-4fd8-99d3-8ffebc0c7370" containerName="revoke-keys" Oct 30 17:15:36.671713 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:36.671468 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05b43e44-eef6-4fd8-99d3-8ffebc0c7370" containerName="revoke-keys" Oct 30 17:15:36.671713 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:36.671481 4493 cpu_manager.go:414] "containerMap: removing container" podUID="e0665578-a6c7-4b36-ba0f-5144760dce17" containerName="cleanup" Oct 30 17:15:36.671713 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:36.671485 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0665578-a6c7-4b36-ba0f-5144760dce17" containerName="cleanup" Oct 30 17:15:36.671713 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:36.671492 4493 cpu_manager.go:414] "containerMap: removing container" podUID="c4086c1e-112f-4518-8ae2-3fa5526f9357" containerName="cleanup" Oct 30 17:15:36.671713 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:36.671496 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4086c1e-112f-4518-8ae2-3fa5526f9357" containerName="cleanup" Oct 30 17:15:36.671713 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:36.671507 4493 cpu_manager.go:414] "containerMap: removing container" podUID="ac7c8bb8-9c6e-4b0e-bb84-2e70fc863d96" containerName="cleanup" Oct 30 17:15:36.671713 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:36.671512 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac7c8bb8-9c6e-4b0e-bb84-2e70fc863d96" containerName="cleanup" Oct 30 17:15:36.671713 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:36.671519 4493 cpu_manager.go:414] "containerMap: removing container" podUID="57140245-e2df-4ee9-9344-1de9618511d0" containerName="cleanup" Oct 30 17:15:36.671713 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:36.671523 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57140245-e2df-4ee9-9344-1de9618511d0" containerName="cleanup" Oct 30 17:15:36.671713 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:36.671619 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="43be3031-1e16-4687-97f9-f7af506d98b5" containerName="cleanup" Oct 30 17:15:36.671713 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:36.671630 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="e13c30d1-8241-414d-b8fb-a31f796a805a" containerName="cleanup" Oct 30 17:15:36.671713 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:36.671637 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="632128dd-dd61-4807-8e04-6c809d0e4711" containerName="cleanup" Oct 30 17:15:36.671713 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:36.671643 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="40a120b1-71f5-40ab-8faf-da2052ab0a56" containerName="cleanup" Oct 30 17:15:36.671713 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:36.671649 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac7c8bb8-9c6e-4b0e-bb84-2e70fc863d96" containerName="cleanup" Oct 30 17:15:36.671713 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:36.671657 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d2cd04d-1a67-45a1-8934-f9d4c9435523" containerName="cleanup" Oct 30 17:15:36.671713 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:36.671662 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0665578-a6c7-4b36-ba0f-5144760dce17" containerName="cleanup" Oct 30 17:15:36.671713 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:36.671670 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="b72976e0-c501-44b4-b42c-5dfa26192e87" containerName="cleanup" Oct 30 17:15:36.671713 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:36.671676 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="89814242-301f-48ab-a666-b8574e1e8116" containerName="cleanup" Oct 30 17:15:36.671713 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:36.671681 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c296662-aa00-4ee9-90e5-2829d66ce1be" containerName="cleanup" Oct 30 17:15:36.671713 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:36.671686 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="05b43e44-eef6-4fd8-99d3-8ffebc0c7370" containerName="revoke-keys" Oct 30 17:15:36.671713 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:36.671691 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="045dd183-6322-48cb-85df-fa7cc872f8cd" containerName="cleanup" Oct 30 17:15:36.671713 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:36.671698 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="9988e846-2f00-4c0a-b72e-b2943e79f01b" containerName="cleanup" Oct 30 17:15:36.671713 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:36.671704 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ff2bc89-4ecd-41d6-86ca-4111cba66bbd" containerName="cleanup" Oct 30 17:15:36.671713 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:36.671712 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4086c1e-112f-4518-8ae2-3fa5526f9357" containerName="cleanup" Oct 30 17:15:36.671713 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:36.671718 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="53dc2f29-cd5a-4a6b-8e97-cc2d60925ade" containerName="cleanup" Oct 30 17:15:36.671713 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:36.671723 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="25dd3114-a3ba-4976-8f11-611923dc090f" containerName="cleanup" Oct 30 17:15:36.677025 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:36.677005 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71a240d6-npfcw" Oct 30 17:15:36.695960 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:36.695932 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71a240d6-npfcw"] Oct 30 17:15:36.704927 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:36.702581 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:15:36.704927 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:36.702587 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:15:36.704927 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:36.702590 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:15:36.704927 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:36.703168 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 17:15:36.888826 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:36.834484 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b851d0f-5c42-4e0a-8dbf-0428b92f90e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71a240d6-npfcw\" (UID: \"2b851d0f-5c42-4e0a-8dbf-0428b92f90e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71a240d6-npfcw" Oct 30 17:15:36.920921 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:36.920085 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b851d0f-5c42-4e0a-8dbf-0428b92f90e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71a240d6-npfcw\" (UID: \"2b851d0f-5c42-4e0a-8dbf-0428b92f90e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71a240d6-npfcw" Oct 30 17:15:36.920921 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:36.920227 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgcx8\" (UniqueName: \"kubernetes.io/projected/2b851d0f-5c42-4e0a-8dbf-0428b92f90e2-kube-api-access-kgcx8\") pod \"maas-api-revoke-sub-71a240d6-npfcw\" (UID: \"2b851d0f-5c42-4e0a-8dbf-0428b92f90e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71a240d6-npfcw" Oct 30 17:15:37.074037 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:37.053883 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgcx8\" (UniqueName: \"kubernetes.io/projected/2b851d0f-5c42-4e0a-8dbf-0428b92f90e2-kube-api-access-kgcx8\") pod \"maas-api-revoke-sub-71a240d6-npfcw\" (UID: \"2b851d0f-5c42-4e0a-8dbf-0428b92f90e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71a240d6-npfcw" Oct 30 17:15:37.074037 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:37.054194 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b851d0f-5c42-4e0a-8dbf-0428b92f90e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71a240d6-npfcw\" (UID: \"2b851d0f-5c42-4e0a-8dbf-0428b92f90e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71a240d6-npfcw" Oct 30 17:15:37.074037 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:37.054286 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b851d0f-5c42-4e0a-8dbf-0428b92f90e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71a240d6-npfcw\" (UID: \"2b851d0f-5c42-4e0a-8dbf-0428b92f90e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71a240d6-npfcw" Oct 30 17:15:37.074037 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:37.055128 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b851d0f-5c42-4e0a-8dbf-0428b92f90e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71a240d6-npfcw\" (UID: \"2b851d0f-5c42-4e0a-8dbf-0428b92f90e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71a240d6-npfcw" Oct 30 17:15:37.074037 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:37.058423 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b851d0f-5c42-4e0a-8dbf-0428b92f90e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71a240d6-npfcw\" (UID: \"2b851d0f-5c42-4e0a-8dbf-0428b92f90e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71a240d6-npfcw" Oct 30 17:15:37.098807 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:37.098589 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgcx8\" (UniqueName: \"kubernetes.io/projected/2b851d0f-5c42-4e0a-8dbf-0428b92f90e2-kube-api-access-kgcx8\") pod \"maas-api-revoke-sub-71a240d6-npfcw\" (UID: \"2b851d0f-5c42-4e0a-8dbf-0428b92f90e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71a240d6-npfcw" Oct 30 17:15:37.294122 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:37.294028 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71a240d6-npfcw" Oct 30 17:15:37.774708 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:37.774680 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71a240d6-npfcw"] Oct 30 17:15:37.775077 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:15:37.775006 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b851d0f_5c42_4e0a_8dbf_0428b92f90e2.slice/crio-a06a730ad9acd599f6f1f95d5a7c13d8fc22abe923403d5373cb6485223c9cfd WatchSource:0}: Error finding container a06a730ad9acd599f6f1f95d5a7c13d8fc22abe923403d5373cb6485223c9cfd: Status 404 returned error can't find the container with id a06a730ad9acd599f6f1f95d5a7c13d8fc22abe923403d5373cb6485223c9cfd Oct 30 17:15:38.120164 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:38.120136 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71a240d6-npfcw" event={"ID":"2b851d0f-5c42-4e0a-8dbf-0428b92f90e2","Type":"ContainerStarted","Data":"a06a730ad9acd599f6f1f95d5a7c13d8fc22abe923403d5373cb6485223c9cfd"} Oct 30 17:15:39.133941 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:39.133845 4493 generic.go:356] "Generic (PLEG): container finished" podID="2b851d0f-5c42-4e0a-8dbf-0428b92f90e2" containerID="12d03469e87b8b49b6473a3617919b907b5ea2179a56c923564f45b90a5bf025" exitCode=0 Oct 30 17:15:39.134316 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:39.133932 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71a240d6-npfcw" event={"ID":"2b851d0f-5c42-4e0a-8dbf-0428b92f90e2","Type":"ContainerDied","Data":"12d03469e87b8b49b6473a3617919b907b5ea2179a56c923564f45b90a5bf025"} Oct 30 17:15:40.322383 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:40.322356 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71a240d6-npfcw" Oct 30 17:15:40.453528 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:40.453434 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b851d0f-5c42-4e0a-8dbf-0428b92f90e2-kube-api-access-kgcx8\" (UniqueName: \"kubernetes.io/projected/2b851d0f-5c42-4e0a-8dbf-0428b92f90e2-kube-api-access-kgcx8\") pod \"2b851d0f-5c42-4e0a-8dbf-0428b92f90e2\" (UID: \"2b851d0f-5c42-4e0a-8dbf-0428b92f90e2\") " Oct 30 17:15:40.453695 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:40.453571 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b851d0f-5c42-4e0a-8dbf-0428b92f90e2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b851d0f-5c42-4e0a-8dbf-0428b92f90e2-maas-api-xks-ca\") pod \"2b851d0f-5c42-4e0a-8dbf-0428b92f90e2\" (UID: \"2b851d0f-5c42-4e0a-8dbf-0428b92f90e2\") " Oct 30 17:15:40.453695 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:40.453628 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b851d0f-5c42-4e0a-8dbf-0428b92f90e2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b851d0f-5c42-4e0a-8dbf-0428b92f90e2-maas-api-openshift-ca\") pod \"2b851d0f-5c42-4e0a-8dbf-0428b92f90e2\" (UID: \"2b851d0f-5c42-4e0a-8dbf-0428b92f90e2\") " Oct 30 17:15:40.454272 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:40.454235 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b851d0f-5c42-4e0a-8dbf-0428b92f90e2-maas-api-openshift-ca" pod "2b851d0f-5c42-4e0a-8dbf-0428b92f90e2" (UID: "2b851d0f-5c42-4e0a-8dbf-0428b92f90e2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:40.456005 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:40.455975 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b851d0f-5c42-4e0a-8dbf-0428b92f90e2-maas-api-xks-ca" pod "2b851d0f-5c42-4e0a-8dbf-0428b92f90e2" (UID: "2b851d0f-5c42-4e0a-8dbf-0428b92f90e2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:15:40.456221 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:40.456206 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b851d0f-5c42-4e0a-8dbf-0428b92f90e2-kube-api-access-kgcx8" pod "2b851d0f-5c42-4e0a-8dbf-0428b92f90e2" (UID: "2b851d0f-5c42-4e0a-8dbf-0428b92f90e2"). InnerVolumeSpecName "kube-api-access-kgcx8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:40.555921 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:40.555888 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgcx8\" (UniqueName: \"kubernetes.io/projected/2b851d0f-5c42-4e0a-8dbf-0428b92f90e2-kube-api-access-kgcx8\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:15:40.555921 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:40.555919 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b851d0f-5c42-4e0a-8dbf-0428b92f90e2-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:15:40.556110 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:40.555932 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b851d0f-5c42-4e0a-8dbf-0428b92f90e2-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:15:41.155132 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:41.155103 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71a240d6-npfcw" Oct 30 17:15:41.155313 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:41.155142 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71a240d6-npfcw" event={"ID":"2b851d0f-5c42-4e0a-8dbf-0428b92f90e2","Type":"ContainerDied","Data":"a06a730ad9acd599f6f1f95d5a7c13d8fc22abe923403d5373cb6485223c9cfd"} Oct 30 17:15:41.155313 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:41.155191 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a06a730ad9acd599f6f1f95d5a7c13d8fc22abe923403d5373cb6485223c9cfd" Oct 30 17:15:45.800811 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:45.794111 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 17:15:47.872967 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:47.872930 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7b49f59-nhvb5"] Oct 30 17:15:47.874763 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:47.874731 4493 cpu_manager.go:414] "containerMap: removing container" podUID="2b851d0f-5c42-4e0a-8dbf-0428b92f90e2" containerName="revoke-keys" Oct 30 17:15:47.874938 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:47.874923 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b851d0f-5c42-4e0a-8dbf-0428b92f90e2" containerName="revoke-keys" Oct 30 17:15:47.875296 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:47.875280 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b851d0f-5c42-4e0a-8dbf-0428b92f90e2" containerName="revoke-keys" Oct 30 17:15:47.914366 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:47.914336 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7b49f59-nhvb5" Oct 30 17:15:47.979806 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:47.942290 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:15:47.979806 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:47.942384 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:15:47.979806 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:47.951165 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7b49f59-nhvb5"] Oct 30 17:15:47.981675 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:47.981648 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:15:47.981934 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:47.981919 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 17:15:48.085528 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:48.085478 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8863e836-3d23-44ee-971c-2fbf9ca21310-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7b49f59-nhvb5\" (UID: \"8863e836-3d23-44ee-971c-2fbf9ca21310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7b49f59-nhvb5" Oct 30 17:15:48.086218 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:48.086196 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8863e836-3d23-44ee-971c-2fbf9ca21310-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7b49f59-nhvb5\" (UID: \"8863e836-3d23-44ee-971c-2fbf9ca21310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7b49f59-nhvb5" Oct 30 17:15:48.086410 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:48.086380 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxp5x\" (UniqueName: \"kubernetes.io/projected/8863e836-3d23-44ee-971c-2fbf9ca21310-kube-api-access-lxp5x\") pod \"maas-api-revoke-sub-b7b49f59-nhvb5\" (UID: \"8863e836-3d23-44ee-971c-2fbf9ca21310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7b49f59-nhvb5" Oct 30 17:15:48.192806 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:48.189108 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8863e836-3d23-44ee-971c-2fbf9ca21310-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7b49f59-nhvb5\" (UID: \"8863e836-3d23-44ee-971c-2fbf9ca21310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7b49f59-nhvb5" Oct 30 17:15:48.199597 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:48.193291 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8863e836-3d23-44ee-971c-2fbf9ca21310-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7b49f59-nhvb5\" (UID: \"8863e836-3d23-44ee-971c-2fbf9ca21310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7b49f59-nhvb5" Oct 30 17:15:48.199932 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:48.199893 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxp5x\" (UniqueName: \"kubernetes.io/projected/8863e836-3d23-44ee-971c-2fbf9ca21310-kube-api-access-lxp5x\") pod \"maas-api-revoke-sub-b7b49f59-nhvb5\" (UID: \"8863e836-3d23-44ee-971c-2fbf9ca21310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7b49f59-nhvb5" Oct 30 17:15:48.200703 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:48.195728 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8863e836-3d23-44ee-971c-2fbf9ca21310-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7b49f59-nhvb5\" (UID: \"8863e836-3d23-44ee-971c-2fbf9ca21310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7b49f59-nhvb5" Oct 30 17:15:48.216059 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:48.216028 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8863e836-3d23-44ee-971c-2fbf9ca21310-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7b49f59-nhvb5\" (UID: \"8863e836-3d23-44ee-971c-2fbf9ca21310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7b49f59-nhvb5" Oct 30 17:15:48.249201 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:48.249128 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxp5x\" (UniqueName: \"kubernetes.io/projected/8863e836-3d23-44ee-971c-2fbf9ca21310-kube-api-access-lxp5x\") pod \"maas-api-revoke-sub-b7b49f59-nhvb5\" (UID: \"8863e836-3d23-44ee-971c-2fbf9ca21310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7b49f59-nhvb5" Oct 30 17:15:48.336333 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:48.336004 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7b49f59-nhvb5" Oct 30 17:15:48.703284 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:48.703239 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7b49f59-nhvb5"] Oct 30 17:15:49.326891 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:49.323462 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7b49f59-nhvb5" event={"ID":"8863e836-3d23-44ee-971c-2fbf9ca21310","Type":"ContainerStarted","Data":"0a0ddbe41471830d37cc1db919819bc6a464ccb393567fdd00c7b79cfb135e9a"} Oct 30 17:15:50.351273 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:50.351157 4493 generic.go:356] "Generic (PLEG): container finished" podID="8863e836-3d23-44ee-971c-2fbf9ca21310" containerID="587806b7b54ecd6909899c6440a8f8c621517f56101aee3d94afbac812187c23" exitCode=0 Oct 30 17:15:50.351923 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:50.351302 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7b49f59-nhvb5" event={"ID":"8863e836-3d23-44ee-971c-2fbf9ca21310","Type":"ContainerDied","Data":"587806b7b54ecd6909899c6440a8f8c621517f56101aee3d94afbac812187c23"} Oct 30 17:15:50.788807 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:50.785193 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x"] Oct 30 17:15:50.792805 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:50.791131 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x" Oct 30 17:15:50.818236 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:50.817498 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c895a8-b0d7-425c-81eb-282dd1766d59-credential-socket\") pod \"e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x\" (UID: \"18c895a8-b0d7-425c-81eb-282dd1766d59\") " pod="openshift-ingress/e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x" Oct 30 17:15:50.818236 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:50.817554 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18c895a8-b0d7-425c-81eb-282dd1766d59-istio-podinfo\") pod \"e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x\" (UID: \"18c895a8-b0d7-425c-81eb-282dd1766d59\") " pod="openshift-ingress/e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x" Oct 30 17:15:50.818236 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:50.817585 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18c895a8-b0d7-425c-81eb-282dd1766d59-istio-data\") pod \"e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x\" (UID: \"18c895a8-b0d7-425c-81eb-282dd1766d59\") " pod="openshift-ingress/e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x" Oct 30 17:15:50.818236 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:50.817607 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c895a8-b0d7-425c-81eb-282dd1766d59-workload-socket\") pod \"e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x\" (UID: \"18c895a8-b0d7-425c-81eb-282dd1766d59\") " pod="openshift-ingress/e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x" Oct 30 17:15:50.818236 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:50.817636 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88vsm\" (UniqueName: \"kubernetes.io/projected/18c895a8-b0d7-425c-81eb-282dd1766d59-kube-api-access-88vsm\") pod \"e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x\" (UID: \"18c895a8-b0d7-425c-81eb-282dd1766d59\") " pod="openshift-ingress/e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x" Oct 30 17:15:50.818236 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:50.817660 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18c895a8-b0d7-425c-81eb-282dd1766d59-istio-envoy\") pod \"e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x\" (UID: \"18c895a8-b0d7-425c-81eb-282dd1766d59\") " pod="openshift-ingress/e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x" Oct 30 17:15:50.818236 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:50.817713 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18c895a8-b0d7-425c-81eb-282dd1766d59-istiod-ca-cert\") pod \"e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x\" (UID: \"18c895a8-b0d7-425c-81eb-282dd1766d59\") " pod="openshift-ingress/e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x" Oct 30 17:15:50.818236 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:50.817890 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18c895a8-b0d7-425c-81eb-282dd1766d59-workload-certs\") pod \"e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x\" (UID: \"18c895a8-b0d7-425c-81eb-282dd1766d59\") " pod="openshift-ingress/e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x" Oct 30 17:15:50.818236 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:50.817938 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18c895a8-b0d7-425c-81eb-282dd1766d59-istio-token\") pod \"e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x\" (UID: \"18c895a8-b0d7-425c-81eb-282dd1766d59\") " pod="openshift-ingress/e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x" Oct 30 17:15:50.832124 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:50.829574 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x"] Oct 30 17:15:50.838167 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:50.838145 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-bbc36f-openshift-default-dockercfg-t9g6z\"" Oct 30 17:15:50.925271 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:50.923231 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c895a8-b0d7-425c-81eb-282dd1766d59-credential-socket\") pod \"e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x\" (UID: \"18c895a8-b0d7-425c-81eb-282dd1766d59\") " pod="openshift-ingress/e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x" Oct 30 17:15:50.925271 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:50.923310 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18c895a8-b0d7-425c-81eb-282dd1766d59-istio-podinfo\") pod \"e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x\" (UID: \"18c895a8-b0d7-425c-81eb-282dd1766d59\") " pod="openshift-ingress/e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x" Oct 30 17:15:50.925271 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:50.923346 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18c895a8-b0d7-425c-81eb-282dd1766d59-istio-data\") pod \"e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x\" (UID: \"18c895a8-b0d7-425c-81eb-282dd1766d59\") " pod="openshift-ingress/e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x" Oct 30 17:15:50.925271 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:50.923416 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c895a8-b0d7-425c-81eb-282dd1766d59-workload-socket\") pod \"e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x\" (UID: \"18c895a8-b0d7-425c-81eb-282dd1766d59\") " pod="openshift-ingress/e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x" Oct 30 17:15:50.925271 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:50.923464 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88vsm\" (UniqueName: \"kubernetes.io/projected/18c895a8-b0d7-425c-81eb-282dd1766d59-kube-api-access-88vsm\") pod \"e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x\" (UID: \"18c895a8-b0d7-425c-81eb-282dd1766d59\") " pod="openshift-ingress/e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x" Oct 30 17:15:50.925271 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:50.923516 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18c895a8-b0d7-425c-81eb-282dd1766d59-istio-envoy\") pod \"e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x\" (UID: \"18c895a8-b0d7-425c-81eb-282dd1766d59\") " pod="openshift-ingress/e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x" Oct 30 17:15:50.925271 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:50.923615 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18c895a8-b0d7-425c-81eb-282dd1766d59-istiod-ca-cert\") pod \"e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x\" (UID: \"18c895a8-b0d7-425c-81eb-282dd1766d59\") " pod="openshift-ingress/e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x" Oct 30 17:15:50.925271 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:50.923953 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18c895a8-b0d7-425c-81eb-282dd1766d59-workload-certs\") pod \"e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x\" (UID: \"18c895a8-b0d7-425c-81eb-282dd1766d59\") " pod="openshift-ingress/e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x" Oct 30 17:15:50.925271 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:50.924013 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18c895a8-b0d7-425c-81eb-282dd1766d59-istio-token\") pod \"e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x\" (UID: \"18c895a8-b0d7-425c-81eb-282dd1766d59\") " pod="openshift-ingress/e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x" Oct 30 17:15:50.940664 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:50.939995 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c895a8-b0d7-425c-81eb-282dd1766d59-workload-socket\") pod \"e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x\" (UID: \"18c895a8-b0d7-425c-81eb-282dd1766d59\") " pod="openshift-ingress/e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x" Oct 30 17:15:50.940664 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:50.940252 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c895a8-b0d7-425c-81eb-282dd1766d59-credential-socket\") pod \"e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x\" (UID: \"18c895a8-b0d7-425c-81eb-282dd1766d59\") " pod="openshift-ingress/e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x" Oct 30 17:15:50.964035 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:50.963606 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18c895a8-b0d7-425c-81eb-282dd1766d59-istio-data\") pod \"e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x\" (UID: \"18c895a8-b0d7-425c-81eb-282dd1766d59\") " pod="openshift-ingress/e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x" Oct 30 17:15:50.992808 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:50.969459 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18c895a8-b0d7-425c-81eb-282dd1766d59-istiod-ca-cert\") pod \"e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x\" (UID: \"18c895a8-b0d7-425c-81eb-282dd1766d59\") " pod="openshift-ingress/e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x" Oct 30 17:15:50.992808 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:50.969684 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18c895a8-b0d7-425c-81eb-282dd1766d59-workload-certs\") pod \"e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x\" (UID: \"18c895a8-b0d7-425c-81eb-282dd1766d59\") " pod="openshift-ingress/e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x" Oct 30 17:15:51.016077 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:51.016002 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18c895a8-b0d7-425c-81eb-282dd1766d59-istio-podinfo\") pod \"e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x\" (UID: \"18c895a8-b0d7-425c-81eb-282dd1766d59\") " pod="openshift-ingress/e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x" Oct 30 17:15:51.148933 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:51.138504 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88vsm\" (UniqueName: \"kubernetes.io/projected/18c895a8-b0d7-425c-81eb-282dd1766d59-kube-api-access-88vsm\") pod \"e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x\" (UID: \"18c895a8-b0d7-425c-81eb-282dd1766d59\") " pod="openshift-ingress/e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x" Oct 30 17:15:51.352815 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:51.340430 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18c895a8-b0d7-425c-81eb-282dd1766d59-istio-token\") pod \"e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x\" (UID: \"18c895a8-b0d7-425c-81eb-282dd1766d59\") " pod="openshift-ingress/e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x" Oct 30 17:15:51.554869 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:51.554435 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18c895a8-b0d7-425c-81eb-282dd1766d59-istio-envoy\") pod \"e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x\" (UID: \"18c895a8-b0d7-425c-81eb-282dd1766d59\") " pod="openshift-ingress/e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x" Oct 30 17:15:51.652909 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:51.652843 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" podUID="e76172b1-659d-45bd-91c6-1a0ce4062335" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:15:51.655427 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:51.655362 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" podUID="18b72648-f523-405e-b5cd-1266ff13f8f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:15:51.678942 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:51.678896 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" podUID="b5abe703-fee0-4185-aa22-88d1db06392b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:15:51.732763 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:51.732456 4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" podUID="08baad08-cd72-4305-b92d-c11a84ef276f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": context deadline exceeded" Oct 30 17:15:51.763818 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:51.760327 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x" Oct 30 17:15:52.164624 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:52.164589 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7b49f59-nhvb5" Oct 30 17:15:52.254420 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:52.254382 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8863e836-3d23-44ee-971c-2fbf9ca21310-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8863e836-3d23-44ee-971c-2fbf9ca21310-maas-api-openshift-ca\") pod \"8863e836-3d23-44ee-971c-2fbf9ca21310\" (UID: \"8863e836-3d23-44ee-971c-2fbf9ca21310\") " Oct 30 17:15:52.254870 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:52.254845 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8863e836-3d23-44ee-971c-2fbf9ca21310-kube-api-access-lxp5x\" (UniqueName: \"kubernetes.io/projected/8863e836-3d23-44ee-971c-2fbf9ca21310-kube-api-access-lxp5x\") pod \"8863e836-3d23-44ee-971c-2fbf9ca21310\" (UID: \"8863e836-3d23-44ee-971c-2fbf9ca21310\") " Oct 30 17:15:52.255071 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:52.255052 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8863e836-3d23-44ee-971c-2fbf9ca21310-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8863e836-3d23-44ee-971c-2fbf9ca21310-maas-api-xks-ca\") pod \"8863e836-3d23-44ee-971c-2fbf9ca21310\" (UID: \"8863e836-3d23-44ee-971c-2fbf9ca21310\") " Oct 30 17:15:52.265528 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:52.265230 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8863e836-3d23-44ee-971c-2fbf9ca21310-maas-api-openshift-ca" pod "8863e836-3d23-44ee-971c-2fbf9ca21310" (UID: "8863e836-3d23-44ee-971c-2fbf9ca21310"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:52.272370 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:52.272327 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8863e836-3d23-44ee-971c-2fbf9ca21310-kube-api-access-lxp5x" pod "8863e836-3d23-44ee-971c-2fbf9ca21310" (UID: "8863e836-3d23-44ee-971c-2fbf9ca21310"). InnerVolumeSpecName "kube-api-access-lxp5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:52.274538 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:52.274498 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxp5x\" (UniqueName: \"kubernetes.io/projected/8863e836-3d23-44ee-971c-2fbf9ca21310-kube-api-access-lxp5x\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:15:52.274721 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:52.274704 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8863e836-3d23-44ee-971c-2fbf9ca21310-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:15:52.279001 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:52.278975 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x"] Oct 30 17:15:52.289003 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:52.288963 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:15:52.289113 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:52.289053 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:15:52.289113 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:52.289097 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:15:52.290395 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:52.290356 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8863e836-3d23-44ee-971c-2fbf9ca21310-maas-api-xks-ca" pod "8863e836-3d23-44ee-971c-2fbf9ca21310" (UID: "8863e836-3d23-44ee-971c-2fbf9ca21310"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:15:52.406922 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:52.398644 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8863e836-3d23-44ee-971c-2fbf9ca21310-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:15:52.647078 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:52.647040 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7b49f59-nhvb5" Oct 30 17:15:52.648030 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:52.647995 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7b49f59-nhvb5" event={"ID":"8863e836-3d23-44ee-971c-2fbf9ca21310","Type":"ContainerDied","Data":"0a0ddbe41471830d37cc1db919819bc6a464ccb393567fdd00c7b79cfb135e9a"} Oct 30 17:15:52.648175 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:52.648051 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a0ddbe41471830d37cc1db919819bc6a464ccb393567fdd00c7b79cfb135e9a" Oct 30 17:15:52.655921 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:52.655887 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x" event={"ID":"18c895a8-b0d7-425c-81eb-282dd1766d59","Type":"ContainerStarted","Data":"1b94d423f4e040453995404381a7c43ce5b63dde548a6758b4e1811a815c94e1"} Oct 30 17:15:52.656051 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:52.655952 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x" event={"ID":"18c895a8-b0d7-425c-81eb-282dd1766d59","Type":"ContainerStarted","Data":"dca96e9821492069910280cd9445ad4dc8bfe22805f31a75c13ba19e90f46663"} Oct 30 17:15:52.738449 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:52.738288 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x" podStartSLOduration=2.738271418 podStartE2EDuration="2.738271418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:52.702469086 +0000 UTC m=+3179.929644218" watchObservedRunningTime="2026-10-30 17:15:52.738271418 +0000 UTC m=+3179.965446517" Oct 30 17:15:52.760846 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:52.760804 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x" Oct 30 17:15:53.178386 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:53.174453 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk"] Oct 30 17:15:53.178386 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:53.175600 4493 cpu_manager.go:414] "containerMap: removing container" podUID="8863e836-3d23-44ee-971c-2fbf9ca21310" containerName="revoke-keys" Oct 30 17:15:53.178386 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:53.175619 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8863e836-3d23-44ee-971c-2fbf9ca21310" containerName="revoke-keys" Oct 30 17:15:53.178386 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:53.175759 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="8863e836-3d23-44ee-971c-2fbf9ca21310" containerName="revoke-keys" Oct 30 17:15:53.225810 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:53.216118 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk"] Oct 30 17:15:53.225810 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:53.216250 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk" Oct 30 17:15:53.254813 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:53.235818 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-4c6596-openshift-default-dockercfg-6zgz9\"" Oct 30 17:15:53.384805 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:53.343972 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e934b3f5-e168-4760-9ea6-3b9cab799328-credential-socket\") pod \"e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk\" (UID: \"e934b3f5-e168-4760-9ea6-3b9cab799328\") " pod="openshift-ingress/e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk" Oct 30 17:15:53.384805 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:53.344033 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e934b3f5-e168-4760-9ea6-3b9cab799328-istio-token\") pod \"e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk\" (UID: \"e934b3f5-e168-4760-9ea6-3b9cab799328\") " pod="openshift-ingress/e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk" Oct 30 17:15:53.384805 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:53.344075 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e934b3f5-e168-4760-9ea6-3b9cab799328-istiod-ca-cert\") pod \"e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk\" (UID: \"e934b3f5-e168-4760-9ea6-3b9cab799328\") " pod="openshift-ingress/e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk" Oct 30 17:15:53.384805 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:53.344149 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xqnd\" (UniqueName: \"kubernetes.io/projected/e934b3f5-e168-4760-9ea6-3b9cab799328-kube-api-access-8xqnd\") pod \"e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk\" (UID: \"e934b3f5-e168-4760-9ea6-3b9cab799328\") " pod="openshift-ingress/e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk" Oct 30 17:15:53.384805 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:53.344416 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e934b3f5-e168-4760-9ea6-3b9cab799328-workload-certs\") pod \"e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk\" (UID: \"e934b3f5-e168-4760-9ea6-3b9cab799328\") " pod="openshift-ingress/e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk" Oct 30 17:15:53.384805 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:53.344480 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e934b3f5-e168-4760-9ea6-3b9cab799328-workload-socket\") pod \"e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk\" (UID: \"e934b3f5-e168-4760-9ea6-3b9cab799328\") " pod="openshift-ingress/e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk" Oct 30 17:15:53.384805 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:53.344530 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e934b3f5-e168-4760-9ea6-3b9cab799328-istio-data\") pod \"e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk\" (UID: \"e934b3f5-e168-4760-9ea6-3b9cab799328\") " pod="openshift-ingress/e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk" Oct 30 17:15:53.384805 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:53.344587 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e934b3f5-e168-4760-9ea6-3b9cab799328-istio-podinfo\") pod \"e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk\" (UID: \"e934b3f5-e168-4760-9ea6-3b9cab799328\") " pod="openshift-ingress/e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk" Oct 30 17:15:53.384805 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:53.344661 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e934b3f5-e168-4760-9ea6-3b9cab799328-istio-envoy\") pod \"e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk\" (UID: \"e934b3f5-e168-4760-9ea6-3b9cab799328\") " pod="openshift-ingress/e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk" Oct 30 17:15:53.474858 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:53.466353 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e934b3f5-e168-4760-9ea6-3b9cab799328-istio-envoy\") pod \"e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk\" (UID: \"e934b3f5-e168-4760-9ea6-3b9cab799328\") " pod="openshift-ingress/e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk" Oct 30 17:15:53.507014 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:53.506975 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e934b3f5-e168-4760-9ea6-3b9cab799328-credential-socket\") pod \"e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk\" (UID: \"e934b3f5-e168-4760-9ea6-3b9cab799328\") " pod="openshift-ingress/e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk" Oct 30 17:15:53.507278 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:53.507253 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e934b3f5-e168-4760-9ea6-3b9cab799328-istio-token\") pod \"e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk\" (UID: \"e934b3f5-e168-4760-9ea6-3b9cab799328\") " pod="openshift-ingress/e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk" Oct 30 17:15:53.507451 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:53.507430 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e934b3f5-e168-4760-9ea6-3b9cab799328-istiod-ca-cert\") pod \"e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk\" (UID: \"e934b3f5-e168-4760-9ea6-3b9cab799328\") " pod="openshift-ingress/e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk" Oct 30 17:15:53.507658 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:53.507641 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xqnd\" (UniqueName: \"kubernetes.io/projected/e934b3f5-e168-4760-9ea6-3b9cab799328-kube-api-access-8xqnd\") pod \"e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk\" (UID: \"e934b3f5-e168-4760-9ea6-3b9cab799328\") " pod="openshift-ingress/e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk" Oct 30 17:15:53.515810 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:53.507907 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e934b3f5-e168-4760-9ea6-3b9cab799328-workload-certs\") pod \"e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk\" (UID: \"e934b3f5-e168-4760-9ea6-3b9cab799328\") " pod="openshift-ingress/e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk" Oct 30 17:15:53.532457 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:53.532421 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e934b3f5-e168-4760-9ea6-3b9cab799328-workload-socket\") pod \"e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk\" (UID: \"e934b3f5-e168-4760-9ea6-3b9cab799328\") " pod="openshift-ingress/e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk" Oct 30 17:15:53.539806 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:53.535893 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e934b3f5-e168-4760-9ea6-3b9cab799328-istio-data\") pod \"e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk\" (UID: \"e934b3f5-e168-4760-9ea6-3b9cab799328\") " pod="openshift-ingress/e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk" Oct 30 17:15:53.557805 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:53.550053 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e934b3f5-e168-4760-9ea6-3b9cab799328-istio-podinfo\") pod \"e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk\" (UID: \"e934b3f5-e168-4760-9ea6-3b9cab799328\") " pod="openshift-ingress/e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk" Oct 30 17:15:53.660717 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:53.516620 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e934b3f5-e168-4760-9ea6-3b9cab799328-credential-socket\") pod \"e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk\" (UID: \"e934b3f5-e168-4760-9ea6-3b9cab799328\") " pod="openshift-ingress/e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk" Oct 30 17:15:53.660717 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:53.630516 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e934b3f5-e168-4760-9ea6-3b9cab799328-workload-socket\") pod \"e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk\" (UID: \"e934b3f5-e168-4760-9ea6-3b9cab799328\") " pod="openshift-ingress/e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk" Oct 30 17:15:53.660717 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:53.517242 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e934b3f5-e168-4760-9ea6-3b9cab799328-istiod-ca-cert\") pod \"e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk\" (UID: \"e934b3f5-e168-4760-9ea6-3b9cab799328\") " pod="openshift-ingress/e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk" Oct 30 17:15:53.660717 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:53.630826 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e934b3f5-e168-4760-9ea6-3b9cab799328-istio-data\") pod \"e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk\" (UID: \"e934b3f5-e168-4760-9ea6-3b9cab799328\") " pod="openshift-ingress/e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk" Oct 30 17:15:53.660717 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:53.525676 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e934b3f5-e168-4760-9ea6-3b9cab799328-workload-certs\") pod \"e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk\" (UID: \"e934b3f5-e168-4760-9ea6-3b9cab799328\") " pod="openshift-ingress/e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk" Oct 30 17:15:53.660717 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:53.631291 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e934b3f5-e168-4760-9ea6-3b9cab799328-istio-podinfo\") pod \"e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk\" (UID: \"e934b3f5-e168-4760-9ea6-3b9cab799328\") " pod="openshift-ingress/e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk" Oct 30 17:15:53.786500 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:53.785954 4493 patch_prober.go:29] interesting pod/e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.43:15021/healthz/ready\": dial tcp 10.134.1.43:15021: connect: connection refused" start-of-body= Oct 30 17:15:53.786500 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:53.786019 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x" podUID="18c895a8-b0d7-425c-81eb-282dd1766d59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.43:15021/healthz/ready\": dial tcp 10.134.1.43:15021: connect: connection refused" Oct 30 17:15:53.840356 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:53.827691 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e934b3f5-e168-4760-9ea6-3b9cab799328-istio-envoy\") pod \"e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk\" (UID: \"e934b3f5-e168-4760-9ea6-3b9cab799328\") " pod="openshift-ingress/e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk" Oct 30 17:15:53.840356 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:53.828667 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e934b3f5-e168-4760-9ea6-3b9cab799328-istio-token\") pod \"e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk\" (UID: \"e934b3f5-e168-4760-9ea6-3b9cab799328\") " pod="openshift-ingress/e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk" Oct 30 17:15:53.875808 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:53.848584 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xqnd\" (UniqueName: \"kubernetes.io/projected/e934b3f5-e168-4760-9ea6-3b9cab799328-kube-api-access-8xqnd\") pod \"e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk\" (UID: \"e934b3f5-e168-4760-9ea6-3b9cab799328\") " pod="openshift-ingress/e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk" Oct 30 17:15:53.876299 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:53.876271 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk" Oct 30 17:15:54.462856 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:54.462511 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk"] Oct 30 17:15:54.468671 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:54.468630 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:15:54.468830 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:54.468701 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:15:54.468830 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:54.468748 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:15:54.767191 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:54.767100 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x" Oct 30 17:15:54.878956 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:54.878924 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk" event={"ID":"e934b3f5-e168-4760-9ea6-3b9cab799328","Type":"ContainerStarted","Data":"2847df9bcd14d1409887a15fb491be9a31a9ff83b9f5ee7e83ca6d7948d700d8"} Oct 30 17:15:54.879513 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:54.879206 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk" event={"ID":"e934b3f5-e168-4760-9ea6-3b9cab799328","Type":"ContainerStarted","Data":"46eb75ec6470f77beb9a08cdabd83639cc45a67cd6f2b60f344e8c6426032006"} Oct 30 17:15:54.879691 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:54.879678 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x" Oct 30 17:15:54.880582 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:54.880540 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x" Oct 30 17:15:54.925242 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:54.925174 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk" podStartSLOduration=1.925155362 podStartE2EDuration="1.925155362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:54.923170959 +0000 UTC m=+3182.150346060" watchObservedRunningTime="2026-10-30 17:15:54.925155362 +0000 UTC m=+3182.152330464" Oct 30 17:15:55.877045 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:55.876993 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk" Oct 30 17:15:55.958821 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:55.952469 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk" Oct 30 17:15:56.939567 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:56.939537 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk" Oct 30 17:15:56.940139 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:56.939634 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk" Oct 30 17:15:57.119806 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:57.119756 4493 scope.go:122] "RemoveContainer" containerID="88ad625b311ade19a7c792ee8678afb4e1d4bd51fe74d56bfbc748b143695695" Oct 30 17:15:57.141691 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:57.141654 4493 scope.go:122] "RemoveContainer" containerID="54ba4a27e1edb2a0aedb7ce3b6311acdabb02860cc4ce55dd6cf0261785a44f9" Oct 30 17:15:57.159154 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:15:57.159126 4493 scope.go:122] "RemoveContainer" containerID="9c80f19d37c35de1aed8434920a182db260688710af160d4ddc982514a538c32" Oct 30 17:16:00.492484 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:16:00.492434 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 17:16:01.338414 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:01.338314 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x"] Oct 30 17:16:01.342320 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:01.342293 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x" Oct 30 17:16:01.348869 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:01.348841 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt"] Oct 30 17:16:01.353016 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:01.352997 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt" Oct 30 17:16:01.370032 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:01.369946 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-9afbb2-openshift-default-dockercfg-vkv8d\"" Oct 30 17:16:01.376974 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:01.375167 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x"] Oct 30 17:16:01.383255 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:01.383216 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46c8b953-69a7-4e2e-8695-a0b9d745529f-istiod-ca-cert\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt\" (UID: \"46c8b953-69a7-4e2e-8695-a0b9d745529f\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt" Oct 30 17:16:01.383416 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:01.383369 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flv5q\" (UniqueName: \"kubernetes.io/projected/46c8b953-69a7-4e2e-8695-a0b9d745529f-kube-api-access-flv5q\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt\" (UID: \"46c8b953-69a7-4e2e-8695-a0b9d745529f\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt" Oct 30 17:16:01.383481 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:01.383412 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46c8b953-69a7-4e2e-8695-a0b9d745529f-istio-podinfo\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt\" (UID: \"46c8b953-69a7-4e2e-8695-a0b9d745529f\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt" Oct 30 17:16:01.383481 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:01.383438 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t78wv\" (UniqueName: \"kubernetes.io/projected/273838d6-034c-4763-a75f-5a357a4953f4-kube-api-access-t78wv\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x\" (UID: \"273838d6-034c-4763-a75f-5a357a4953f4\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x" Oct 30 17:16:01.383481 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:01.383473 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/273838d6-034c-4763-a75f-5a357a4953f4-istiod-ca-cert\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x\" (UID: \"273838d6-034c-4763-a75f-5a357a4953f4\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x" Oct 30 17:16:01.383645 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:01.383508 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/273838d6-034c-4763-a75f-5a357a4953f4-istio-token\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x\" (UID: \"273838d6-034c-4763-a75f-5a357a4953f4\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x" Oct 30 17:16:01.383645 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:01.383531 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/273838d6-034c-4763-a75f-5a357a4953f4-workload-certs\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x\" (UID: \"273838d6-034c-4763-a75f-5a357a4953f4\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x" Oct 30 17:16:01.383645 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:01.383559 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46c8b953-69a7-4e2e-8695-a0b9d745529f-workload-socket\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt\" (UID: \"46c8b953-69a7-4e2e-8695-a0b9d745529f\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt" Oct 30 17:16:01.383645 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:01.383598 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46c8b953-69a7-4e2e-8695-a0b9d745529f-istio-token\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt\" (UID: \"46c8b953-69a7-4e2e-8695-a0b9d745529f\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt" Oct 30 17:16:01.383645 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:01.383627 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46c8b953-69a7-4e2e-8695-a0b9d745529f-workload-certs\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt\" (UID: \"46c8b953-69a7-4e2e-8695-a0b9d745529f\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt" Oct 30 17:16:01.383967 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:01.383656 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46c8b953-69a7-4e2e-8695-a0b9d745529f-istio-data\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt\" (UID: \"46c8b953-69a7-4e2e-8695-a0b9d745529f\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt" Oct 30 17:16:01.383967 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:01.383716 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46c8b953-69a7-4e2e-8695-a0b9d745529f-credential-socket\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt\" (UID: \"46c8b953-69a7-4e2e-8695-a0b9d745529f\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt" Oct 30 17:16:01.383967 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:01.383743 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/273838d6-034c-4763-a75f-5a357a4953f4-credential-socket\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x\" (UID: \"273838d6-034c-4763-a75f-5a357a4953f4\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x" Oct 30 17:16:01.383967 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:01.383854 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46c8b953-69a7-4e2e-8695-a0b9d745529f-istio-envoy\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt\" (UID: \"46c8b953-69a7-4e2e-8695-a0b9d745529f\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt" Oct 30 17:16:01.383967 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:01.383915 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/273838d6-034c-4763-a75f-5a357a4953f4-workload-socket\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x\" (UID: \"273838d6-034c-4763-a75f-5a357a4953f4\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x" Oct 30 17:16:01.383967 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:01.383967 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/273838d6-034c-4763-a75f-5a357a4953f4-istio-data\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x\" (UID: \"273838d6-034c-4763-a75f-5a357a4953f4\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x" Oct 30 17:16:01.384284 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:01.383984 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/273838d6-034c-4763-a75f-5a357a4953f4-istio-envoy\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x\" (UID: \"273838d6-034c-4763-a75f-5a357a4953f4\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x" Oct 30 17:16:01.384284 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:01.383999 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/273838d6-034c-4763-a75f-5a357a4953f4-istio-podinfo\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x\" (UID: \"273838d6-034c-4763-a75f-5a357a4953f4\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x" Oct 30 17:16:01.402691 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:01.402660 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt"] Oct 30 17:16:01.485582 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:01.485477 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/273838d6-034c-4763-a75f-5a357a4953f4-workload-socket\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x\" (UID: \"273838d6-034c-4763-a75f-5a357a4953f4\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x" Oct 30 17:16:01.485582 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:01.485564 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/273838d6-034c-4763-a75f-5a357a4953f4-istio-data\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x\" (UID: \"273838d6-034c-4763-a75f-5a357a4953f4\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x" Oct 30 17:16:01.485885 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:01.485592 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/273838d6-034c-4763-a75f-5a357a4953f4-istio-envoy\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x\" (UID: \"273838d6-034c-4763-a75f-5a357a4953f4\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x" Oct 30 17:16:01.485885 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:01.485616 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/273838d6-034c-4763-a75f-5a357a4953f4-istio-podinfo\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x\" (UID: \"273838d6-034c-4763-a75f-5a357a4953f4\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x" Oct 30 17:16:01.485885 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:01.485663 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46c8b953-69a7-4e2e-8695-a0b9d745529f-istiod-ca-cert\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt\" (UID: \"46c8b953-69a7-4e2e-8695-a0b9d745529f\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt" Oct 30 17:16:01.485885 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:01.485843 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flv5q\" (UniqueName: \"kubernetes.io/projected/46c8b953-69a7-4e2e-8695-a0b9d745529f-kube-api-access-flv5q\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt\" (UID: \"46c8b953-69a7-4e2e-8695-a0b9d745529f\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt" Oct 30 17:16:01.485885 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:01.485882 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46c8b953-69a7-4e2e-8695-a0b9d745529f-istio-podinfo\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt\" (UID: \"46c8b953-69a7-4e2e-8695-a0b9d745529f\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt" Oct 30 17:16:01.486209 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:01.485917 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t78wv\" (UniqueName: \"kubernetes.io/projected/273838d6-034c-4763-a75f-5a357a4953f4-kube-api-access-t78wv\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x\" (UID: \"273838d6-034c-4763-a75f-5a357a4953f4\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x" Oct 30 17:16:01.486209 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:01.485953 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/273838d6-034c-4763-a75f-5a357a4953f4-istiod-ca-cert\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x\" (UID: \"273838d6-034c-4763-a75f-5a357a4953f4\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x" Oct 30 17:16:01.486209 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:01.485991 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/273838d6-034c-4763-a75f-5a357a4953f4-istio-token\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x\" (UID: \"273838d6-034c-4763-a75f-5a357a4953f4\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x" Oct 30 17:16:01.486209 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:01.486017 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/273838d6-034c-4763-a75f-5a357a4953f4-workload-certs\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x\" (UID: \"273838d6-034c-4763-a75f-5a357a4953f4\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x" Oct 30 17:16:01.486209 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:01.486018 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/273838d6-034c-4763-a75f-5a357a4953f4-workload-socket\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x\" (UID: \"273838d6-034c-4763-a75f-5a357a4953f4\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x" Oct 30 17:16:01.486209 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:01.486045 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46c8b953-69a7-4e2e-8695-a0b9d745529f-workload-socket\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt\" (UID: \"46c8b953-69a7-4e2e-8695-a0b9d745529f\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt" Oct 30 17:16:01.486209 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:01.486085 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46c8b953-69a7-4e2e-8695-a0b9d745529f-istio-token\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt\" (UID: \"46c8b953-69a7-4e2e-8695-a0b9d745529f\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt" Oct 30 17:16:01.486209 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:01.486116 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46c8b953-69a7-4e2e-8695-a0b9d745529f-workload-certs\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt\" (UID: \"46c8b953-69a7-4e2e-8695-a0b9d745529f\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt" Oct 30 17:16:01.486209 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:01.486148 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46c8b953-69a7-4e2e-8695-a0b9d745529f-istio-data\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt\" (UID: \"46c8b953-69a7-4e2e-8695-a0b9d745529f\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt" Oct 30 17:16:01.486935 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:01.486217 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46c8b953-69a7-4e2e-8695-a0b9d745529f-credential-socket\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt\" (UID: \"46c8b953-69a7-4e2e-8695-a0b9d745529f\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt" Oct 30 17:16:01.486935 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:01.486244 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/273838d6-034c-4763-a75f-5a357a4953f4-istio-data\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x\" (UID: \"273838d6-034c-4763-a75f-5a357a4953f4\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x" Oct 30 17:16:01.486935 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:01.486258 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/273838d6-034c-4763-a75f-5a357a4953f4-credential-socket\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x\" (UID: \"273838d6-034c-4763-a75f-5a357a4953f4\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x" Oct 30 17:16:01.486935 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:01.486395 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46c8b953-69a7-4e2e-8695-a0b9d745529f-istio-envoy\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt\" (UID: \"46c8b953-69a7-4e2e-8695-a0b9d745529f\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt" Oct 30 17:16:01.486935 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:01.486480 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46c8b953-69a7-4e2e-8695-a0b9d745529f-istiod-ca-cert\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt\" (UID: \"46c8b953-69a7-4e2e-8695-a0b9d745529f\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt" Oct 30 17:16:01.486935 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:01.486511 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/273838d6-034c-4763-a75f-5a357a4953f4-credential-socket\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x\" (UID: \"273838d6-034c-4763-a75f-5a357a4953f4\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x" Oct 30 17:16:01.486935 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:01.486628 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/273838d6-034c-4763-a75f-5a357a4953f4-istiod-ca-cert\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x\" (UID: \"273838d6-034c-4763-a75f-5a357a4953f4\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x" Oct 30 17:16:01.486935 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:01.486822 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46c8b953-69a7-4e2e-8695-a0b9d745529f-workload-socket\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt\" (UID: \"46c8b953-69a7-4e2e-8695-a0b9d745529f\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt" Oct 30 17:16:01.486935 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:01.486877 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46c8b953-69a7-4e2e-8695-a0b9d745529f-credential-socket\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt\" (UID: \"46c8b953-69a7-4e2e-8695-a0b9d745529f\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt" Oct 30 17:16:01.486935 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:01.486882 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46c8b953-69a7-4e2e-8695-a0b9d745529f-istio-data\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt\" (UID: \"46c8b953-69a7-4e2e-8695-a0b9d745529f\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt" Oct 30 17:16:01.487507 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:01.486962 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46c8b953-69a7-4e2e-8695-a0b9d745529f-workload-certs\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt\" (UID: \"46c8b953-69a7-4e2e-8695-a0b9d745529f\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt" Oct 30 17:16:01.487507 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:01.487102 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/273838d6-034c-4763-a75f-5a357a4953f4-workload-certs\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x\" (UID: \"273838d6-034c-4763-a75f-5a357a4953f4\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x" Oct 30 17:16:01.488609 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:01.488583 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/273838d6-034c-4763-a75f-5a357a4953f4-istio-podinfo\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x\" (UID: \"273838d6-034c-4763-a75f-5a357a4953f4\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x" Oct 30 17:16:01.489084 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:01.489064 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/273838d6-034c-4763-a75f-5a357a4953f4-istio-envoy\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x\" (UID: \"273838d6-034c-4763-a75f-5a357a4953f4\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x" Oct 30 17:16:01.489420 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:01.489398 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46c8b953-69a7-4e2e-8695-a0b9d745529f-istio-envoy\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt\" (UID: \"46c8b953-69a7-4e2e-8695-a0b9d745529f\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt" Oct 30 17:16:01.489582 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:01.489565 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46c8b953-69a7-4e2e-8695-a0b9d745529f-istio-podinfo\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt\" (UID: \"46c8b953-69a7-4e2e-8695-a0b9d745529f\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt" Oct 30 17:16:01.518216 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:01.518146 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flv5q\" (UniqueName: \"kubernetes.io/projected/46c8b953-69a7-4e2e-8695-a0b9d745529f-kube-api-access-flv5q\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt\" (UID: \"46c8b953-69a7-4e2e-8695-a0b9d745529f\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt" Oct 30 17:16:01.568849 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:01.555833 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46c8b953-69a7-4e2e-8695-a0b9d745529f-istio-token\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt\" (UID: \"46c8b953-69a7-4e2e-8695-a0b9d745529f\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt" Oct 30 17:16:01.568849 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:01.558602 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/273838d6-034c-4763-a75f-5a357a4953f4-istio-token\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x\" (UID: \"273838d6-034c-4763-a75f-5a357a4953f4\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x" Oct 30 17:16:01.616811 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:01.594063 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t78wv\" (UniqueName: \"kubernetes.io/projected/273838d6-034c-4763-a75f-5a357a4953f4-kube-api-access-t78wv\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x\" (UID: \"273838d6-034c-4763-a75f-5a357a4953f4\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x" Oct 30 17:16:01.660886 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:01.660579 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x" Oct 30 17:16:01.672140 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:01.672100 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt" Oct 30 17:16:02.022283 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:02.022249 4493 generic.go:356] "Generic (PLEG): container finished" podID="ba3aa602-ca3b-4d37-9156-7c95b17a99cd" containerID="33f6e9b7a3fe74fad17b82952f1bd31a08384ffaf3a4a9c54df250e09ebb4b54" exitCode=0 Oct 30 17:16:02.022479 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:02.022326 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6" event={"ID":"ba3aa602-ca3b-4d37-9156-7c95b17a99cd","Type":"ContainerDied","Data":"33f6e9b7a3fe74fad17b82952f1bd31a08384ffaf3a4a9c54df250e09ebb4b54"} Oct 30 17:16:02.022880 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:02.022862 4493 scope.go:122] "RemoveContainer" containerID="33f6e9b7a3fe74fad17b82952f1bd31a08384ffaf3a4a9c54df250e09ebb4b54" Oct 30 17:16:02.024901 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:02.024870 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:16:02.025007 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:02.024944 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:16:02.025007 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:02.024978 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:16:02.322206 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:02.322124 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x"] Oct 30 17:16:02.336226 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:02.332859 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:16:02.336226 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:02.332956 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:16:02.336226 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:02.332995 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:16:02.348952 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:02.340400 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:16:02.348952 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:02.340486 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:16:02.348952 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:02.340526 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:16:02.371797 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:02.371744 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt"] Oct 30 17:16:03.052746 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:03.052701 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x" event={"ID":"273838d6-034c-4763-a75f-5a357a4953f4","Type":"ContainerStarted","Data":"f9eba3fa07ffe13d21325f081b011b36a7f097ec03725b08099c1f83423407e4"} Oct 30 17:16:03.053562 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:03.053534 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x" event={"ID":"273838d6-034c-4763-a75f-5a357a4953f4","Type":"ContainerStarted","Data":"62d1d03cc3477a59810bd27cd321c9cf18d865d5ff11ec0c0bc0aa4a1bca1735"} Oct 30 17:16:03.056526 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:03.056491 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6" event={"ID":"ba3aa602-ca3b-4d37-9156-7c95b17a99cd","Type":"ContainerStarted","Data":"0eb9a58c2bc15c366b9af7e475d1124a7155c0befceb1171492d5d6143e969c2"} Oct 30 17:16:03.059647 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:03.059623 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt" event={"ID":"46c8b953-69a7-4e2e-8695-a0b9d745529f","Type":"ContainerStarted","Data":"83c7dfe4cc1f60ab3d1349a28f17511fe13e8aa88b83ef1ef858ddb9d83a0559"} Oct 30 17:16:03.059826 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:03.059777 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt" event={"ID":"46c8b953-69a7-4e2e-8695-a0b9d745529f","Type":"ContainerStarted","Data":"8494b0d3ea0b168f3afdba6fc34020090b497d291c1e9b6d5403c886ea6a4d44"} Oct 30 17:16:03.133749 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:03.133681 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x" podStartSLOduration=2.133660664 podStartE2EDuration="2.133660664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:16:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:16:03.117580004 +0000 UTC m=+3190.344755105" watchObservedRunningTime="2026-10-30 17:16:03.133660664 +0000 UTC m=+3190.360835758" Oct 30 17:16:03.663898 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:03.663863 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x" Oct 30 17:16:03.677808 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:03.675209 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt" Oct 30 17:16:03.677808 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:03.675330 4493 patch_prober.go:29] interesting pod/e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.46:15021/healthz/ready\": dial tcp 10.134.1.46:15021: connect: connection refused" start-of-body= Oct 30 17:16:03.677808 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:03.675374 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt" podUID="46c8b953-69a7-4e2e-8695-a0b9d745529f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.46:15021/healthz/ready\": dial tcp 10.134.1.46:15021: connect: connection refused" Oct 30 17:16:03.679167 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:03.679141 4493 patch_prober.go:29] interesting pod/e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.45:15021/healthz/ready\": dial tcp 10.134.1.45:15021: connect: connection refused" start-of-body= Oct 30 17:16:03.679370 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:03.679346 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x" podUID="273838d6-034c-4763-a75f-5a357a4953f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.45:15021/healthz/ready\": dial tcp 10.134.1.45:15021: connect: connection refused" Oct 30 17:16:04.040000 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:04.039901 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6" Oct 30 17:16:04.040191 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:04.040056 4493 patch_prober.go:29] interesting pod/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.19:15021/healthz/ready\": dial tcp 10.134.1.19:15021: connect: connection refused" start-of-body= Oct 30 17:16:04.040191 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:04.040104 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6" podUID="ba3aa602-ca3b-4d37-9156-7c95b17a99cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.19:15021/healthz/ready\": dial tcp 10.134.1.19:15021: connect: connection refused" Oct 30 17:16:04.434991 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:04.434945 4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" podUID="14805045-a727-41ba-8bd0-100d41f9f27d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:16:04.501986 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:04.501921 4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" podUID="b5abe703-fee0-4185-aa22-88d1db06392b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:16:04.578811 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:04.577946 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" podUID="e90f43d3-f5eb-45ab-af1e-340d0ce18966" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:16:04.678809 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:04.677945 4493 patch_prober.go:29] interesting pod/e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.45:15021/healthz/ready\": dial tcp 10.134.1.45:15021: connect: connection refused" start-of-body= Oct 30 17:16:04.678809 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:04.678017 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x" podUID="273838d6-034c-4763-a75f-5a357a4953f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.45:15021/healthz/ready\": dial tcp 10.134.1.45:15021: connect: connection refused" Oct 30 17:16:04.682808 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:04.679824 4493 patch_prober.go:29] interesting pod/e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.46:15021/healthz/ready\": dial tcp 10.134.1.46:15021: connect: connection refused" start-of-body= Oct 30 17:16:04.682808 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:04.679874 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt" podUID="46c8b953-69a7-4e2e-8695-a0b9d745529f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.46:15021/healthz/ready\": dial tcp 10.134.1.46:15021: connect: connection refused" Oct 30 17:16:05.022811 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:05.022617 4493 patch_prober.go:29] interesting pod/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.19:15021/healthz/ready\": dial tcp 10.134.1.19:15021: connect: connection refused" start-of-body= Oct 30 17:16:05.022811 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:05.022675 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6" podUID="ba3aa602-ca3b-4d37-9156-7c95b17a99cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.19:15021/healthz/ready\": dial tcp 10.134.1.19:15021: connect: connection refused" Oct 30 17:16:05.665195 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:05.665158 4493 patch_prober.go:29] interesting pod/e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.45:15021/healthz/ready\": dial tcp 10.134.1.45:15021: connect: connection refused" start-of-body= Oct 30 17:16:05.665834 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:05.665801 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x" podUID="273838d6-034c-4763-a75f-5a357a4953f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.45:15021/healthz/ready\": dial tcp 10.134.1.45:15021: connect: connection refused" Oct 30 17:16:05.714808 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:05.714506 4493 patch_prober.go:29] interesting pod/e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.46:15021/healthz/ready\": dial tcp 10.134.1.46:15021: connect: connection refused" start-of-body= Oct 30 17:16:05.714808 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:05.714564 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt" podUID="46c8b953-69a7-4e2e-8695-a0b9d745529f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.46:15021/healthz/ready\": dial tcp 10.134.1.46:15021: connect: connection refused" Oct 30 17:16:06.057862 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:06.000113 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6" Oct 30 17:16:06.489438 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:06.488035 4493 patch_prober.go:29] interesting pod/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.19:15021/healthz/ready\": dial tcp 10.134.1.19:15021: connect: connection refused" start-of-body= Oct 30 17:16:06.489438 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:06.488106 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6" podUID="ba3aa602-ca3b-4d37-9156-7c95b17a99cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.19:15021/healthz/ready\": dial tcp 10.134.1.19:15021: connect: connection refused" Oct 30 17:16:06.673824 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:06.668090 4493 patch_prober.go:29] interesting pod/e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.45:15021/healthz/ready\": dial tcp 10.134.1.45:15021: connect: connection refused" start-of-body= Oct 30 17:16:06.673824 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:06.668154 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x" podUID="273838d6-034c-4763-a75f-5a357a4953f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.45:15021/healthz/ready\": dial tcp 10.134.1.45:15021: connect: connection refused" Oct 30 17:16:06.700473 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:06.696914 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" podUID="e76172b1-659d-45bd-91c6-1a0ce4062335" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 17:16:06.720826 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:06.703495 4493 patch_prober.go:29] interesting pod/e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.46:15021/healthz/ready\": dial tcp 10.134.1.46:15021: connect: connection refused" start-of-body= Oct 30 17:16:06.720826 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:06.703582 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt" podUID="46c8b953-69a7-4e2e-8695-a0b9d745529f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.46:15021/healthz/ready\": dial tcp 10.134.1.46:15021: connect: connection refused" Oct 30 17:16:07.026958 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:07.026908 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" podUID="e43738f3-d55f-4c0a-a364-bf2ac76605c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:16:07.028163 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:07.028134 4493 patch_prober.go:29] interesting pod/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.19:15021/healthz/ready\": dial tcp 10.134.1.19:15021: connect: connection refused" start-of-body= Oct 30 17:16:07.028835 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:07.028805 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6" podUID="ba3aa602-ca3b-4d37-9156-7c95b17a99cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.19:15021/healthz/ready\": dial tcp 10.134.1.19:15021: connect: connection refused" Oct 30 17:16:07.234904 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:07.234204 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-24acb2/test-model-24acb2-kserve-946ff6bb7-h8cjs" podUID="cd1ff57b-1a55-4ef7-95f6-ee9389a00d99" containerName="main" probeResult="failure" output="Get \"https://10.134.1.20:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:16:07.355869 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:07.352881 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" podUID="14805045-a727-41ba-8bd0-100d41f9f27d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 17:16:07.355869 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:07.353799 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" podUID="0eadc5d4-98c3-4bf2-b3c2-f73fe249182f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 30 17:16:07.672981 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:07.668979 4493 patch_prober.go:29] interesting pod/e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.45:15021/healthz/ready\": dial tcp 10.134.1.45:15021: connect: connection refused" start-of-body= Oct 30 17:16:07.692121 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:07.692074 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x" podUID="273838d6-034c-4763-a75f-5a357a4953f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.45:15021/healthz/ready\": dial tcp 10.134.1.45:15021: connect: connection refused" Oct 30 17:16:07.744813 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:07.683109 4493 patch_prober.go:29] interesting pod/e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.46:15021/healthz/ready\": dial tcp 10.134.1.46:15021: connect: connection refused" start-of-body= Oct 30 17:16:07.744813 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:07.719015 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt" podUID="46c8b953-69a7-4e2e-8695-a0b9d745529f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.46:15021/healthz/ready\": dial tcp 10.134.1.46:15021: connect: connection refused" Oct 30 17:16:07.744813 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:07.722125 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" podUID="08baad08-cd72-4305-b92d-c11a84ef276f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 17:16:08.027113 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:08.027026 4493 patch_prober.go:29] interesting pod/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.19:15021/healthz/ready\": dial tcp 10.134.1.19:15021: connect: connection refused" start-of-body= Oct 30 17:16:08.027400 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:08.027358 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6" podUID="ba3aa602-ca3b-4d37-9156-7c95b17a99cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.19:15021/healthz/ready\": dial tcp 10.134.1.19:15021: connect: connection refused" Oct 30 17:16:08.502809 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:08.476297 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt" podStartSLOduration=7.476274697 podStartE2EDuration="7.476274697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:16:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:16:03.224085861 +0000 UTC m=+3190.451260962" watchObservedRunningTime="2026-10-30 17:16:08.476274697 +0000 UTC m=+3195.703449798" Oct 30 17:16:08.709551 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:08.709493 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" podUID="e90f43d3-f5eb-45ab-af1e-340d0ce18966" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Oct 30 17:16:08.743231 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:08.711640 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th"] Oct 30 17:16:08.796943 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:08.712278 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" podUID="e76172b1-659d-45bd-91c6-1a0ce4062335" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:16:08.797257 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:08.712598 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" podUID="18b72648-f523-405e-b5cd-1266ff13f8f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:16:08.967303 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:08.712647 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" podUID="b5abe703-fee0-4185-aa22-88d1db06392b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:16:09.102163 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:08.805459 4493 patch_prober.go:29] interesting pod/e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.46:15021/healthz/ready\": dial tcp 10.134.1.46:15021: connect: connection refused" start-of-body= Oct 30 17:16:09.164035 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:09.031708 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" podUID="e43738f3-d55f-4c0a-a364-bf2ac76605c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:16:09.165483 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:09.054404 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" podUID="08baad08-cd72-4305-b92d-c11a84ef276f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:16:09.166710 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:09.104757 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-24acb2/test-model-24acb2-kserve-946ff6bb7-h8cjs" podUID="cd1ff57b-1a55-4ef7-95f6-ee9389a00d99" containerName="main" probeResult="failure" output="Get \"https://10.134.1.20:8000/ready\": context deadline exceeded" Oct 30 17:16:09.216299 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:09.216240 4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" podUID="80b9bd9a-30d6-45ff-b318-26fcdc608e5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:16:09.246840 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:08.805991 4493 patch_prober.go:29] interesting pod/e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.45:15021/healthz/ready\": dial tcp 10.134.1.45:15021: connect: connection refused" start-of-body= Oct 30 17:16:09.246840 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:09.244985 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x" podUID="273838d6-034c-4763-a75f-5a357a4953f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.45:15021/healthz/ready\": dial tcp 10.134.1.45:15021: connect: connection refused" Oct 30 17:16:09.246840 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:09.034594 4493 patch_prober.go:29] interesting pod/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.19:15021/healthz/ready\": dial tcp 10.134.1.19:15021: connect: connection refused" start-of-body= Oct 30 17:16:09.246840 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:09.245054 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6" podUID="ba3aa602-ca3b-4d37-9156-7c95b17a99cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.19:15021/healthz/ready\": dial tcp 10.134.1.19:15021: connect: connection refused" Oct 30 17:16:09.308820 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:09.307761 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th"] Oct 30 17:16:09.309381 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:09.309359 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th" Oct 30 17:16:09.347884 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:09.347832 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66f731c4-e019-4f25-83d6-5c6519009981-istio-envoy\") pod \"e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th\" (UID: \"66f731c4-e019-4f25-83d6-5c6519009981\") " pod="openshift-ingress/e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th" Oct 30 17:16:09.354929 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:09.350979 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66f731c4-e019-4f25-83d6-5c6519009981-istio-token\") pod \"e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th\" (UID: \"66f731c4-e019-4f25-83d6-5c6519009981\") " pod="openshift-ingress/e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th" Oct 30 17:16:09.358298 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:09.358210 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" Oct 30 17:16:09.359727 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:09.359709 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-d80ebb-openshift-default-dockercfg-5gb5t\"" Oct 30 17:16:09.392980 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:09.365960 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" podUID="0eadc5d4-98c3-4bf2-b3c2-f73fe249182f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:16:09.392980 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:09.366015 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" podUID="14805045-a727-41ba-8bd0-100d41f9f27d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:16:09.651873 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:09.639998 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" podUID="f3e1c8ff-fcc3-4a1f-9730-c617d57b5193" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:16:09.651873 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:09.640058 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt" podUID="46c8b953-69a7-4e2e-8695-a0b9d745529f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.46:15021/healthz/ready\": dial tcp 10.134.1.46:15021: connect: connection refused" Oct 30 17:16:09.651873 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:09.648020 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66f731c4-e019-4f25-83d6-5c6519009981-istio-podinfo\") pod \"e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th\" (UID: \"66f731c4-e019-4f25-83d6-5c6519009981\") " pod="openshift-ingress/e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th" Oct 30 17:16:09.785934 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:09.785891 4493 patch_prober.go:29] interesting pod/e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.45:15021/healthz/ready\": dial tcp 10.134.1.45:15021: connect: connection refused" start-of-body= Oct 30 17:16:09.881586 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:09.881536 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x" podUID="273838d6-034c-4763-a75f-5a357a4953f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.45:15021/healthz/ready\": dial tcp 10.134.1.45:15021: connect: connection refused" Oct 30 17:16:09.898808 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:09.883656 4493 patch_prober.go:29] interesting pod/e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.46:15021/healthz/ready\": dial tcp 10.134.1.46:15021: connect: connection refused" start-of-body= Oct 30 17:16:09.901190 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:09.901129 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt" podUID="46c8b953-69a7-4e2e-8695-a0b9d745529f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.46:15021/healthz/ready\": dial tcp 10.134.1.46:15021: connect: connection refused" Oct 30 17:16:09.926905 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:09.903337 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66f731c4-e019-4f25-83d6-5c6519009981-istio-data\") pod \"e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th\" (UID: \"66f731c4-e019-4f25-83d6-5c6519009981\") " pod="openshift-ingress/e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th" Oct 30 17:16:09.926905 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:09.903469 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw76x\" (UniqueName: \"kubernetes.io/projected/66f731c4-e019-4f25-83d6-5c6519009981-kube-api-access-hw76x\") pod \"e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th\" (UID: \"66f731c4-e019-4f25-83d6-5c6519009981\") " pod="openshift-ingress/e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th" Oct 30 17:16:09.926905 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:09.903518 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66f731c4-e019-4f25-83d6-5c6519009981-istiod-ca-cert\") pod \"e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th\" (UID: \"66f731c4-e019-4f25-83d6-5c6519009981\") " pod="openshift-ingress/e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th" Oct 30 17:16:09.926905 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:09.903583 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66f731c4-e019-4f25-83d6-5c6519009981-workload-certs\") pod \"e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th\" (UID: \"66f731c4-e019-4f25-83d6-5c6519009981\") " pod="openshift-ingress/e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th" Oct 30 17:16:09.926905 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:09.903693 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66f731c4-e019-4f25-83d6-5c6519009981-workload-socket\") pod \"e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th\" (UID: \"66f731c4-e019-4f25-83d6-5c6519009981\") " pod="openshift-ingress/e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th" Oct 30 17:16:09.926905 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:09.903741 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66f731c4-e019-4f25-83d6-5c6519009981-credential-socket\") pod \"e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th\" (UID: \"66f731c4-e019-4f25-83d6-5c6519009981\") " pod="openshift-ingress/e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th" Oct 30 17:16:09.926905 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:09.908582 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" podUID="80b9bd9a-30d6-45ff-b318-26fcdc608e5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 17:16:09.926905 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:09.911701 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" podUID="e76172b1-659d-45bd-91c6-1a0ce4062335" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 17:16:09.926905 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:09.912414 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" Oct 30 17:16:09.945073 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:09.945031 4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" podUID="f3e1c8ff-fcc3-4a1f-9730-c617d57b5193" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 17:16:09.946007 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:09.945913 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" podUID="e90f43d3-f5eb-45ab-af1e-340d0ce18966" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:16:09.968815 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:09.965475 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" Oct 30 17:16:10.049693 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:10.026413 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66f731c4-e019-4f25-83d6-5c6519009981-workload-certs\") pod \"e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th\" (UID: \"66f731c4-e019-4f25-83d6-5c6519009981\") " pod="openshift-ingress/e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th" Oct 30 17:16:10.049693 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:10.026514 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66f731c4-e019-4f25-83d6-5c6519009981-workload-socket\") pod \"e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th\" (UID: \"66f731c4-e019-4f25-83d6-5c6519009981\") " pod="openshift-ingress/e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th" Oct 30 17:16:10.049693 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:10.026555 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66f731c4-e019-4f25-83d6-5c6519009981-credential-socket\") pod \"e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th\" (UID: \"66f731c4-e019-4f25-83d6-5c6519009981\") " pod="openshift-ingress/e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th" Oct 30 17:16:10.049693 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:10.039903 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66f731c4-e019-4f25-83d6-5c6519009981-workload-certs\") pod \"e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th\" (UID: \"66f731c4-e019-4f25-83d6-5c6519009981\") " pod="openshift-ingress/e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th" Oct 30 17:16:10.049693 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:10.040193 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66f731c4-e019-4f25-83d6-5c6519009981-workload-socket\") pod \"e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th\" (UID: \"66f731c4-e019-4f25-83d6-5c6519009981\") " pod="openshift-ingress/e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th" Oct 30 17:16:10.049693 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:10.040437 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66f731c4-e019-4f25-83d6-5c6519009981-credential-socket\") pod \"e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th\" (UID: \"66f731c4-e019-4f25-83d6-5c6519009981\") " pod="openshift-ingress/e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th" Oct 30 17:16:10.049693 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:10.042409 4493 patch_prober.go:29] interesting pod/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.19:15021/healthz/ready\": dial tcp 10.134.1.19:15021: connect: connection refused" start-of-body= Oct 30 17:16:10.049693 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:10.042456 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6" podUID="ba3aa602-ca3b-4d37-9156-7c95b17a99cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.19:15021/healthz/ready\": dial tcp 10.134.1.19:15021: connect: connection refused" Oct 30 17:16:10.105668 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:10.103925 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" podUID="b5abe703-fee0-4185-aa22-88d1db06392b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:16:10.111539 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:10.110307 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" Oct 30 17:16:10.176192 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:10.174074 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66f731c4-e019-4f25-83d6-5c6519009981-istio-envoy\") pod \"e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th\" (UID: \"66f731c4-e019-4f25-83d6-5c6519009981\") " pod="openshift-ingress/e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th" Oct 30 17:16:10.176192 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:10.174148 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66f731c4-e019-4f25-83d6-5c6519009981-istio-token\") pod \"e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th\" (UID: \"66f731c4-e019-4f25-83d6-5c6519009981\") " pod="openshift-ingress/e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th" Oct 30 17:16:10.176192 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:10.174240 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66f731c4-e019-4f25-83d6-5c6519009981-istio-podinfo\") pod \"e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th\" (UID: \"66f731c4-e019-4f25-83d6-5c6519009981\") " pod="openshift-ingress/e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th" Oct 30 17:16:10.176192 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:10.174404 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66f731c4-e019-4f25-83d6-5c6519009981-istio-data\") pod \"e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th\" (UID: \"66f731c4-e019-4f25-83d6-5c6519009981\") " pod="openshift-ingress/e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th" Oct 30 17:16:10.176192 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:10.174582 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw76x\" (UniqueName: \"kubernetes.io/projected/66f731c4-e019-4f25-83d6-5c6519009981-kube-api-access-hw76x\") pod \"e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th\" (UID: \"66f731c4-e019-4f25-83d6-5c6519009981\") " pod="openshift-ingress/e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th" Oct 30 17:16:10.176192 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:10.174679 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66f731c4-e019-4f25-83d6-5c6519009981-istiod-ca-cert\") pod \"e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th\" (UID: \"66f731c4-e019-4f25-83d6-5c6519009981\") " pod="openshift-ingress/e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th" Oct 30 17:16:10.241654 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:10.240530 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" Oct 30 17:16:10.241942 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:10.241850 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66f731c4-e019-4f25-83d6-5c6519009981-istio-data\") pod \"e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th\" (UID: \"66f731c4-e019-4f25-83d6-5c6519009981\") " pod="openshift-ingress/e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th" Oct 30 17:16:10.254590 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:10.254566 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" Oct 30 17:16:10.255132 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:10.255100 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" Oct 30 17:16:10.270380 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:10.266850 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66f731c4-e019-4f25-83d6-5c6519009981-istio-envoy\") pod \"e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th\" (UID: \"66f731c4-e019-4f25-83d6-5c6519009981\") " pod="openshift-ingress/e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th" Oct 30 17:16:10.270380 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:10.267335 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66f731c4-e019-4f25-83d6-5c6519009981-istio-podinfo\") pod \"e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th\" (UID: \"66f731c4-e019-4f25-83d6-5c6519009981\") " pod="openshift-ingress/e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th" Oct 30 17:16:10.270380 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:10.269378 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-24acb2/test-model-24acb2-kserve-946ff6bb7-h8cjs" podUID="cd1ff57b-1a55-4ef7-95f6-ee9389a00d99" containerName="main" probeResult="failure" output="Get \"https://10.134.1.20:8000/ready\": context deadline exceeded" Oct 30 17:16:10.283813 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:10.271210 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-24acb2/test-model-24acb2-kserve-946ff6bb7-h8cjs" Oct 30 17:16:10.283813 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:10.273992 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw76x\" (UniqueName: \"kubernetes.io/projected/66f731c4-e019-4f25-83d6-5c6519009981-kube-api-access-hw76x\") pod \"e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th\" (UID: \"66f731c4-e019-4f25-83d6-5c6519009981\") " pod="openshift-ingress/e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th" Oct 30 17:16:10.283813 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:10.274385 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" podUID="08baad08-cd72-4305-b92d-c11a84ef276f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:16:10.316821 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:10.286232 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66f731c4-e019-4f25-83d6-5c6519009981-istiod-ca-cert\") pod \"e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th\" (UID: \"66f731c4-e019-4f25-83d6-5c6519009981\") " pod="openshift-ingress/e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th" Oct 30 17:16:10.336902 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:10.329276 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-24acb2/test-model-24acb2-kserve-946ff6bb7-h8cjs" Oct 30 17:16:10.369847 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:10.369816 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66f731c4-e019-4f25-83d6-5c6519009981-istio-token\") pod \"e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th\" (UID: \"66f731c4-e019-4f25-83d6-5c6519009981\") " pod="openshift-ingress/e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th" Oct 30 17:16:10.402227 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:10.402190 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th" Oct 30 17:16:10.630459 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:10.606583 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" Oct 30 17:16:10.733273 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:10.728632 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt" Oct 30 17:16:10.733273 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:10.730177 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt" Oct 30 17:16:10.738456 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:10.738390 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt" Oct 30 17:16:10.818887 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:10.818835 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x" Oct 30 17:16:10.820885 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:10.820342 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x" Oct 30 17:16:10.823221 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:10.823202 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x" Oct 30 17:16:11.039380 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:11.038593 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th"] Oct 30 17:16:11.039380 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:11.039191 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6" Oct 30 17:16:11.045079 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:11.042417 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6" Oct 30 17:16:11.050597 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:11.050550 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:16:11.050725 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:11.050639 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:16:11.050725 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:11.050699 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:16:11.450553 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:11.450521 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th" event={"ID":"66f731c4-e019-4f25-83d6-5c6519009981","Type":"ContainerStarted","Data":"2d82cd31282f5633e31b54b6244e1dd4e244e1150a09557c59ea70c481e25ca0"} Oct 30 17:16:11.450877 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:11.450576 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th" event={"ID":"66f731c4-e019-4f25-83d6-5c6519009981","Type":"ContainerStarted","Data":"09fc9c10c117f8fdf7fe1af54ff0370cd578e509e5bed13e9078519875ef6303"} Oct 30 17:16:11.502363 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:11.502309 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th" podStartSLOduration=3.502294914 podStartE2EDuration="3.502294914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:16:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:16:11.498131003 +0000 UTC m=+3198.725306104" watchObservedRunningTime="2026-10-30 17:16:11.502294914 +0000 UTC m=+3198.729470011" Oct 30 17:16:12.444433 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:12.443094 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th" Oct 30 17:16:12.451767 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:12.450716 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th" Oct 30 17:16:12.461247 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:12.461217 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th" Oct 30 17:16:12.461854 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:12.461831 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th" Oct 30 17:16:30.349363 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:16:30.349317 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 17:16:54.189674 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:54.189641 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 17:16:57.459794 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:57.459740 4493 scope.go:122] "RemoveContainer" containerID="eaa2c70a4937e82a0fe5b38d21f84f8875dd56c24b73759954205343d6e6664c" Oct 30 17:16:57.471448 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:57.471427 4493 scope.go:122] "RemoveContainer" containerID="2c984d23330fc1b8a018652cba6542db6099cb0a4b8a62216991aa2a3752d601" Oct 30 17:16:57.481489 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:57.481467 4493 scope.go:122] "RemoveContainer" containerID="34b85aaf2ead0ef148d9cd134fb8803ef86053542a7c6fcdbc0d14ddfaf19c16" Oct 30 17:16:57.491980 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:57.491962 4493 scope.go:122] "RemoveContainer" containerID="6da98b64478f96b4f1ff533819db0c1919ec5a1b1be6999abb2fe4c9e43e591f" Oct 30 17:16:57.502626 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:57.502606 4493 scope.go:122] "RemoveContainer" containerID="21ef935358c22b5398eabf61afce23fbca38693f3aa0aee23c627cda24996036" Oct 30 17:16:57.515049 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:16:57.515026 4493 scope.go:122] "RemoveContainer" containerID="4ed64a9ddd5eb0bcd614c03f8cdda1a916a9e691006b83283206ad641dd7dfad" Oct 30 17:17:00.364463 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:17:00.364429 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 17:17:04.479067 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:04.479031 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fl6dw"] Oct 30 17:17:04.479512 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:04.479284 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fl6dw" podUID="15e78f14-6c3c-4bfa-93ea-bf58c888b372" containerName="istio-proxy" containerID="cri-o://fc739947d000efae787b77ed56d9a0cca572582d78a7b436981200dff3b53e84" gracePeriod=30 Oct 30 17:17:09.783044 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:09.782969 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fl6dw" Oct 30 17:17:09.795667 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:09.795639 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15e78f14-6c3c-4bfa-93ea-bf58c888b372-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15e78f14-6c3c-4bfa-93ea-bf58c888b372-istiod-ca-cert\") pod \"15e78f14-6c3c-4bfa-93ea-bf58c888b372\" (UID: \"15e78f14-6c3c-4bfa-93ea-bf58c888b372\") " Oct 30 17:17:09.795829 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:09.795687 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15e78f14-6c3c-4bfa-93ea-bf58c888b372-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15e78f14-6c3c-4bfa-93ea-bf58c888b372-workload-socket\") pod \"15e78f14-6c3c-4bfa-93ea-bf58c888b372\" (UID: \"15e78f14-6c3c-4bfa-93ea-bf58c888b372\") " Oct 30 17:17:09.795829 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:09.795772 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/15e78f14-6c3c-4bfa-93ea-bf58c888b372-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15e78f14-6c3c-4bfa-93ea-bf58c888b372-istio-podinfo\") pod \"15e78f14-6c3c-4bfa-93ea-bf58c888b372\" (UID: \"15e78f14-6c3c-4bfa-93ea-bf58c888b372\") " Oct 30 17:17:09.795933 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:09.795924 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15e78f14-6c3c-4bfa-93ea-bf58c888b372-kube-api-access-vcqbj\" (UniqueName: \"kubernetes.io/projected/15e78f14-6c3c-4bfa-93ea-bf58c888b372-kube-api-access-vcqbj\") pod \"15e78f14-6c3c-4bfa-93ea-bf58c888b372\" (UID: \"15e78f14-6c3c-4bfa-93ea-bf58c888b372\") " Oct 30 17:17:09.795984 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:09.795952 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15e78f14-6c3c-4bfa-93ea-bf58c888b372-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15e78f14-6c3c-4bfa-93ea-bf58c888b372-workload-certs\") pod \"15e78f14-6c3c-4bfa-93ea-bf58c888b372\" (UID: \"15e78f14-6c3c-4bfa-93ea-bf58c888b372\") " Oct 30 17:17:09.796047 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:09.795995 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15e78f14-6c3c-4bfa-93ea-bf58c888b372-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15e78f14-6c3c-4bfa-93ea-bf58c888b372-istio-envoy\") pod \"15e78f14-6c3c-4bfa-93ea-bf58c888b372\" (UID: \"15e78f14-6c3c-4bfa-93ea-bf58c888b372\") " Oct 30 17:17:09.796047 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:09.796005 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15e78f14-6c3c-4bfa-93ea-bf58c888b372-workload-socket" pod "15e78f14-6c3c-4bfa-93ea-bf58c888b372" (UID: "15e78f14-6c3c-4bfa-93ea-bf58c888b372"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:09.796047 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:09.796015 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15e78f14-6c3c-4bfa-93ea-bf58c888b372-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15e78f14-6c3c-4bfa-93ea-bf58c888b372-istio-data\") pod \"15e78f14-6c3c-4bfa-93ea-bf58c888b372\" (UID: \"15e78f14-6c3c-4bfa-93ea-bf58c888b372\") " Oct 30 17:17:09.796047 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:09.796033 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15e78f14-6c3c-4bfa-93ea-bf58c888b372-istiod-ca-cert" pod "15e78f14-6c3c-4bfa-93ea-bf58c888b372" (UID: "15e78f14-6c3c-4bfa-93ea-bf58c888b372"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:17:09.796274 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:09.796118 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15e78f14-6c3c-4bfa-93ea-bf58c888b372-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15e78f14-6c3c-4bfa-93ea-bf58c888b372-credential-socket\") pod \"15e78f14-6c3c-4bfa-93ea-bf58c888b372\" (UID: \"15e78f14-6c3c-4bfa-93ea-bf58c888b372\") " Oct 30 17:17:09.796274 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:09.796181 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15e78f14-6c3c-4bfa-93ea-bf58c888b372-istio-token\" (UniqueName: \"kubernetes.io/projected/15e78f14-6c3c-4bfa-93ea-bf58c888b372-istio-token\") pod \"15e78f14-6c3c-4bfa-93ea-bf58c888b372\" (UID: \"15e78f14-6c3c-4bfa-93ea-bf58c888b372\") " Oct 30 17:17:09.796274 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:09.796235 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15e78f14-6c3c-4bfa-93ea-bf58c888b372-istio-data" pod "15e78f14-6c3c-4bfa-93ea-bf58c888b372" (UID: "15e78f14-6c3c-4bfa-93ea-bf58c888b372"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:09.796274 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:09.796250 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15e78f14-6c3c-4bfa-93ea-bf58c888b372-workload-certs" pod "15e78f14-6c3c-4bfa-93ea-bf58c888b372" (UID: "15e78f14-6c3c-4bfa-93ea-bf58c888b372"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:09.796509 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:09.796380 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15e78f14-6c3c-4bfa-93ea-bf58c888b372-credential-socket" pod "15e78f14-6c3c-4bfa-93ea-bf58c888b372" (UID: "15e78f14-6c3c-4bfa-93ea-bf58c888b372"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:09.796810 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:09.796770 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15e78f14-6c3c-4bfa-93ea-bf58c888b372-workload-certs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:17:09.797027 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:09.796813 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15e78f14-6c3c-4bfa-93ea-bf58c888b372-istio-data\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:17:09.797027 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:09.796826 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15e78f14-6c3c-4bfa-93ea-bf58c888b372-credential-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:17:09.797027 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:09.796835 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15e78f14-6c3c-4bfa-93ea-bf58c888b372-istiod-ca-cert\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:17:09.797027 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:09.796843 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15e78f14-6c3c-4bfa-93ea-bf58c888b372-workload-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:17:09.798380 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:09.798346 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/15e78f14-6c3c-4bfa-93ea-bf58c888b372-istio-podinfo" pod "15e78f14-6c3c-4bfa-93ea-bf58c888b372" (UID: "15e78f14-6c3c-4bfa-93ea-bf58c888b372"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:17:09.798494 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:09.798416 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15e78f14-6c3c-4bfa-93ea-bf58c888b372-istio-envoy" pod "15e78f14-6c3c-4bfa-93ea-bf58c888b372" (UID: "15e78f14-6c3c-4bfa-93ea-bf58c888b372"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:09.798494 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:09.798440 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15e78f14-6c3c-4bfa-93ea-bf58c888b372-kube-api-access-vcqbj" pod "15e78f14-6c3c-4bfa-93ea-bf58c888b372" (UID: "15e78f14-6c3c-4bfa-93ea-bf58c888b372"). InnerVolumeSpecName "kube-api-access-vcqbj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:17:09.798615 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:09.798598 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15e78f14-6c3c-4bfa-93ea-bf58c888b372-istio-token" pod "15e78f14-6c3c-4bfa-93ea-bf58c888b372" (UID: "15e78f14-6c3c-4bfa-93ea-bf58c888b372"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:17:09.898339 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:09.898298 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcqbj\" (UniqueName: \"kubernetes.io/projected/15e78f14-6c3c-4bfa-93ea-bf58c888b372-kube-api-access-vcqbj\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:17:09.898339 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:09.898339 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15e78f14-6c3c-4bfa-93ea-bf58c888b372-istio-envoy\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:17:09.898621 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:09.898352 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15e78f14-6c3c-4bfa-93ea-bf58c888b372-istio-token\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:17:09.898621 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:09.898365 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15e78f14-6c3c-4bfa-93ea-bf58c888b372-istio-podinfo\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:17:10.552445 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:10.552406 4493 generic.go:356] "Generic (PLEG): container finished" podID="15e78f14-6c3c-4bfa-93ea-bf58c888b372" containerID="fc739947d000efae787b77ed56d9a0cca572582d78a7b436981200dff3b53e84" exitCode=0 Oct 30 17:17:10.552659 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:10.552453 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fl6dw" Oct 30 17:17:10.552659 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:10.552492 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fl6dw" event={"ID":"15e78f14-6c3c-4bfa-93ea-bf58c888b372","Type":"ContainerDied","Data":"fc739947d000efae787b77ed56d9a0cca572582d78a7b436981200dff3b53e84"} Oct 30 17:17:10.552659 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:10.552558 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fl6dw" event={"ID":"15e78f14-6c3c-4bfa-93ea-bf58c888b372","Type":"ContainerDied","Data":"c1459ad7364fc96ddb40e3e8f11d277242754a5ec09dc6d4241e3b39c882ab4e"} Oct 30 17:17:10.552659 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:10.552603 4493 scope.go:122] "RemoveContainer" containerID="fc739947d000efae787b77ed56d9a0cca572582d78a7b436981200dff3b53e84" Oct 30 17:17:10.565470 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:10.565452 4493 scope.go:122] "RemoveContainer" containerID="fc739947d000efae787b77ed56d9a0cca572582d78a7b436981200dff3b53e84" Oct 30 17:17:10.565774 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:17:10.565755 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc739947d000efae787b77ed56d9a0cca572582d78a7b436981200dff3b53e84\": container with ID starting with fc739947d000efae787b77ed56d9a0cca572582d78a7b436981200dff3b53e84 not found: ID does not exist" containerID="fc739947d000efae787b77ed56d9a0cca572582d78a7b436981200dff3b53e84" Oct 30 17:17:10.565849 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:10.565814 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc739947d000efae787b77ed56d9a0cca572582d78a7b436981200dff3b53e84"} err="failed to get container status \"fc739947d000efae787b77ed56d9a0cca572582d78a7b436981200dff3b53e84\": rpc error: code = NotFound desc = could not find container \"fc739947d000efae787b77ed56d9a0cca572582d78a7b436981200dff3b53e84\": container with ID starting with fc739947d000efae787b77ed56d9a0cca572582d78a7b436981200dff3b53e84 not found: ID does not exist" Oct 30 17:17:10.622963 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:10.622909 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fl6dw"] Oct 30 17:17:10.635147 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:10.635110 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-fl6dw"] Oct 30 17:17:10.938059 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:10.938020 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15e78f14-6c3c-4bfa-93ea-bf58c888b372" path="/var/lib/kubelet/pods/15e78f14-6c3c-4bfa-93ea-bf58c888b372/volumes" Oct 30 17:17:27.976962 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:27.976918 4493 patch_prober.go:29] interesting pod/e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:17:27.977415 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:27.976996 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk" podUID="e934b3f5-e168-4760-9ea6-3b9cab799328" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:17:30.350634 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:17:30.350539 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 17:17:30.842875 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:30.842831 4493 generic.go:356] "Generic (PLEG): container finished" podID="e934b3f5-e168-4760-9ea6-3b9cab799328" containerID="2847df9bcd14d1409887a15fb491be9a31a9ff83b9f5ee7e83ca6d7948d700d8" exitCode=0 Oct 30 17:17:30.843089 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:30.842951 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk" event={"ID":"e934b3f5-e168-4760-9ea6-3b9cab799328","Type":"ContainerDied","Data":"2847df9bcd14d1409887a15fb491be9a31a9ff83b9f5ee7e83ca6d7948d700d8"} Oct 30 17:17:30.843550 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:30.843525 4493 scope.go:122] "RemoveContainer" containerID="2847df9bcd14d1409887a15fb491be9a31a9ff83b9f5ee7e83ca6d7948d700d8" Oct 30 17:17:30.846036 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:30.846002 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:17:30.846163 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:30.846096 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:17:30.846163 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:30.846147 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:17:31.855240 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:31.855202 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk" event={"ID":"e934b3f5-e168-4760-9ea6-3b9cab799328","Type":"ContainerStarted","Data":"53b1681933dfad608e75adebf0a3b5e227685beec5a62059d65ef8dcc36d2c56"} Oct 30 17:17:31.877392 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:31.877364 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk" Oct 30 17:17:31.877585 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:31.877565 4493 patch_prober.go:29] interesting pod/e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.44:15021/healthz/ready\": dial tcp 10.134.1.44:15021: connect: connection refused" start-of-body= Oct 30 17:17:31.877638 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:31.877613 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk" podUID="e934b3f5-e168-4760-9ea6-3b9cab799328" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.44:15021/healthz/ready\": dial tcp 10.134.1.44:15021: connect: connection refused" Oct 30 17:17:32.877309 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:32.877273 4493 patch_prober.go:29] interesting pod/e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.44:15021/healthz/ready\": dial tcp 10.134.1.44:15021: connect: connection refused" start-of-body= Oct 30 17:17:32.877905 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:32.877321 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk" podUID="e934b3f5-e168-4760-9ea6-3b9cab799328" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.44:15021/healthz/ready\": dial tcp 10.134.1.44:15021: connect: connection refused" Oct 30 17:17:33.877161 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:33.877129 4493 patch_prober.go:29] interesting pod/e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.44:15021/healthz/ready\": dial tcp 10.134.1.44:15021: connect: connection refused" start-of-body= Oct 30 17:17:33.877351 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:33.877179 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk" podUID="e934b3f5-e168-4760-9ea6-3b9cab799328" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.44:15021/healthz/ready\": dial tcp 10.134.1.44:15021: connect: connection refused" Oct 30 17:17:34.877049 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:34.877013 4493 patch_prober.go:29] interesting pod/e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.44:15021/healthz/ready\": dial tcp 10.134.1.44:15021: connect: connection refused" start-of-body= Oct 30 17:17:34.877238 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:34.877067 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk" podUID="e934b3f5-e168-4760-9ea6-3b9cab799328" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.44:15021/healthz/ready\": dial tcp 10.134.1.44:15021: connect: connection refused" Oct 30 17:17:35.877554 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:35.877519 4493 patch_prober.go:29] interesting pod/e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.44:15021/healthz/ready\": dial tcp 10.134.1.44:15021: connect: connection refused" start-of-body= Oct 30 17:17:35.878089 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:35.877572 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk" podUID="e934b3f5-e168-4760-9ea6-3b9cab799328" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.44:15021/healthz/ready\": dial tcp 10.134.1.44:15021: connect: connection refused" Oct 30 17:17:36.877931 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:36.877893 4493 patch_prober.go:29] interesting pod/e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.44:15021/healthz/ready\": dial tcp 10.134.1.44:15021: connect: connection refused" start-of-body= Oct 30 17:17:36.878365 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:36.877952 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk" podUID="e934b3f5-e168-4760-9ea6-3b9cab799328" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.44:15021/healthz/ready\": dial tcp 10.134.1.44:15021: connect: connection refused" Oct 30 17:17:37.882757 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:37.882727 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk" Oct 30 17:17:37.883263 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:37.883163 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk" Oct 30 17:17:37.883755 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:37.883732 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk" Oct 30 17:17:43.217825 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:43.216749 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 17:17:54.886537 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:54.886511 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-56v9s_d3a1c417-2698-4462-b7ff-1d2e66d7a10f/ovn-acl-logging/0.log" Oct 30 17:17:54.892878 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:54.892856 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-56v9s_d3a1c417-2698-4462-b7ff-1d2e66d7a10f/ovn-acl-logging/0.log" Oct 30 17:17:57.217512 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:57.217464 4493 generic.go:356] "Generic (PLEG): container finished" podID="66f731c4-e019-4f25-83d6-5c6519009981" containerID="2d82cd31282f5633e31b54b6244e1dd4e244e1150a09557c59ea70c481e25ca0" exitCode=0 Oct 30 17:17:57.218094 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:57.217607 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th" event={"ID":"66f731c4-e019-4f25-83d6-5c6519009981","Type":"ContainerDied","Data":"2d82cd31282f5633e31b54b6244e1dd4e244e1150a09557c59ea70c481e25ca0"} Oct 30 17:17:57.221806 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:57.218564 4493 scope.go:122] "RemoveContainer" containerID="2d82cd31282f5633e31b54b6244e1dd4e244e1150a09557c59ea70c481e25ca0" Oct 30 17:17:57.221806 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:57.221699 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:17:57.234912 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:57.230622 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:17:57.234912 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:57.230701 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:17:57.712571 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:57.466992 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th" Oct 30 17:17:58.253991 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:58.253944 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th" event={"ID":"66f731c4-e019-4f25-83d6-5c6519009981","Type":"ContainerStarted","Data":"be3349f6bb5e573c5ef3c308300d19f26166b5d8c9f8eb3800b81343985a567d"} Oct 30 17:17:58.452245 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:58.452213 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th" Oct 30 17:17:58.452609 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:58.452588 4493 patch_prober.go:29] interesting pod/e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.47:15021/healthz/ready\": dial tcp 10.134.1.47:15021: connect: connection refused" start-of-body= Oct 30 17:17:58.452814 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:58.452772 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th" podUID="66f731c4-e019-4f25-83d6-5c6519009981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.47:15021/healthz/ready\": dial tcp 10.134.1.47:15021: connect: connection refused" Oct 30 17:17:59.445191 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:59.445156 4493 patch_prober.go:29] interesting pod/e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.47:15021/healthz/ready\": dial tcp 10.134.1.47:15021: connect: connection refused" start-of-body= Oct 30 17:17:59.445765 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:59.445737 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th" podUID="66f731c4-e019-4f25-83d6-5c6519009981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.47:15021/healthz/ready\": dial tcp 10.134.1.47:15021: connect: connection refused" Oct 30 17:17:59.850861 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:59.850742 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-6e08bd/test-model-6e08bd-kserve-79f874df7-qk2h6"] Oct 30 17:17:59.852236 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:59.852194 4493 cpu_manager.go:414] "containerMap: removing container" podUID="15e78f14-6c3c-4bfa-93ea-bf58c888b372" containerName="istio-proxy" Oct 30 17:17:59.852413 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:59.852397 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15e78f14-6c3c-4bfa-93ea-bf58c888b372" containerName="istio-proxy" Oct 30 17:17:59.852694 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:59.852679 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="15e78f14-6c3c-4bfa-93ea-bf58c888b372" containerName="istio-proxy" Oct 30 17:17:59.859247 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:59.857972 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6e08bd/test-model-6e08bd-kserve-79f874df7-qk2h6" Oct 30 17:17:59.879693 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:59.879633 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-6e08bd/test-model-6e08bd-kserve-79f874df7-qk2h6"] Oct 30 17:17:59.883552 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:59.883344 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-6e08bd\"/\"test-model-6e08bd-kserve-self-signed-certs\"" Oct 30 17:17:59.883552 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:59.883481 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-6e08bd\"/\"default-dockercfg-fl6hf\"" Oct 30 17:17:59.885603 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:59.885490 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-6e08bd\"/\"openshift-service-ca.crt\"" Oct 30 17:17:59.885603 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:59.885575 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-6e08bd\"/\"kube-root-ca.crt\"" Oct 30 17:17:59.984197 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:59.984151 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877-model-cache\") pod \"test-model-6e08bd-kserve-79f874df7-qk2h6\" (UID: \"c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877\") " pod="ai-tenant-e2e-inf-a-6e08bd/test-model-6e08bd-kserve-79f874df7-qk2h6" Oct 30 17:17:59.984489 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:59.984454 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwjnb\" (UniqueName: \"kubernetes.io/projected/c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877-kube-api-access-mwjnb\") pod \"test-model-6e08bd-kserve-79f874df7-qk2h6\" (UID: \"c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877\") " pod="ai-tenant-e2e-inf-a-6e08bd/test-model-6e08bd-kserve-79f874df7-qk2h6" Oct 30 17:17:59.984694 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:59.984677 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877-tmp-dir\") pod \"test-model-6e08bd-kserve-79f874df7-qk2h6\" (UID: \"c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877\") " pod="ai-tenant-e2e-inf-a-6e08bd/test-model-6e08bd-kserve-79f874df7-qk2h6" Oct 30 17:17:59.984910 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:59.984881 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877-tls-certs\") pod \"test-model-6e08bd-kserve-79f874df7-qk2h6\" (UID: \"c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877\") " pod="ai-tenant-e2e-inf-a-6e08bd/test-model-6e08bd-kserve-79f874df7-qk2h6" Oct 30 17:17:59.985111 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:59.985083 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877-home\") pod \"test-model-6e08bd-kserve-79f874df7-qk2h6\" (UID: \"c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877\") " pod="ai-tenant-e2e-inf-a-6e08bd/test-model-6e08bd-kserve-79f874df7-qk2h6" Oct 30 17:17:59.985646 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:17:59.985381 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877-dshm\") pod \"test-model-6e08bd-kserve-79f874df7-qk2h6\" (UID: \"c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877\") " pod="ai-tenant-e2e-inf-a-6e08bd/test-model-6e08bd-kserve-79f874df7-qk2h6" Oct 30 17:18:00.087368 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:00.087327 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877-tmp-dir\") pod \"test-model-6e08bd-kserve-79f874df7-qk2h6\" (UID: \"c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877\") " pod="ai-tenant-e2e-inf-a-6e08bd/test-model-6e08bd-kserve-79f874df7-qk2h6" Oct 30 17:18:00.087578 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:00.087434 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877-tls-certs\") pod \"test-model-6e08bd-kserve-79f874df7-qk2h6\" (UID: \"c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877\") " pod="ai-tenant-e2e-inf-a-6e08bd/test-model-6e08bd-kserve-79f874df7-qk2h6" Oct 30 17:18:00.087578 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:00.087484 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877-home\") pod \"test-model-6e08bd-kserve-79f874df7-qk2h6\" (UID: \"c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877\") " pod="ai-tenant-e2e-inf-a-6e08bd/test-model-6e08bd-kserve-79f874df7-qk2h6" Oct 30 17:18:00.087709 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:00.087638 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877-dshm\") pod \"test-model-6e08bd-kserve-79f874df7-qk2h6\" (UID: \"c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877\") " pod="ai-tenant-e2e-inf-a-6e08bd/test-model-6e08bd-kserve-79f874df7-qk2h6" Oct 30 17:18:00.087873 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:00.087839 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877-model-cache\") pod \"test-model-6e08bd-kserve-79f874df7-qk2h6\" (UID: \"c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877\") " pod="ai-tenant-e2e-inf-a-6e08bd/test-model-6e08bd-kserve-79f874df7-qk2h6" Oct 30 17:18:00.088022 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:00.087877 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwjnb\" (UniqueName: \"kubernetes.io/projected/c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877-kube-api-access-mwjnb\") pod \"test-model-6e08bd-kserve-79f874df7-qk2h6\" (UID: \"c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877\") " pod="ai-tenant-e2e-inf-a-6e08bd/test-model-6e08bd-kserve-79f874df7-qk2h6" Oct 30 17:18:00.088650 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:00.088234 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877-home\") pod \"test-model-6e08bd-kserve-79f874df7-qk2h6\" (UID: \"c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877\") " pod="ai-tenant-e2e-inf-a-6e08bd/test-model-6e08bd-kserve-79f874df7-qk2h6" Oct 30 17:18:00.088650 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:00.088517 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877-model-cache\") pod \"test-model-6e08bd-kserve-79f874df7-qk2h6\" (UID: \"c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877\") " pod="ai-tenant-e2e-inf-a-6e08bd/test-model-6e08bd-kserve-79f874df7-qk2h6" Oct 30 17:18:00.088971 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:00.088950 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877-tmp-dir\") pod \"test-model-6e08bd-kserve-79f874df7-qk2h6\" (UID: \"c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877\") " pod="ai-tenant-e2e-inf-a-6e08bd/test-model-6e08bd-kserve-79f874df7-qk2h6" Oct 30 17:18:00.091193 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:00.091171 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877-dshm\") pod \"test-model-6e08bd-kserve-79f874df7-qk2h6\" (UID: \"c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877\") " pod="ai-tenant-e2e-inf-a-6e08bd/test-model-6e08bd-kserve-79f874df7-qk2h6" Oct 30 17:18:00.091908 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:00.091885 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877-tls-certs\") pod \"test-model-6e08bd-kserve-79f874df7-qk2h6\" (UID: \"c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877\") " pod="ai-tenant-e2e-inf-a-6e08bd/test-model-6e08bd-kserve-79f874df7-qk2h6" Oct 30 17:18:00.116920 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:00.116825 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwjnb\" (UniqueName: \"kubernetes.io/projected/c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877-kube-api-access-mwjnb\") pod \"test-model-6e08bd-kserve-79f874df7-qk2h6\" (UID: \"c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877\") " pod="ai-tenant-e2e-inf-a-6e08bd/test-model-6e08bd-kserve-79f874df7-qk2h6" Oct 30 17:18:00.200971 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:00.200127 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6e08bd/test-model-6e08bd-kserve-79f874df7-qk2h6" Oct 30 17:18:00.463823 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:00.461416 4493 patch_prober.go:29] interesting pod/e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.47:15021/healthz/ready\": dial tcp 10.134.1.47:15021: connect: connection refused" start-of-body= Oct 30 17:18:00.463823 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:00.461478 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th" podUID="66f731c4-e019-4f25-83d6-5c6519009981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.47:15021/healthz/ready\": dial tcp 10.134.1.47:15021: connect: connection refused" Oct 30 17:18:00.523836 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:18:00.523010 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 17:18:00.936872 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:00.936249 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-6e08bd/test-model-6e08bd-kserve-79f874df7-qk2h6"] Oct 30 17:18:01.374849 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:01.374749 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6e08bd/test-model-6e08bd-kserve-79f874df7-qk2h6" event={"ID":"c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877","Type":"ContainerStarted","Data":"5881d7dafc5e1770cf00f1105b50e3e1cfb66e30a344cbc19bd6ba3d0d8f561e"} Oct 30 17:18:01.374849 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:01.374833 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6e08bd/test-model-6e08bd-kserve-79f874df7-qk2h6" event={"ID":"c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877","Type":"ContainerStarted","Data":"0b514b9cc78b12e56346b8fa3e2cfecbd7f4f03fb4eca42b3ccbe332e44c0750"} Oct 30 17:18:01.443199 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:01.443156 4493 patch_prober.go:29] interesting pod/e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.47:15021/healthz/ready\": dial tcp 10.134.1.47:15021: connect: connection refused" start-of-body= Oct 30 17:18:01.443411 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:01.443233 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th" podUID="66f731c4-e019-4f25-83d6-5c6519009981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.47:15021/healthz/ready\": dial tcp 10.134.1.47:15021: connect: connection refused" Oct 30 17:18:01.444375 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:01.444328 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-6e08bd/test-model-6e08bd-kserve-79f874df7-qk2h6" podStartSLOduration=2.444314016 podStartE2EDuration="2.444314016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:17:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:18:01.424205462 +0000 UTC m=+3308.651380563" watchObservedRunningTime="2026-10-30 17:18:01.444314016 +0000 UTC m=+3308.671489115" Oct 30 17:18:02.201195 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:02.201158 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-6e08bd/test-model-6e08bd-kserve-79f874df7-qk2h6" Oct 30 17:18:02.444132 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:02.444087 4493 patch_prober.go:29] interesting pod/e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.47:15021/healthz/ready\": dial tcp 10.134.1.47:15021: connect: connection refused" start-of-body= Oct 30 17:18:02.444341 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:02.444147 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th" podUID="66f731c4-e019-4f25-83d6-5c6519009981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.47:15021/healthz/ready\": dial tcp 10.134.1.47:15021: connect: connection refused" Oct 30 17:18:03.443372 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:03.443339 4493 patch_prober.go:29] interesting pod/e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.47:15021/healthz/ready\": dial tcp 10.134.1.47:15021: connect: connection refused" start-of-body= Oct 30 17:18:03.444036 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:03.444003 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th" podUID="66f731c4-e019-4f25-83d6-5c6519009981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.47:15021/healthz/ready\": dial tcp 10.134.1.47:15021: connect: connection refused" Oct 30 17:18:04.451190 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:04.451155 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th" Oct 30 17:18:04.451771 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:04.451714 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th" Oct 30 17:18:04.452687 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:04.452665 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th" Oct 30 17:18:07.444908 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:07.444871 4493 generic.go:356] "Generic (PLEG): container finished" podID="18c895a8-b0d7-425c-81eb-282dd1766d59" containerID="1b94d423f4e040453995404381a7c43ce5b63dde548a6758b4e1811a815c94e1" exitCode=0 Oct 30 17:18:07.445385 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:07.444944 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x" event={"ID":"18c895a8-b0d7-425c-81eb-282dd1766d59","Type":"ContainerDied","Data":"1b94d423f4e040453995404381a7c43ce5b63dde548a6758b4e1811a815c94e1"} Oct 30 17:18:07.445456 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:07.445441 4493 scope.go:122] "RemoveContainer" containerID="1b94d423f4e040453995404381a7c43ce5b63dde548a6758b4e1811a815c94e1" Oct 30 17:18:07.449027 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:07.448983 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:18:07.449143 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:07.449077 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:18:07.449143 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:07.449122 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:18:08.458079 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:08.458042 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x" event={"ID":"18c895a8-b0d7-425c-81eb-282dd1766d59","Type":"ContainerStarted","Data":"9b6d7c415fb7a23bb4445a3949732b1afcd83c421b71f8bb96ab82d02f129d21"} Oct 30 17:18:08.761593 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:08.761495 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x" Oct 30 17:18:08.762973 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:08.762945 4493 patch_prober.go:29] interesting pod/e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.43:15021/healthz/ready\": dial tcp 10.134.1.43:15021: connect: connection refused" start-of-body= Oct 30 17:18:08.763111 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:08.762994 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x" podUID="18c895a8-b0d7-425c-81eb-282dd1766d59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.43:15021/healthz/ready\": dial tcp 10.134.1.43:15021: connect: connection refused" Oct 30 17:18:09.760868 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:09.760831 4493 patch_prober.go:29] interesting pod/e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.43:15021/healthz/ready\": dial tcp 10.134.1.43:15021: connect: connection refused" start-of-body= Oct 30 17:18:09.761266 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:09.760893 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x" podUID="18c895a8-b0d7-425c-81eb-282dd1766d59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.43:15021/healthz/ready\": dial tcp 10.134.1.43:15021: connect: connection refused" Oct 30 17:18:09.879804 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:09.879739 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x" Oct 30 17:18:10.203105 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:10.203074 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-6e08bd/test-model-6e08bd-kserve-79f874df7-qk2h6" Oct 30 17:18:10.250714 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:10.250682 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-6e08bd/test-model-6e08bd-kserve-79f874df7-qk2h6" Oct 30 17:18:10.486434 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:10.486358 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-6e08bd/test-model-6e08bd-kserve-79f874df7-qk2h6" Oct 30 17:18:10.761432 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:10.761343 4493 patch_prober.go:29] interesting pod/e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.43:15021/healthz/ready\": dial tcp 10.134.1.43:15021: connect: connection refused" start-of-body= Oct 30 17:18:10.761432 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:10.761392 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x" podUID="18c895a8-b0d7-425c-81eb-282dd1766d59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.43:15021/healthz/ready\": dial tcp 10.134.1.43:15021: connect: connection refused" Oct 30 17:18:11.760754 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:11.760714 4493 patch_prober.go:29] interesting pod/e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.43:15021/healthz/ready\": dial tcp 10.134.1.43:15021: connect: connection refused" start-of-body= Oct 30 17:18:11.760980 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:11.760768 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x" podUID="18c895a8-b0d7-425c-81eb-282dd1766d59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.43:15021/healthz/ready\": dial tcp 10.134.1.43:15021: connect: connection refused" Oct 30 17:18:12.762808 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:12.761725 4493 patch_prober.go:29] interesting pod/e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.43:15021/healthz/ready\": dial tcp 10.134.1.43:15021: connect: connection refused" start-of-body= Oct 30 17:18:12.762808 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:12.761848 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x" podUID="18c895a8-b0d7-425c-81eb-282dd1766d59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.43:15021/healthz/ready\": dial tcp 10.134.1.43:15021: connect: connection refused" Oct 30 17:18:13.761490 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:13.761455 4493 patch_prober.go:29] interesting pod/e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.43:15021/healthz/ready\": dial tcp 10.134.1.43:15021: connect: connection refused" start-of-body= Oct 30 17:18:13.761685 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:13.761522 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x" podUID="18c895a8-b0d7-425c-81eb-282dd1766d59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.43:15021/healthz/ready\": dial tcp 10.134.1.43:15021: connect: connection refused" Oct 30 17:18:14.767388 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:14.767357 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x" Oct 30 17:18:14.768552 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:14.768524 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x" Oct 30 17:18:17.661589 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:17.661548 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4"] Oct 30 17:18:17.677451 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:17.677412 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4"] Oct 30 17:18:17.677639 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:17.677530 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4" Oct 30 17:18:17.866811 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:17.843384 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-0b09d3-openshift-default-dockercfg-fkwt8\"" Oct 30 17:18:17.936158 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:17.935902 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a18c02c8-1a51-4dcf-aeef-696acb2d2280-istio-podinfo\") pod \"e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4\" (UID: \"a18c02c8-1a51-4dcf-aeef-696acb2d2280\") " pod="openshift-ingress/e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4" Oct 30 17:18:17.936531 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:17.936507 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a18c02c8-1a51-4dcf-aeef-696acb2d2280-workload-socket\") pod \"e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4\" (UID: \"a18c02c8-1a51-4dcf-aeef-696acb2d2280\") " pod="openshift-ingress/e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4" Oct 30 17:18:17.940899 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:17.940875 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a18c02c8-1a51-4dcf-aeef-696acb2d2280-istio-data\") pod \"e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4\" (UID: \"a18c02c8-1a51-4dcf-aeef-696acb2d2280\") " pod="openshift-ingress/e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4" Oct 30 17:18:17.959154 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:17.959126 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a18c02c8-1a51-4dcf-aeef-696acb2d2280-istiod-ca-cert\") pod \"e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4\" (UID: \"a18c02c8-1a51-4dcf-aeef-696acb2d2280\") " pod="openshift-ingress/e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4" Oct 30 17:18:17.959407 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:17.959388 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp9nt\" (UniqueName: \"kubernetes.io/projected/a18c02c8-1a51-4dcf-aeef-696acb2d2280-kube-api-access-rp9nt\") pod \"e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4\" (UID: \"a18c02c8-1a51-4dcf-aeef-696acb2d2280\") " pod="openshift-ingress/e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4" Oct 30 17:18:17.959583 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:17.959567 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a18c02c8-1a51-4dcf-aeef-696acb2d2280-credential-socket\") pod \"e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4\" (UID: \"a18c02c8-1a51-4dcf-aeef-696acb2d2280\") " pod="openshift-ingress/e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4" Oct 30 17:18:17.959739 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:17.959725 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a18c02c8-1a51-4dcf-aeef-696acb2d2280-istio-token\") pod \"e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4\" (UID: \"a18c02c8-1a51-4dcf-aeef-696acb2d2280\") " pod="openshift-ingress/e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4" Oct 30 17:18:17.960150 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:17.960127 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a18c02c8-1a51-4dcf-aeef-696acb2d2280-workload-certs\") pod \"e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4\" (UID: \"a18c02c8-1a51-4dcf-aeef-696acb2d2280\") " pod="openshift-ingress/e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4" Oct 30 17:18:17.960345 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:17.960327 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a18c02c8-1a51-4dcf-aeef-696acb2d2280-istio-envoy\") pod \"e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4\" (UID: \"a18c02c8-1a51-4dcf-aeef-696acb2d2280\") " pod="openshift-ingress/e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4" Oct 30 17:18:18.063197 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:18.063156 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a18c02c8-1a51-4dcf-aeef-696acb2d2280-istio-podinfo\") pod \"e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4\" (UID: \"a18c02c8-1a51-4dcf-aeef-696acb2d2280\") " pod="openshift-ingress/e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4" Oct 30 17:18:18.063410 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:18.063223 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a18c02c8-1a51-4dcf-aeef-696acb2d2280-workload-socket\") pod \"e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4\" (UID: \"a18c02c8-1a51-4dcf-aeef-696acb2d2280\") " pod="openshift-ingress/e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4" Oct 30 17:18:18.063410 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:18.063285 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a18c02c8-1a51-4dcf-aeef-696acb2d2280-istio-data\") pod \"e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4\" (UID: \"a18c02c8-1a51-4dcf-aeef-696acb2d2280\") " pod="openshift-ingress/e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4" Oct 30 17:18:18.063410 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:18.063347 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a18c02c8-1a51-4dcf-aeef-696acb2d2280-istiod-ca-cert\") pod \"e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4\" (UID: \"a18c02c8-1a51-4dcf-aeef-696acb2d2280\") " pod="openshift-ingress/e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4" Oct 30 17:18:18.063603 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:18.063462 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp9nt\" (UniqueName: \"kubernetes.io/projected/a18c02c8-1a51-4dcf-aeef-696acb2d2280-kube-api-access-rp9nt\") pod \"e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4\" (UID: \"a18c02c8-1a51-4dcf-aeef-696acb2d2280\") " pod="openshift-ingress/e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4" Oct 30 17:18:18.063603 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:18.063534 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a18c02c8-1a51-4dcf-aeef-696acb2d2280-credential-socket\") pod \"e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4\" (UID: \"a18c02c8-1a51-4dcf-aeef-696acb2d2280\") " pod="openshift-ingress/e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4" Oct 30 17:18:18.063603 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:18.063588 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a18c02c8-1a51-4dcf-aeef-696acb2d2280-istio-token\") pod \"e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4\" (UID: \"a18c02c8-1a51-4dcf-aeef-696acb2d2280\") " pod="openshift-ingress/e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4" Oct 30 17:18:18.063771 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:18.063624 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a18c02c8-1a51-4dcf-aeef-696acb2d2280-workload-certs\") pod \"e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4\" (UID: \"a18c02c8-1a51-4dcf-aeef-696acb2d2280\") " pod="openshift-ingress/e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4" Oct 30 17:18:18.063771 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:18.063688 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a18c02c8-1a51-4dcf-aeef-696acb2d2280-istio-envoy\") pod \"e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4\" (UID: \"a18c02c8-1a51-4dcf-aeef-696acb2d2280\") " pod="openshift-ingress/e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4" Oct 30 17:18:18.089726 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:18.078593 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a18c02c8-1a51-4dcf-aeef-696acb2d2280-workload-socket\") pod \"e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4\" (UID: \"a18c02c8-1a51-4dcf-aeef-696acb2d2280\") " pod="openshift-ingress/e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4" Oct 30 17:18:18.089726 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:18.078886 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a18c02c8-1a51-4dcf-aeef-696acb2d2280-istio-data\") pod \"e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4\" (UID: \"a18c02c8-1a51-4dcf-aeef-696acb2d2280\") " pod="openshift-ingress/e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4" Oct 30 17:18:18.089726 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:18.079831 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a18c02c8-1a51-4dcf-aeef-696acb2d2280-istiod-ca-cert\") pod \"e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4\" (UID: \"a18c02c8-1a51-4dcf-aeef-696acb2d2280\") " pod="openshift-ingress/e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4" Oct 30 17:18:18.089726 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:18.080420 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a18c02c8-1a51-4dcf-aeef-696acb2d2280-workload-certs\") pod \"e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4\" (UID: \"a18c02c8-1a51-4dcf-aeef-696acb2d2280\") " pod="openshift-ingress/e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4" Oct 30 17:18:18.089726 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:18.080659 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a18c02c8-1a51-4dcf-aeef-696acb2d2280-credential-socket\") pod \"e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4\" (UID: \"a18c02c8-1a51-4dcf-aeef-696acb2d2280\") " pod="openshift-ingress/e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4" Oct 30 17:18:18.110089 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:18.109628 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp9nt\" (UniqueName: \"kubernetes.io/projected/a18c02c8-1a51-4dcf-aeef-696acb2d2280-kube-api-access-rp9nt\") pod \"e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4\" (UID: \"a18c02c8-1a51-4dcf-aeef-696acb2d2280\") " pod="openshift-ingress/e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4" Oct 30 17:18:18.120586 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:18.120423 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a18c02c8-1a51-4dcf-aeef-696acb2d2280-istio-token\") pod \"e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4\" (UID: \"a18c02c8-1a51-4dcf-aeef-696acb2d2280\") " pod="openshift-ingress/e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4" Oct 30 17:18:18.128808 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:18.128397 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a18c02c8-1a51-4dcf-aeef-696acb2d2280-istio-podinfo\") pod \"e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4\" (UID: \"a18c02c8-1a51-4dcf-aeef-696acb2d2280\") " pod="openshift-ingress/e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4" Oct 30 17:18:18.133808 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:18.129006 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a18c02c8-1a51-4dcf-aeef-696acb2d2280-istio-envoy\") pod \"e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4\" (UID: \"a18c02c8-1a51-4dcf-aeef-696acb2d2280\") " pod="openshift-ingress/e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4" Oct 30 17:18:18.303415 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:18.302142 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4" Oct 30 17:18:18.736682 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:18.736572 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4"] Oct 30 17:18:18.739743 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:18.739700 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:18:18.739913 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:18.739771 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:18:18.739913 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:18.739829 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:18:19.710261 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:19.710225 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4" event={"ID":"a18c02c8-1a51-4dcf-aeef-696acb2d2280","Type":"ContainerStarted","Data":"49cc2b2335dd4d05149323f545e7f7c0c490009af8f91c804dfc19053918d7fb"} Oct 30 17:18:19.710450 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:19.710289 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4" event={"ID":"a18c02c8-1a51-4dcf-aeef-696acb2d2280","Type":"ContainerStarted","Data":"6c54b1050471c4abe67cefee17f9e473dcbb1d8856d7ddc9e9671ba419f5606d"} Oct 30 17:18:19.772130 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:19.772073 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4" podStartSLOduration=2.772058698 podStartE2EDuration="2.772058698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:18:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:18:19.768417221 +0000 UTC m=+3326.995592321" watchObservedRunningTime="2026-10-30 17:18:19.772058698 +0000 UTC m=+3326.999233797" Oct 30 17:18:20.308172 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:20.308136 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4" Oct 30 17:18:20.461118 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:20.461087 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4" Oct 30 17:18:20.718453 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:20.718420 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4" Oct 30 17:18:20.719464 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:20.719443 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4" Oct 30 17:18:25.659946 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:25.659905 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-bbc36f/test-model-bbc36f-kserve-6c7bbb6b47-dsbs2"] Oct 30 17:18:25.667897 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:25.666341 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-bbc36f/test-model-bbc36f-kserve-6c7bbb6b47-dsbs2" Oct 30 17:18:25.685112 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:25.685089 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-bbc36f/test-model-bbc36f-kserve-6c7bbb6b47-dsbs2"] Oct 30 17:18:25.689137 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:25.689113 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-bbc36f\"/\"test-model-bbc36f-kserve-self-signed-certs\"" Oct 30 17:18:25.689255 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:25.689146 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-bbc36f\"/\"default-dockercfg-79ggl\"" Oct 30 17:18:25.694108 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:25.694054 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-bbc36f\"/\"kube-root-ca.crt\"" Oct 30 17:18:25.694224 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:25.694199 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-bbc36f\"/\"openshift-service-ca.crt\"" Oct 30 17:18:25.855901 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:25.855857 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86707cad-e7dd-4a77-b04b-3bf99540cc18-tls-certs\") pod \"test-model-bbc36f-kserve-6c7bbb6b47-dsbs2\" (UID: \"86707cad-e7dd-4a77-b04b-3bf99540cc18\") " pod="ai-tenant-e2e-inf-b-bbc36f/test-model-bbc36f-kserve-6c7bbb6b47-dsbs2" Oct 30 17:18:25.856091 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:25.855937 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhnkd\" (UniqueName: \"kubernetes.io/projected/86707cad-e7dd-4a77-b04b-3bf99540cc18-kube-api-access-qhnkd\") pod \"test-model-bbc36f-kserve-6c7bbb6b47-dsbs2\" (UID: \"86707cad-e7dd-4a77-b04b-3bf99540cc18\") " pod="ai-tenant-e2e-inf-b-bbc36f/test-model-bbc36f-kserve-6c7bbb6b47-dsbs2" Oct 30 17:18:25.856206 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:25.856177 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86707cad-e7dd-4a77-b04b-3bf99540cc18-dshm\") pod \"test-model-bbc36f-kserve-6c7bbb6b47-dsbs2\" (UID: \"86707cad-e7dd-4a77-b04b-3bf99540cc18\") " pod="ai-tenant-e2e-inf-b-bbc36f/test-model-bbc36f-kserve-6c7bbb6b47-dsbs2" Oct 30 17:18:25.856350 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:25.856327 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86707cad-e7dd-4a77-b04b-3bf99540cc18-tmp-dir\") pod \"test-model-bbc36f-kserve-6c7bbb6b47-dsbs2\" (UID: \"86707cad-e7dd-4a77-b04b-3bf99540cc18\") " pod="ai-tenant-e2e-inf-b-bbc36f/test-model-bbc36f-kserve-6c7bbb6b47-dsbs2" Oct 30 17:18:25.856411 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:25.856392 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86707cad-e7dd-4a77-b04b-3bf99540cc18-model-cache\") pod \"test-model-bbc36f-kserve-6c7bbb6b47-dsbs2\" (UID: \"86707cad-e7dd-4a77-b04b-3bf99540cc18\") " pod="ai-tenant-e2e-inf-b-bbc36f/test-model-bbc36f-kserve-6c7bbb6b47-dsbs2" Oct 30 17:18:25.856556 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:25.856536 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86707cad-e7dd-4a77-b04b-3bf99540cc18-home\") pod \"test-model-bbc36f-kserve-6c7bbb6b47-dsbs2\" (UID: \"86707cad-e7dd-4a77-b04b-3bf99540cc18\") " pod="ai-tenant-e2e-inf-b-bbc36f/test-model-bbc36f-kserve-6c7bbb6b47-dsbs2" Oct 30 17:18:25.978853 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:25.965256 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86707cad-e7dd-4a77-b04b-3bf99540cc18-dshm\") pod \"test-model-bbc36f-kserve-6c7bbb6b47-dsbs2\" (UID: \"86707cad-e7dd-4a77-b04b-3bf99540cc18\") " pod="ai-tenant-e2e-inf-b-bbc36f/test-model-bbc36f-kserve-6c7bbb6b47-dsbs2" Oct 30 17:18:25.978853 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:25.965366 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86707cad-e7dd-4a77-b04b-3bf99540cc18-tmp-dir\") pod \"test-model-bbc36f-kserve-6c7bbb6b47-dsbs2\" (UID: \"86707cad-e7dd-4a77-b04b-3bf99540cc18\") " pod="ai-tenant-e2e-inf-b-bbc36f/test-model-bbc36f-kserve-6c7bbb6b47-dsbs2" Oct 30 17:18:25.978853 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:25.965395 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86707cad-e7dd-4a77-b04b-3bf99540cc18-model-cache\") pod \"test-model-bbc36f-kserve-6c7bbb6b47-dsbs2\" (UID: \"86707cad-e7dd-4a77-b04b-3bf99540cc18\") " pod="ai-tenant-e2e-inf-b-bbc36f/test-model-bbc36f-kserve-6c7bbb6b47-dsbs2" Oct 30 17:18:25.978853 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:25.965519 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86707cad-e7dd-4a77-b04b-3bf99540cc18-home\") pod \"test-model-bbc36f-kserve-6c7bbb6b47-dsbs2\" (UID: \"86707cad-e7dd-4a77-b04b-3bf99540cc18\") " pod="ai-tenant-e2e-inf-b-bbc36f/test-model-bbc36f-kserve-6c7bbb6b47-dsbs2" Oct 30 17:18:25.978853 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:25.965591 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86707cad-e7dd-4a77-b04b-3bf99540cc18-tls-certs\") pod \"test-model-bbc36f-kserve-6c7bbb6b47-dsbs2\" (UID: \"86707cad-e7dd-4a77-b04b-3bf99540cc18\") " pod="ai-tenant-e2e-inf-b-bbc36f/test-model-bbc36f-kserve-6c7bbb6b47-dsbs2" Oct 30 17:18:25.978853 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:25.965651 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhnkd\" (UniqueName: \"kubernetes.io/projected/86707cad-e7dd-4a77-b04b-3bf99540cc18-kube-api-access-qhnkd\") pod \"test-model-bbc36f-kserve-6c7bbb6b47-dsbs2\" (UID: \"86707cad-e7dd-4a77-b04b-3bf99540cc18\") " pod="ai-tenant-e2e-inf-b-bbc36f/test-model-bbc36f-kserve-6c7bbb6b47-dsbs2" Oct 30 17:18:25.978853 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:25.967181 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86707cad-e7dd-4a77-b04b-3bf99540cc18-model-cache\") pod \"test-model-bbc36f-kserve-6c7bbb6b47-dsbs2\" (UID: \"86707cad-e7dd-4a77-b04b-3bf99540cc18\") " pod="ai-tenant-e2e-inf-b-bbc36f/test-model-bbc36f-kserve-6c7bbb6b47-dsbs2" Oct 30 17:18:25.978853 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:25.967465 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86707cad-e7dd-4a77-b04b-3bf99540cc18-tmp-dir\") pod \"test-model-bbc36f-kserve-6c7bbb6b47-dsbs2\" (UID: \"86707cad-e7dd-4a77-b04b-3bf99540cc18\") " pod="ai-tenant-e2e-inf-b-bbc36f/test-model-bbc36f-kserve-6c7bbb6b47-dsbs2" Oct 30 17:18:25.978853 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:25.967722 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86707cad-e7dd-4a77-b04b-3bf99540cc18-home\") pod \"test-model-bbc36f-kserve-6c7bbb6b47-dsbs2\" (UID: \"86707cad-e7dd-4a77-b04b-3bf99540cc18\") " pod="ai-tenant-e2e-inf-b-bbc36f/test-model-bbc36f-kserve-6c7bbb6b47-dsbs2" Oct 30 17:18:25.994022 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:25.991128 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86707cad-e7dd-4a77-b04b-3bf99540cc18-dshm\") pod \"test-model-bbc36f-kserve-6c7bbb6b47-dsbs2\" (UID: \"86707cad-e7dd-4a77-b04b-3bf99540cc18\") " pod="ai-tenant-e2e-inf-b-bbc36f/test-model-bbc36f-kserve-6c7bbb6b47-dsbs2" Oct 30 17:18:26.007221 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:25.999848 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86707cad-e7dd-4a77-b04b-3bf99540cc18-tls-certs\") pod \"test-model-bbc36f-kserve-6c7bbb6b47-dsbs2\" (UID: \"86707cad-e7dd-4a77-b04b-3bf99540cc18\") " pod="ai-tenant-e2e-inf-b-bbc36f/test-model-bbc36f-kserve-6c7bbb6b47-dsbs2" Oct 30 17:18:26.007221 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:26.003414 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhnkd\" (UniqueName: \"kubernetes.io/projected/86707cad-e7dd-4a77-b04b-3bf99540cc18-kube-api-access-qhnkd\") pod \"test-model-bbc36f-kserve-6c7bbb6b47-dsbs2\" (UID: \"86707cad-e7dd-4a77-b04b-3bf99540cc18\") " pod="ai-tenant-e2e-inf-b-bbc36f/test-model-bbc36f-kserve-6c7bbb6b47-dsbs2" Oct 30 17:18:26.292051 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:26.291960 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-bbc36f/test-model-bbc36f-kserve-6c7bbb6b47-dsbs2" Oct 30 17:18:26.618223 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:26.618188 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-bbc36f/test-model-bbc36f-kserve-6c7bbb6b47-dsbs2"] Oct 30 17:18:26.618887 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:18:26.618861 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod86707cad_e7dd_4a77_b04b_3bf99540cc18.slice/crio-0990298a7a001339d62571b2687ecaeae2bab785728c6ce36dd3943978e2fb0e WatchSource:0}: Error finding container 0990298a7a001339d62571b2687ecaeae2bab785728c6ce36dd3943978e2fb0e: Status 404 returned error can't find the container with id 0990298a7a001339d62571b2687ecaeae2bab785728c6ce36dd3943978e2fb0e Oct 30 17:18:26.820951 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:26.820863 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-bbc36f/test-model-bbc36f-kserve-6c7bbb6b47-dsbs2" event={"ID":"86707cad-e7dd-4a77-b04b-3bf99540cc18","Type":"ContainerStarted","Data":"0990298a7a001339d62571b2687ecaeae2bab785728c6ce36dd3943978e2fb0e"} Oct 30 17:18:27.833612 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:27.833577 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-bbc36f/test-model-bbc36f-kserve-6c7bbb6b47-dsbs2" event={"ID":"86707cad-e7dd-4a77-b04b-3bf99540cc18","Type":"ContainerStarted","Data":"e2229f8e473381fdbfabf3b52afb310e14211ab21a16abce81087bff8f720494"} Oct 30 17:18:27.878389 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:27.878336 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-bbc36f/test-model-bbc36f-kserve-6c7bbb6b47-dsbs2" podStartSLOduration=2.87832162 podStartE2EDuration="2.87832162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:18:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:18:27.876659422 +0000 UTC m=+3335.103834523" watchObservedRunningTime="2026-10-30 17:18:27.87832162 +0000 UTC m=+3335.105496720" Oct 30 17:18:28.277082 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:28.277045 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-9afbb2/test-model-9afbb2-kserve-6c5d6c968c-sfbmg"] Oct 30 17:18:28.297517 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:28.287664 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9afbb2/test-model-9afbb2-kserve-6c5d6c968c-sfbmg" Oct 30 17:18:28.301927 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:28.298980 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-bbc36f/test-model-bbc36f-kserve-6c7bbb6b47-dsbs2" Oct 30 17:18:28.306545 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:28.306172 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-9afbb2/test-model-9afbb2-kserve-6c5d6c968c-sfbmg"] Oct 30 17:18:28.310808 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:28.309713 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-9afbb2\"/\"openshift-service-ca.crt\"" Oct 30 17:18:28.310808 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:28.309814 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-9afbb2\"/\"default-dockercfg-462b6\"" Oct 30 17:18:28.310808 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:28.309910 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-9afbb2\"/\"test-model-9afbb2-kserve-self-signed-certs\"" Oct 30 17:18:28.315624 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:28.315603 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-9afbb2\"/\"kube-root-ca.crt\"" Oct 30 17:18:28.365807 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:28.364251 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a94190b0-d6ef-4227-9b06-d10cbf89c567-tls-certs\") pod \"test-model-9afbb2-kserve-6c5d6c968c-sfbmg\" (UID: \"a94190b0-d6ef-4227-9b06-d10cbf89c567\") " pod="ai-tenant-e2e-inf-a-9afbb2/test-model-9afbb2-kserve-6c5d6c968c-sfbmg" Oct 30 17:18:28.366203 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:28.366177 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a94190b0-d6ef-4227-9b06-d10cbf89c567-tmp-dir\") pod \"test-model-9afbb2-kserve-6c5d6c968c-sfbmg\" (UID: \"a94190b0-d6ef-4227-9b06-d10cbf89c567\") " pod="ai-tenant-e2e-inf-a-9afbb2/test-model-9afbb2-kserve-6c5d6c968c-sfbmg" Oct 30 17:18:28.366479 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:28.366458 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a94190b0-d6ef-4227-9b06-d10cbf89c567-home\") pod \"test-model-9afbb2-kserve-6c5d6c968c-sfbmg\" (UID: \"a94190b0-d6ef-4227-9b06-d10cbf89c567\") " pod="ai-tenant-e2e-inf-a-9afbb2/test-model-9afbb2-kserve-6c5d6c968c-sfbmg" Oct 30 17:18:28.366744 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:28.366726 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a94190b0-d6ef-4227-9b06-d10cbf89c567-dshm\") pod \"test-model-9afbb2-kserve-6c5d6c968c-sfbmg\" (UID: \"a94190b0-d6ef-4227-9b06-d10cbf89c567\") " pod="ai-tenant-e2e-inf-a-9afbb2/test-model-9afbb2-kserve-6c5d6c968c-sfbmg" Oct 30 17:18:28.367112 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:28.367090 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a94190b0-d6ef-4227-9b06-d10cbf89c567-model-cache\") pod \"test-model-9afbb2-kserve-6c5d6c968c-sfbmg\" (UID: \"a94190b0-d6ef-4227-9b06-d10cbf89c567\") " pod="ai-tenant-e2e-inf-a-9afbb2/test-model-9afbb2-kserve-6c5d6c968c-sfbmg" Oct 30 17:18:28.367316 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:28.367299 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w66qv\" (UniqueName: \"kubernetes.io/projected/a94190b0-d6ef-4227-9b06-d10cbf89c567-kube-api-access-w66qv\") pod \"test-model-9afbb2-kserve-6c5d6c968c-sfbmg\" (UID: \"a94190b0-d6ef-4227-9b06-d10cbf89c567\") " pod="ai-tenant-e2e-inf-a-9afbb2/test-model-9afbb2-kserve-6c5d6c968c-sfbmg" Oct 30 17:18:28.468469 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:28.468436 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a94190b0-d6ef-4227-9b06-d10cbf89c567-tls-certs\") pod \"test-model-9afbb2-kserve-6c5d6c968c-sfbmg\" (UID: \"a94190b0-d6ef-4227-9b06-d10cbf89c567\") " pod="ai-tenant-e2e-inf-a-9afbb2/test-model-9afbb2-kserve-6c5d6c968c-sfbmg" Oct 30 17:18:28.468767 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:28.468740 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a94190b0-d6ef-4227-9b06-d10cbf89c567-tmp-dir\") pod \"test-model-9afbb2-kserve-6c5d6c968c-sfbmg\" (UID: \"a94190b0-d6ef-4227-9b06-d10cbf89c567\") " pod="ai-tenant-e2e-inf-a-9afbb2/test-model-9afbb2-kserve-6c5d6c968c-sfbmg" Oct 30 17:18:28.468916 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:28.468808 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a94190b0-d6ef-4227-9b06-d10cbf89c567-home\") pod \"test-model-9afbb2-kserve-6c5d6c968c-sfbmg\" (UID: \"a94190b0-d6ef-4227-9b06-d10cbf89c567\") " pod="ai-tenant-e2e-inf-a-9afbb2/test-model-9afbb2-kserve-6c5d6c968c-sfbmg" Oct 30 17:18:28.468916 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:28.468901 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a94190b0-d6ef-4227-9b06-d10cbf89c567-dshm\") pod \"test-model-9afbb2-kserve-6c5d6c968c-sfbmg\" (UID: \"a94190b0-d6ef-4227-9b06-d10cbf89c567\") " pod="ai-tenant-e2e-inf-a-9afbb2/test-model-9afbb2-kserve-6c5d6c968c-sfbmg" Oct 30 17:18:28.469092 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:28.469059 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a94190b0-d6ef-4227-9b06-d10cbf89c567-model-cache\") pod \"test-model-9afbb2-kserve-6c5d6c968c-sfbmg\" (UID: \"a94190b0-d6ef-4227-9b06-d10cbf89c567\") " pod="ai-tenant-e2e-inf-a-9afbb2/test-model-9afbb2-kserve-6c5d6c968c-sfbmg" Oct 30 17:18:28.469210 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:28.469123 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w66qv\" (UniqueName: \"kubernetes.io/projected/a94190b0-d6ef-4227-9b06-d10cbf89c567-kube-api-access-w66qv\") pod \"test-model-9afbb2-kserve-6c5d6c968c-sfbmg\" (UID: \"a94190b0-d6ef-4227-9b06-d10cbf89c567\") " pod="ai-tenant-e2e-inf-a-9afbb2/test-model-9afbb2-kserve-6c5d6c968c-sfbmg" Oct 30 17:18:28.469510 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:28.469432 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a94190b0-d6ef-4227-9b06-d10cbf89c567-tmp-dir\") pod \"test-model-9afbb2-kserve-6c5d6c968c-sfbmg\" (UID: \"a94190b0-d6ef-4227-9b06-d10cbf89c567\") " pod="ai-tenant-e2e-inf-a-9afbb2/test-model-9afbb2-kserve-6c5d6c968c-sfbmg" Oct 30 17:18:28.469699 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:28.469678 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a94190b0-d6ef-4227-9b06-d10cbf89c567-model-cache\") pod \"test-model-9afbb2-kserve-6c5d6c968c-sfbmg\" (UID: \"a94190b0-d6ef-4227-9b06-d10cbf89c567\") " pod="ai-tenant-e2e-inf-a-9afbb2/test-model-9afbb2-kserve-6c5d6c968c-sfbmg" Oct 30 17:18:28.469823 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:28.469720 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a94190b0-d6ef-4227-9b06-d10cbf89c567-home\") pod \"test-model-9afbb2-kserve-6c5d6c968c-sfbmg\" (UID: \"a94190b0-d6ef-4227-9b06-d10cbf89c567\") " pod="ai-tenant-e2e-inf-a-9afbb2/test-model-9afbb2-kserve-6c5d6c968c-sfbmg" Oct 30 17:18:28.471422 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:28.471404 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a94190b0-d6ef-4227-9b06-d10cbf89c567-tls-certs\") pod \"test-model-9afbb2-kserve-6c5d6c968c-sfbmg\" (UID: \"a94190b0-d6ef-4227-9b06-d10cbf89c567\") " pod="ai-tenant-e2e-inf-a-9afbb2/test-model-9afbb2-kserve-6c5d6c968c-sfbmg" Oct 30 17:18:28.471868 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:28.471853 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a94190b0-d6ef-4227-9b06-d10cbf89c567-dshm\") pod \"test-model-9afbb2-kserve-6c5d6c968c-sfbmg\" (UID: \"a94190b0-d6ef-4227-9b06-d10cbf89c567\") " pod="ai-tenant-e2e-inf-a-9afbb2/test-model-9afbb2-kserve-6c5d6c968c-sfbmg" Oct 30 17:18:28.493195 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:28.493167 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w66qv\" (UniqueName: \"kubernetes.io/projected/a94190b0-d6ef-4227-9b06-d10cbf89c567-kube-api-access-w66qv\") pod \"test-model-9afbb2-kserve-6c5d6c968c-sfbmg\" (UID: \"a94190b0-d6ef-4227-9b06-d10cbf89c567\") " pod="ai-tenant-e2e-inf-a-9afbb2/test-model-9afbb2-kserve-6c5d6c968c-sfbmg" Oct 30 17:18:28.633860 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:28.620086 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9afbb2/test-model-9afbb2-kserve-6c5d6c968c-sfbmg" Oct 30 17:18:29.127651 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:18:29.127620 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda94190b0_d6ef_4227_9b06_d10cbf89c567.slice/crio-f282ec52e11aeecc05c24173bbe6f0661f9fe7acbbc213a659803d2bb5950389 WatchSource:0}: Error finding container f282ec52e11aeecc05c24173bbe6f0661f9fe7acbbc213a659803d2bb5950389: Status 404 returned error can't find the container with id f282ec52e11aeecc05c24173bbe6f0661f9fe7acbbc213a659803d2bb5950389 Oct 30 17:18:29.130213 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:29.130178 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-9afbb2/test-model-9afbb2-kserve-6c5d6c968c-sfbmg"] Oct 30 17:18:29.859997 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:29.859957 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9afbb2/test-model-9afbb2-kserve-6c5d6c968c-sfbmg" event={"ID":"a94190b0-d6ef-4227-9b06-d10cbf89c567","Type":"ContainerStarted","Data":"4ec528c3b973232a1ff94187310213b0246c32f3d46d5f8d4241f8ba9205b680"} Oct 30 17:18:29.859997 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:29.859999 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9afbb2/test-model-9afbb2-kserve-6c5d6c968c-sfbmg" event={"ID":"a94190b0-d6ef-4227-9b06-d10cbf89c567","Type":"ContainerStarted","Data":"f282ec52e11aeecc05c24173bbe6f0661f9fe7acbbc213a659803d2bb5950389"} Oct 30 17:18:29.904091 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:29.904031 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-9afbb2/test-model-9afbb2-kserve-6c5d6c968c-sfbmg" podStartSLOduration=1.904014016 podStartE2EDuration="1.904014016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:18:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:18:29.901281483 +0000 UTC m=+3337.128456583" watchObservedRunningTime="2026-10-30 17:18:29.904014016 +0000 UTC m=+3337.131189117" Oct 30 17:18:30.389322 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:18:30.389283 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 17:18:30.620863 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:30.620772 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-9afbb2/test-model-9afbb2-kserve-6c5d6c968c-sfbmg" Oct 30 17:18:32.041452 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:32.041422 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f6949a3-l2mvr"] Oct 30 17:18:32.047054 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:32.047025 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2cc23d3-fb7n8"] Oct 30 17:18:32.053862 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:32.053834 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0fbf5ac-69flm"] Oct 30 17:18:32.058605 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:32.058579 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f6949a3-l2mvr"] Oct 30 17:18:32.062834 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:32.062812 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2cc23d3-fb7n8"] Oct 30 17:18:32.066584 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:32.066564 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0fbf5ac-69flm"] Oct 30 17:18:32.937907 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:32.937869 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7dfcd4f-5c74-41c1-926f-02a3d1b525e5" path="/var/lib/kubelet/pods/a7dfcd4f-5c74-41c1-926f-02a3d1b525e5/volumes" Oct 30 17:18:32.938404 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:32.938388 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aad374be-4bce-4729-bf69-73a4e855b79b" path="/var/lib/kubelet/pods/aad374be-4bce-4729-bf69-73a4e855b79b/volumes" Oct 30 17:18:32.938866 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:32.938846 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4821fb3-ff5f-4095-ab78-f33f56fc464d" path="/var/lib/kubelet/pods/b4821fb3-ff5f-4095-ab78-f33f56fc464d/volumes" Oct 30 17:18:36.293107 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:36.293058 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-bbc36f/test-model-bbc36f-kserve-6c7bbb6b47-dsbs2" Oct 30 17:18:36.306318 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:36.306290 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-bbc36f/test-model-bbc36f-kserve-6c7bbb6b47-dsbs2" Oct 30 17:18:36.962159 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:36.962131 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-bbc36f/test-model-bbc36f-kserve-6c7bbb6b47-dsbs2" Oct 30 17:18:38.620822 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:38.620753 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-9afbb2/test-model-9afbb2-kserve-6c5d6c968c-sfbmg" Oct 30 17:18:38.634171 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:38.634141 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-9afbb2/test-model-9afbb2-kserve-6c5d6c968c-sfbmg" Oct 30 17:18:38.979709 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:38.979682 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-9afbb2/test-model-9afbb2-kserve-6c5d6c968c-sfbmg" Oct 30 17:18:40.270357 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:40.270324 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f63f380-kvm2k"] Oct 30 17:18:40.276616 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:40.276597 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f63f380-kvm2k" Oct 30 17:18:40.297085 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:40.297061 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f63f380-kvm2k"] Oct 30 17:18:40.303098 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:40.303076 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:18:40.303266 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:40.303169 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:18:40.303355 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:40.303184 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 17:18:40.304258 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:40.304232 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:18:40.375456 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:40.375421 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6033dc55-f3f8-45e2-bfe3-0d968ff023a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f63f380-kvm2k\" (UID: \"6033dc55-f3f8-45e2-bfe3-0d968ff023a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f63f380-kvm2k" Oct 30 17:18:40.375651 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:40.375584 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6033dc55-f3f8-45e2-bfe3-0d968ff023a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f63f380-kvm2k\" (UID: \"6033dc55-f3f8-45e2-bfe3-0d968ff023a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f63f380-kvm2k" Oct 30 17:18:40.375857 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:40.375832 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kwhr\" (UniqueName: \"kubernetes.io/projected/6033dc55-f3f8-45e2-bfe3-0d968ff023a1-kube-api-access-6kwhr\") pod \"maas-api-revoke-sub-8f63f380-kvm2k\" (UID: \"6033dc55-f3f8-45e2-bfe3-0d968ff023a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f63f380-kvm2k" Oct 30 17:18:40.477269 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:40.477238 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6033dc55-f3f8-45e2-bfe3-0d968ff023a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f63f380-kvm2k\" (UID: \"6033dc55-f3f8-45e2-bfe3-0d968ff023a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f63f380-kvm2k" Oct 30 17:18:40.477456 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:40.477390 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kwhr\" (UniqueName: \"kubernetes.io/projected/6033dc55-f3f8-45e2-bfe3-0d968ff023a1-kube-api-access-6kwhr\") pod \"maas-api-revoke-sub-8f63f380-kvm2k\" (UID: \"6033dc55-f3f8-45e2-bfe3-0d968ff023a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f63f380-kvm2k" Oct 30 17:18:40.477638 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:40.477616 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6033dc55-f3f8-45e2-bfe3-0d968ff023a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f63f380-kvm2k\" (UID: \"6033dc55-f3f8-45e2-bfe3-0d968ff023a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f63f380-kvm2k" Oct 30 17:18:40.478013 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:40.477993 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6033dc55-f3f8-45e2-bfe3-0d968ff023a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f63f380-kvm2k\" (UID: \"6033dc55-f3f8-45e2-bfe3-0d968ff023a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f63f380-kvm2k" Oct 30 17:18:40.480354 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:40.480323 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6033dc55-f3f8-45e2-bfe3-0d968ff023a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f63f380-kvm2k\" (UID: \"6033dc55-f3f8-45e2-bfe3-0d968ff023a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f63f380-kvm2k" Oct 30 17:18:40.501024 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:40.500996 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kwhr\" (UniqueName: \"kubernetes.io/projected/6033dc55-f3f8-45e2-bfe3-0d968ff023a1-kube-api-access-6kwhr\") pod \"maas-api-revoke-sub-8f63f380-kvm2k\" (UID: \"6033dc55-f3f8-45e2-bfe3-0d968ff023a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f63f380-kvm2k" Oct 30 17:18:40.594946 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:40.594865 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f63f380-kvm2k" Oct 30 17:18:40.913829 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:40.913797 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f63f380-kvm2k"] Oct 30 17:18:40.914082 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:18:40.914057 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6033dc55_f3f8_45e2_bfe3_0d968ff023a1.slice/crio-d5855154e0da3ef86614691b5408fba8d58e1372e231c1aabc52df3387732c69 WatchSource:0}: Error finding container d5855154e0da3ef86614691b5408fba8d58e1372e231c1aabc52df3387732c69: Status 404 returned error can't find the container with id d5855154e0da3ef86614691b5408fba8d58e1372e231c1aabc52df3387732c69 Oct 30 17:18:40.989941 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:40.989903 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f63f380-kvm2k" event={"ID":"6033dc55-f3f8-45e2-bfe3-0d968ff023a1","Type":"ContainerStarted","Data":"d5855154e0da3ef86614691b5408fba8d58e1372e231c1aabc52df3387732c69"} Oct 30 17:18:42.000149 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:42.000116 4493 generic.go:356] "Generic (PLEG): container finished" podID="6033dc55-f3f8-45e2-bfe3-0d968ff023a1" containerID="0f30add92d2aca98bb755b6276bba5f85cafe5d3e31cfd4c0c5af97ae0ce6fec" exitCode=0 Oct 30 17:18:42.000510 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:42.000154 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f63f380-kvm2k" event={"ID":"6033dc55-f3f8-45e2-bfe3-0d968ff023a1","Type":"ContainerDied","Data":"0f30add92d2aca98bb755b6276bba5f85cafe5d3e31cfd4c0c5af97ae0ce6fec"} Oct 30 17:18:42.034529 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:42.034497 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8efa9d2-t2m6h"] Oct 30 17:18:42.039463 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:42.039434 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8efa9d2-t2m6h"] Oct 30 17:18:42.045550 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:42.045518 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c81cd66-lqh87"] Oct 30 17:18:42.051277 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:42.051248 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c81cd66-lqh87"] Oct 30 17:18:42.938109 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:42.938078 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94e42682-3d3b-4fca-8bc7-e6c6807950f5" path="/var/lib/kubelet/pods/94e42682-3d3b-4fca-8bc7-e6c6807950f5/volumes" Oct 30 17:18:42.938624 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:42.938608 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0fad0bd-9b34-443a-8262-ec55c07fc934" path="/var/lib/kubelet/pods/e0fad0bd-9b34-443a-8262-ec55c07fc934/volumes" Oct 30 17:18:43.283774 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:43.283744 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f63f380-kvm2k" Oct 30 17:18:43.339416 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:43.339382 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6033dc55-f3f8-45e2-bfe3-0d968ff023a1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6033dc55-f3f8-45e2-bfe3-0d968ff023a1-maas-api-xks-ca\") pod \"6033dc55-f3f8-45e2-bfe3-0d968ff023a1\" (UID: \"6033dc55-f3f8-45e2-bfe3-0d968ff023a1\") " Oct 30 17:18:43.340931 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:43.340904 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6033dc55-f3f8-45e2-bfe3-0d968ff023a1-maas-api-openshift-ca" pod "6033dc55-f3f8-45e2-bfe3-0d968ff023a1" (UID: "6033dc55-f3f8-45e2-bfe3-0d968ff023a1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:18:43.341152 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:43.341130 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6033dc55-f3f8-45e2-bfe3-0d968ff023a1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6033dc55-f3f8-45e2-bfe3-0d968ff023a1-maas-api-openshift-ca\") pod \"6033dc55-f3f8-45e2-bfe3-0d968ff023a1\" (UID: \"6033dc55-f3f8-45e2-bfe3-0d968ff023a1\") " Oct 30 17:18:43.341864 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:43.341844 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6033dc55-f3f8-45e2-bfe3-0d968ff023a1-kube-api-access-6kwhr\" (UniqueName: \"kubernetes.io/projected/6033dc55-f3f8-45e2-bfe3-0d968ff023a1-kube-api-access-6kwhr\") pod \"6033dc55-f3f8-45e2-bfe3-0d968ff023a1\" (UID: \"6033dc55-f3f8-45e2-bfe3-0d968ff023a1\") " Oct 30 17:18:43.343471 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:43.343449 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6033dc55-f3f8-45e2-bfe3-0d968ff023a1-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:18:43.345206 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:43.345186 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6033dc55-f3f8-45e2-bfe3-0d968ff023a1-maas-api-xks-ca" pod "6033dc55-f3f8-45e2-bfe3-0d968ff023a1" (UID: "6033dc55-f3f8-45e2-bfe3-0d968ff023a1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:18:43.347330 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:43.347298 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6033dc55-f3f8-45e2-bfe3-0d968ff023a1-kube-api-access-6kwhr" pod "6033dc55-f3f8-45e2-bfe3-0d968ff023a1" (UID: "6033dc55-f3f8-45e2-bfe3-0d968ff023a1"). InnerVolumeSpecName "kube-api-access-6kwhr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:18:43.445351 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:43.445319 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kwhr\" (UniqueName: \"kubernetes.io/projected/6033dc55-f3f8-45e2-bfe3-0d968ff023a1-kube-api-access-6kwhr\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:18:43.445351 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:43.445352 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6033dc55-f3f8-45e2-bfe3-0d968ff023a1-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:18:44.032721 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:44.032695 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ef1fb16-mnsb6"] Oct 30 17:18:44.033003 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:44.032983 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f63f380-kvm2k" Oct 30 17:18:44.033070 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:44.033006 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f63f380-kvm2k" event={"ID":"6033dc55-f3f8-45e2-bfe3-0d968ff023a1","Type":"ContainerDied","Data":"d5855154e0da3ef86614691b5408fba8d58e1372e231c1aabc52df3387732c69"} Oct 30 17:18:44.033070 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:44.033048 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5855154e0da3ef86614691b5408fba8d58e1372e231c1aabc52df3387732c69" Oct 30 17:18:44.040158 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:44.040133 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ef1fb16-mnsb6"] Oct 30 17:18:44.942897 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:44.942860 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="711c8b22-46bd-4989-8a27-829c81465f5a" path="/var/lib/kubelet/pods/711c8b22-46bd-4989-8a27-829c81465f5a/volumes" Oct 30 17:18:52.251455 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:52.251391 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-85eb87/test-model-85eb87-kserve-686468c445-ssq2z"] Oct 30 17:18:52.252757 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:52.252721 4493 cpu_manager.go:414] "containerMap: removing container" podUID="6033dc55-f3f8-45e2-bfe3-0d968ff023a1" containerName="revoke-keys" Oct 30 17:18:52.252979 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:52.252950 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6033dc55-f3f8-45e2-bfe3-0d968ff023a1" containerName="revoke-keys" Oct 30 17:18:52.253275 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:52.253254 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="6033dc55-f3f8-45e2-bfe3-0d968ff023a1" containerName="revoke-keys" Oct 30 17:18:52.273078 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:52.273046 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-85eb87/test-model-85eb87-kserve-686468c445-ssq2z" Oct 30 17:18:52.279671 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:52.279645 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-85eb87/test-model-85eb87-kserve-686468c445-ssq2z"] Oct 30 17:18:52.288479 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:52.288420 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-85eb87\"/\"openshift-service-ca.crt\"" Oct 30 17:18:52.288479 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:52.288438 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-85eb87\"/\"kube-root-ca.crt\"" Oct 30 17:18:52.301172 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:52.300503 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-85eb87\"/\"default-dockercfg-bqgxg\"" Oct 30 17:18:52.301172 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:52.300533 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-85eb87\"/\"test-model-85eb87-kserve-self-signed-certs\"" Oct 30 17:18:52.416208 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:52.416161 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6fxg\" (UniqueName: \"kubernetes.io/projected/0d5eb370-685d-4d9f-b450-47d754945375-kube-api-access-s6fxg\") pod \"test-model-85eb87-kserve-686468c445-ssq2z\" (UID: \"0d5eb370-685d-4d9f-b450-47d754945375\") " pod="ai-tenant-e2e-inf-b-85eb87/test-model-85eb87-kserve-686468c445-ssq2z" Oct 30 17:18:52.416404 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:52.416357 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d5eb370-685d-4d9f-b450-47d754945375-model-cache\") pod \"test-model-85eb87-kserve-686468c445-ssq2z\" (UID: \"0d5eb370-685d-4d9f-b450-47d754945375\") " pod="ai-tenant-e2e-inf-b-85eb87/test-model-85eb87-kserve-686468c445-ssq2z" Oct 30 17:18:52.416567 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:52.416546 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d5eb370-685d-4d9f-b450-47d754945375-tmp-dir\") pod \"test-model-85eb87-kserve-686468c445-ssq2z\" (UID: \"0d5eb370-685d-4d9f-b450-47d754945375\") " pod="ai-tenant-e2e-inf-b-85eb87/test-model-85eb87-kserve-686468c445-ssq2z" Oct 30 17:18:52.416685 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:52.416661 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d5eb370-685d-4d9f-b450-47d754945375-home\") pod \"test-model-85eb87-kserve-686468c445-ssq2z\" (UID: \"0d5eb370-685d-4d9f-b450-47d754945375\") " pod="ai-tenant-e2e-inf-b-85eb87/test-model-85eb87-kserve-686468c445-ssq2z" Oct 30 17:18:52.416927 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:52.416909 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d5eb370-685d-4d9f-b450-47d754945375-dshm\") pod \"test-model-85eb87-kserve-686468c445-ssq2z\" (UID: \"0d5eb370-685d-4d9f-b450-47d754945375\") " pod="ai-tenant-e2e-inf-b-85eb87/test-model-85eb87-kserve-686468c445-ssq2z" Oct 30 17:18:52.417002 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:52.416952 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d5eb370-685d-4d9f-b450-47d754945375-tls-certs\") pod \"test-model-85eb87-kserve-686468c445-ssq2z\" (UID: \"0d5eb370-685d-4d9f-b450-47d754945375\") " pod="ai-tenant-e2e-inf-b-85eb87/test-model-85eb87-kserve-686468c445-ssq2z" Oct 30 17:18:52.528856 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:52.522927 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6fxg\" (UniqueName: \"kubernetes.io/projected/0d5eb370-685d-4d9f-b450-47d754945375-kube-api-access-s6fxg\") pod \"test-model-85eb87-kserve-686468c445-ssq2z\" (UID: \"0d5eb370-685d-4d9f-b450-47d754945375\") " pod="ai-tenant-e2e-inf-b-85eb87/test-model-85eb87-kserve-686468c445-ssq2z" Oct 30 17:18:52.528856 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:52.523047 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d5eb370-685d-4d9f-b450-47d754945375-model-cache\") pod \"test-model-85eb87-kserve-686468c445-ssq2z\" (UID: \"0d5eb370-685d-4d9f-b450-47d754945375\") " pod="ai-tenant-e2e-inf-b-85eb87/test-model-85eb87-kserve-686468c445-ssq2z" Oct 30 17:18:52.528856 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:52.523189 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d5eb370-685d-4d9f-b450-47d754945375-tmp-dir\") pod \"test-model-85eb87-kserve-686468c445-ssq2z\" (UID: \"0d5eb370-685d-4d9f-b450-47d754945375\") " pod="ai-tenant-e2e-inf-b-85eb87/test-model-85eb87-kserve-686468c445-ssq2z" Oct 30 17:18:52.528856 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:52.523257 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d5eb370-685d-4d9f-b450-47d754945375-home\") pod \"test-model-85eb87-kserve-686468c445-ssq2z\" (UID: \"0d5eb370-685d-4d9f-b450-47d754945375\") " pod="ai-tenant-e2e-inf-b-85eb87/test-model-85eb87-kserve-686468c445-ssq2z" Oct 30 17:18:52.528856 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:52.523404 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d5eb370-685d-4d9f-b450-47d754945375-dshm\") pod \"test-model-85eb87-kserve-686468c445-ssq2z\" (UID: \"0d5eb370-685d-4d9f-b450-47d754945375\") " pod="ai-tenant-e2e-inf-b-85eb87/test-model-85eb87-kserve-686468c445-ssq2z" Oct 30 17:18:52.528856 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:52.523436 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d5eb370-685d-4d9f-b450-47d754945375-tls-certs\") pod \"test-model-85eb87-kserve-686468c445-ssq2z\" (UID: \"0d5eb370-685d-4d9f-b450-47d754945375\") " pod="ai-tenant-e2e-inf-b-85eb87/test-model-85eb87-kserve-686468c445-ssq2z" Oct 30 17:18:52.528856 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:52.524525 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d5eb370-685d-4d9f-b450-47d754945375-tmp-dir\") pod \"test-model-85eb87-kserve-686468c445-ssq2z\" (UID: \"0d5eb370-685d-4d9f-b450-47d754945375\") " pod="ai-tenant-e2e-inf-b-85eb87/test-model-85eb87-kserve-686468c445-ssq2z" Oct 30 17:18:52.528856 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:52.524593 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d5eb370-685d-4d9f-b450-47d754945375-home\") pod \"test-model-85eb87-kserve-686468c445-ssq2z\" (UID: \"0d5eb370-685d-4d9f-b450-47d754945375\") " pod="ai-tenant-e2e-inf-b-85eb87/test-model-85eb87-kserve-686468c445-ssq2z" Oct 30 17:18:52.528856 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:52.524881 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d5eb370-685d-4d9f-b450-47d754945375-model-cache\") pod \"test-model-85eb87-kserve-686468c445-ssq2z\" (UID: \"0d5eb370-685d-4d9f-b450-47d754945375\") " pod="ai-tenant-e2e-inf-b-85eb87/test-model-85eb87-kserve-686468c445-ssq2z" Oct 30 17:18:52.538809 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:52.529583 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d5eb370-685d-4d9f-b450-47d754945375-tls-certs\") pod \"test-model-85eb87-kserve-686468c445-ssq2z\" (UID: \"0d5eb370-685d-4d9f-b450-47d754945375\") " pod="ai-tenant-e2e-inf-b-85eb87/test-model-85eb87-kserve-686468c445-ssq2z" Oct 30 17:18:52.545840 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:52.539476 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d5eb370-685d-4d9f-b450-47d754945375-dshm\") pod \"test-model-85eb87-kserve-686468c445-ssq2z\" (UID: \"0d5eb370-685d-4d9f-b450-47d754945375\") " pod="ai-tenant-e2e-inf-b-85eb87/test-model-85eb87-kserve-686468c445-ssq2z" Oct 30 17:18:52.574014 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:52.562086 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6fxg\" (UniqueName: \"kubernetes.io/projected/0d5eb370-685d-4d9f-b450-47d754945375-kube-api-access-s6fxg\") pod \"test-model-85eb87-kserve-686468c445-ssq2z\" (UID: \"0d5eb370-685d-4d9f-b450-47d754945375\") " pod="ai-tenant-e2e-inf-b-85eb87/test-model-85eb87-kserve-686468c445-ssq2z" Oct 30 17:18:52.614845 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:52.603097 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-85eb87/test-model-85eb87-kserve-686468c445-ssq2z" Oct 30 17:18:53.181555 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:53.181495 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-85eb87/test-model-85eb87-kserve-686468c445-ssq2z"] Oct 30 17:18:53.181875 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:18:53.181852 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0d5eb370_685d_4d9f_b450_47d754945375.slice/crio-07e8fe4d2ef21f9a19ff4d92ea4f565b8833c889c6509a42b384a4734eecbdef WatchSource:0}: Error finding container 07e8fe4d2ef21f9a19ff4d92ea4f565b8833c889c6509a42b384a4734eecbdef: Status 404 returned error can't find the container with id 07e8fe4d2ef21f9a19ff4d92ea4f565b8833c889c6509a42b384a4734eecbdef Oct 30 17:18:54.132965 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:54.132926 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-85eb87/test-model-85eb87-kserve-686468c445-ssq2z" event={"ID":"0d5eb370-685d-4d9f-b450-47d754945375","Type":"ContainerStarted","Data":"28f01c79c71f88d779da209dc4badbe457a0bc7779efcd09cebffddefe51e70f"} Oct 30 17:18:54.133488 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:54.132981 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-85eb87/test-model-85eb87-kserve-686468c445-ssq2z" event={"ID":"0d5eb370-685d-4d9f-b450-47d754945375","Type":"ContainerStarted","Data":"07e8fe4d2ef21f9a19ff4d92ea4f565b8833c889c6509a42b384a4734eecbdef"} Oct 30 17:18:54.178546 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:54.178469 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-85eb87/test-model-85eb87-kserve-686468c445-ssq2z" podStartSLOduration=2.17844996 podStartE2EDuration="2.17844996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:18:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:18:54.176013198 +0000 UTC m=+3361.403188298" watchObservedRunningTime="2026-10-30 17:18:54.17844996 +0000 UTC m=+3361.405625063" Oct 30 17:18:54.604859 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:54.604773 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-85eb87/test-model-85eb87-kserve-686468c445-ssq2z" Oct 30 17:18:58.194155 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:18:58.194123 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 17:19:00.267808 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:00.265444 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" podUID="e90f43d3-f5eb-45ab-af1e-340d0ce18966" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:19:00.267808 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:00.266373 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" podUID="b5abe703-fee0-4185-aa22-88d1db06392b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:19:00.295898 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:00.281890 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" podUID="08baad08-cd72-4305-b92d-c11a84ef276f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:19:00.349856 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:00.338763 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-24acb2/test-model-24acb2-kserve-946ff6bb7-h8cjs" podUID="cd1ff57b-1a55-4ef7-95f6-ee9389a00d99" containerName="main" probeResult="failure" output="Get \"https://10.134.1.20:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:19:00.359840 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:00.356552 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" podUID="0eadc5d4-98c3-4bf2-b3c2-f73fe249182f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:19:00.363808 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:00.361860 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" podUID="14805045-a727-41ba-8bd0-100d41f9f27d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:19:00.461807 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:00.454459 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" podUID="f3e1c8ff-fcc3-4a1f-9730-c617d57b5193" containerName="main" probeResult="failure" output="Get \"https://10.134.0.96:8000/ready\": context deadline exceeded" Oct 30 17:19:00.496808 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:00.487939 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-6e08bd/test-model-6e08bd-kserve-79f874df7-qk2h6" podUID="c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877" containerName="main" probeResult="failure" output="Get \"https://10.134.1.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:19:00.514946 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:19:00.509750 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 17:19:00.653401 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:00.651657 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" podUID="80b9bd9a-30d6-45ff-b318-26fcdc608e5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:19:00.663632 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:00.661084 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" podUID="18b72648-f523-405e-b5cd-1266ff13f8f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:19:01.428075 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:01.428028 4493 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" podUID="18b72648-f523-405e-b5cd-1266ff13f8f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:19:01.667809 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:01.662875 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" podUID="18b72648-f523-405e-b5cd-1266ff13f8f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:19:01.667809 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:01.663498 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" Oct 30 17:19:02.017814 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:01.981946 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" podUID="e43738f3-d55f-4c0a-a364-bf2ac76605c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 17:19:02.110980 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:02.110951 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" Oct 30 17:19:02.607230 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:02.607195 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-85eb87/test-model-85eb87-kserve-686468c445-ssq2z" Oct 30 17:19:02.872034 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:02.871920 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-85eb87/test-model-85eb87-kserve-686468c445-ssq2z" Oct 30 17:19:03.077850 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:03.077809 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-725577c0-cg2v9"] Oct 30 17:19:03.085298 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:03.085269 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-725577c0-cg2v9" Oct 30 17:19:03.108985 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:03.108944 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:19:03.109205 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:03.109181 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:19:03.109361 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:03.109345 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 17:19:03.109511 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:03.109496 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:19:03.127567 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:03.127488 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-725577c0-cg2v9"] Oct 30 17:19:03.243323 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:03.233469 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04ae6844-f50a-401d-b174-3cc4edfd7d18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-725577c0-cg2v9\" (UID: \"04ae6844-f50a-401d-b174-3cc4edfd7d18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-725577c0-cg2v9" Oct 30 17:19:03.248880 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:03.248727 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6b5f\" (UniqueName: \"kubernetes.io/projected/04ae6844-f50a-401d-b174-3cc4edfd7d18-kube-api-access-r6b5f\") pod \"maas-api-revoke-sub-725577c0-cg2v9\" (UID: \"04ae6844-f50a-401d-b174-3cc4edfd7d18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-725577c0-cg2v9" Oct 30 17:19:03.249704 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:03.249675 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04ae6844-f50a-401d-b174-3cc4edfd7d18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-725577c0-cg2v9\" (UID: \"04ae6844-f50a-401d-b174-3cc4edfd7d18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-725577c0-cg2v9" Oct 30 17:19:03.350103 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:03.350075 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-85eb87/test-model-85eb87-kserve-686468c445-ssq2z" Oct 30 17:19:03.355200 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:03.355168 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04ae6844-f50a-401d-b174-3cc4edfd7d18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-725577c0-cg2v9\" (UID: \"04ae6844-f50a-401d-b174-3cc4edfd7d18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-725577c0-cg2v9" Oct 30 17:19:03.355363 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:03.355284 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6b5f\" (UniqueName: \"kubernetes.io/projected/04ae6844-f50a-401d-b174-3cc4edfd7d18-kube-api-access-r6b5f\") pod \"maas-api-revoke-sub-725577c0-cg2v9\" (UID: \"04ae6844-f50a-401d-b174-3cc4edfd7d18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-725577c0-cg2v9" Oct 30 17:19:03.355494 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:03.355477 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04ae6844-f50a-401d-b174-3cc4edfd7d18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-725577c0-cg2v9\" (UID: \"04ae6844-f50a-401d-b174-3cc4edfd7d18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-725577c0-cg2v9" Oct 30 17:19:03.359902 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:03.359862 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04ae6844-f50a-401d-b174-3cc4edfd7d18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-725577c0-cg2v9\" (UID: \"04ae6844-f50a-401d-b174-3cc4edfd7d18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-725577c0-cg2v9" Oct 30 17:19:03.360141 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:03.360116 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04ae6844-f50a-401d-b174-3cc4edfd7d18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-725577c0-cg2v9\" (UID: \"04ae6844-f50a-401d-b174-3cc4edfd7d18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-725577c0-cg2v9" Oct 30 17:19:03.385461 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:03.385389 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6b5f\" (UniqueName: \"kubernetes.io/projected/04ae6844-f50a-401d-b174-3cc4edfd7d18-kube-api-access-r6b5f\") pod \"maas-api-revoke-sub-725577c0-cg2v9\" (UID: \"04ae6844-f50a-401d-b174-3cc4edfd7d18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-725577c0-cg2v9" Oct 30 17:19:03.435866 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:03.435818 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-725577c0-cg2v9" Oct 30 17:19:03.869948 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:03.869917 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-725577c0-cg2v9"] Oct 30 17:19:04.348024 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:04.347989 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-725577c0-cg2v9" event={"ID":"04ae6844-f50a-401d-b174-3cc4edfd7d18","Type":"ContainerStarted","Data":"2f70336b19e8e4774473603a27f241a40b7f60e66bbcf9240de57f87a8c39658"} Oct 30 17:19:05.359816 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:05.359651 4493 generic.go:356] "Generic (PLEG): container finished" podID="04ae6844-f50a-401d-b174-3cc4edfd7d18" containerID="975592ab9e7a3bed678359171ae0bcb5a527a05aefbdb1d0498f7ebf813c6644" exitCode=0 Oct 30 17:19:05.359816 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:05.359711 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-725577c0-cg2v9" event={"ID":"04ae6844-f50a-401d-b174-3cc4edfd7d18","Type":"ContainerDied","Data":"975592ab9e7a3bed678359171ae0bcb5a527a05aefbdb1d0498f7ebf813c6644"} Oct 30 17:19:06.601316 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:06.601285 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-725577c0-cg2v9" Oct 30 17:19:06.683545 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:06.683503 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04ae6844-f50a-401d-b174-3cc4edfd7d18-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04ae6844-f50a-401d-b174-3cc4edfd7d18-maas-api-openshift-ca\") pod \"04ae6844-f50a-401d-b174-3cc4edfd7d18\" (UID: \"04ae6844-f50a-401d-b174-3cc4edfd7d18\") " Oct 30 17:19:06.683759 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:06.683661 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04ae6844-f50a-401d-b174-3cc4edfd7d18-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04ae6844-f50a-401d-b174-3cc4edfd7d18-maas-api-xks-ca\") pod \"04ae6844-f50a-401d-b174-3cc4edfd7d18\" (UID: \"04ae6844-f50a-401d-b174-3cc4edfd7d18\") " Oct 30 17:19:06.683759 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:06.683708 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04ae6844-f50a-401d-b174-3cc4edfd7d18-kube-api-access-r6b5f\" (UniqueName: \"kubernetes.io/projected/04ae6844-f50a-401d-b174-3cc4edfd7d18-kube-api-access-r6b5f\") pod \"04ae6844-f50a-401d-b174-3cc4edfd7d18\" (UID: \"04ae6844-f50a-401d-b174-3cc4edfd7d18\") " Oct 30 17:19:06.683949 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:06.683898 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04ae6844-f50a-401d-b174-3cc4edfd7d18-maas-api-openshift-ca" pod "04ae6844-f50a-401d-b174-3cc4edfd7d18" (UID: "04ae6844-f50a-401d-b174-3cc4edfd7d18"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:19:06.684750 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:06.684723 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04ae6844-f50a-401d-b174-3cc4edfd7d18-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:19:06.686117 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:06.686084 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04ae6844-f50a-401d-b174-3cc4edfd7d18-maas-api-xks-ca" pod "04ae6844-f50a-401d-b174-3cc4edfd7d18" (UID: "04ae6844-f50a-401d-b174-3cc4edfd7d18"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:19:06.686229 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:06.686185 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04ae6844-f50a-401d-b174-3cc4edfd7d18-kube-api-access-r6b5f" pod "04ae6844-f50a-401d-b174-3cc4edfd7d18" (UID: "04ae6844-f50a-401d-b174-3cc4edfd7d18"). InnerVolumeSpecName "kube-api-access-r6b5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:19:06.786433 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:06.786392 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04ae6844-f50a-401d-b174-3cc4edfd7d18-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:19:06.786433 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:06.786432 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6b5f\" (UniqueName: \"kubernetes.io/projected/04ae6844-f50a-401d-b174-3cc4edfd7d18-kube-api-access-r6b5f\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:19:07.391310 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:07.391277 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-725577c0-cg2v9" event={"ID":"04ae6844-f50a-401d-b174-3cc4edfd7d18","Type":"ContainerDied","Data":"2f70336b19e8e4774473603a27f241a40b7f60e66bbcf9240de57f87a8c39658"} Oct 30 17:19:07.391496 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:07.391291 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-725577c0-cg2v9" Oct 30 17:19:07.391496 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:07.391399 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f70336b19e8e4774473603a27f241a40b7f60e66bbcf9240de57f87a8c39658" Oct 30 17:19:15.647600 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:15.645854 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-24acb2/test-model-24acb2-kserve-946ff6bb7-h8cjs"] Oct 30 17:19:16.063001 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:16.062906 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6acc78-zbdt2"] Oct 30 17:19:16.064425 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:16.064395 4493 cpu_manager.go:414] "containerMap: removing container" podUID="04ae6844-f50a-401d-b174-3cc4edfd7d18" containerName="revoke-keys" Oct 30 17:19:16.064606 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:16.064591 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04ae6844-f50a-401d-b174-3cc4edfd7d18" containerName="revoke-keys" Oct 30 17:19:16.068078 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:16.068055 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="04ae6844-f50a-401d-b174-3cc4edfd7d18" containerName="revoke-keys" Oct 30 17:19:16.092805 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:16.091333 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6acc78-zbdt2" Oct 30 17:19:16.101029 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:16.091238 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6acc78-zbdt2"] Oct 30 17:19:16.107416 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:16.107384 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:19:16.108121 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:16.108101 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 17:19:16.108235 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:16.108175 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:19:16.108306 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:16.108234 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:19:16.149953 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:16.149916 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8jxl\" (UniqueName: \"kubernetes.io/projected/172ba705-a53b-4b3b-b91e-28ef5d55075a-kube-api-access-v8jxl\") pod \"maas-api-revoke-keys-e2e-inf-a-6acc78-zbdt2\" (UID: \"172ba705-a53b-4b3b-b91e-28ef5d55075a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6acc78-zbdt2" Oct 30 17:19:16.150147 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:16.149983 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/172ba705-a53b-4b3b-b91e-28ef5d55075a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6acc78-zbdt2\" (UID: \"172ba705-a53b-4b3b-b91e-28ef5d55075a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6acc78-zbdt2" Oct 30 17:19:16.150210 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:16.150142 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/172ba705-a53b-4b3b-b91e-28ef5d55075a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6acc78-zbdt2\" (UID: \"172ba705-a53b-4b3b-b91e-28ef5d55075a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6acc78-zbdt2" Oct 30 17:19:16.252758 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:16.252723 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/172ba705-a53b-4b3b-b91e-28ef5d55075a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6acc78-zbdt2\" (UID: \"172ba705-a53b-4b3b-b91e-28ef5d55075a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6acc78-zbdt2" Oct 30 17:19:16.252960 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:16.252921 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/172ba705-a53b-4b3b-b91e-28ef5d55075a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6acc78-zbdt2\" (UID: \"172ba705-a53b-4b3b-b91e-28ef5d55075a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6acc78-zbdt2" Oct 30 17:19:16.253208 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:16.253186 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8jxl\" (UniqueName: \"kubernetes.io/projected/172ba705-a53b-4b3b-b91e-28ef5d55075a-kube-api-access-v8jxl\") pod \"maas-api-revoke-keys-e2e-inf-a-6acc78-zbdt2\" (UID: \"172ba705-a53b-4b3b-b91e-28ef5d55075a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6acc78-zbdt2" Oct 30 17:19:16.253574 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:16.253550 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/172ba705-a53b-4b3b-b91e-28ef5d55075a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6acc78-zbdt2\" (UID: \"172ba705-a53b-4b3b-b91e-28ef5d55075a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6acc78-zbdt2" Oct 30 17:19:16.255698 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:16.255672 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/172ba705-a53b-4b3b-b91e-28ef5d55075a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6acc78-zbdt2\" (UID: \"172ba705-a53b-4b3b-b91e-28ef5d55075a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6acc78-zbdt2" Oct 30 17:19:16.276799 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:16.276757 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8jxl\" (UniqueName: \"kubernetes.io/projected/172ba705-a53b-4b3b-b91e-28ef5d55075a-kube-api-access-v8jxl\") pod \"maas-api-revoke-keys-e2e-inf-a-6acc78-zbdt2\" (UID: \"172ba705-a53b-4b3b-b91e-28ef5d55075a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6acc78-zbdt2" Oct 30 17:19:16.440265 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:16.440236 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6acc78-zbdt2" Oct 30 17:19:16.755539 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:16.755498 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6acc78-zbdt2"] Oct 30 17:19:16.756090 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:19:16.756063 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod172ba705_a53b_4b3b_b91e_28ef5d55075a.slice/crio-192045a465bf7b0d8adb7bbf249adf010632d5e604e7b1c844d45151425adb69 WatchSource:0}: Error finding container 192045a465bf7b0d8adb7bbf249adf010632d5e604e7b1c844d45151425adb69: Status 404 returned error can't find the container with id 192045a465bf7b0d8adb7bbf249adf010632d5e604e7b1c844d45151425adb69 Oct 30 17:19:17.563213 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:17.563117 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6acc78-zbdt2" event={"ID":"172ba705-a53b-4b3b-b91e-28ef5d55075a","Type":"ContainerStarted","Data":"5057cd19efef10c441a82b5982bd901396945aa8a69fd76c9d434829b9d22a3e"} Oct 30 17:19:17.563213 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:17.563198 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6acc78-zbdt2" event={"ID":"172ba705-a53b-4b3b-b91e-28ef5d55075a","Type":"ContainerStarted","Data":"192045a465bf7b0d8adb7bbf249adf010632d5e604e7b1c844d45151425adb69"} Oct 30 17:19:17.605834 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:17.604573 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6acc78-zbdt2" podStartSLOduration=2.604556132 podStartE2EDuration="2.604556132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:19:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:19:17.603318642 +0000 UTC m=+3384.830493742" watchObservedRunningTime="2026-10-30 17:19:17.604556132 +0000 UTC m=+3384.831731233" Oct 30 17:19:18.574071 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:18.574038 4493 generic.go:356] "Generic (PLEG): container finished" podID="172ba705-a53b-4b3b-b91e-28ef5d55075a" containerID="5057cd19efef10c441a82b5982bd901396945aa8a69fd76c9d434829b9d22a3e" exitCode=0 Oct 30 17:19:18.574526 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:18.574130 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6acc78-zbdt2" event={"ID":"172ba705-a53b-4b3b-b91e-28ef5d55075a","Type":"ContainerDied","Data":"5057cd19efef10c441a82b5982bd901396945aa8a69fd76c9d434829b9d22a3e"} Oct 30 17:19:19.794541 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:19.794515 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6acc78-zbdt2" Oct 30 17:19:19.922238 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:19.922204 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/172ba705-a53b-4b3b-b91e-28ef5d55075a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/172ba705-a53b-4b3b-b91e-28ef5d55075a-maas-api-openshift-ca\") pod \"172ba705-a53b-4b3b-b91e-28ef5d55075a\" (UID: \"172ba705-a53b-4b3b-b91e-28ef5d55075a\") " Oct 30 17:19:19.922454 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:19.922347 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/172ba705-a53b-4b3b-b91e-28ef5d55075a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/172ba705-a53b-4b3b-b91e-28ef5d55075a-maas-api-xks-ca\") pod \"172ba705-a53b-4b3b-b91e-28ef5d55075a\" (UID: \"172ba705-a53b-4b3b-b91e-28ef5d55075a\") " Oct 30 17:19:19.922454 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:19.922401 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/172ba705-a53b-4b3b-b91e-28ef5d55075a-kube-api-access-v8jxl\" (UniqueName: \"kubernetes.io/projected/172ba705-a53b-4b3b-b91e-28ef5d55075a-kube-api-access-v8jxl\") pod \"172ba705-a53b-4b3b-b91e-28ef5d55075a\" (UID: \"172ba705-a53b-4b3b-b91e-28ef5d55075a\") " Oct 30 17:19:19.922662 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:19.922638 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/172ba705-a53b-4b3b-b91e-28ef5d55075a-maas-api-openshift-ca" pod "172ba705-a53b-4b3b-b91e-28ef5d55075a" (UID: "172ba705-a53b-4b3b-b91e-28ef5d55075a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:19:19.923194 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:19.923128 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/172ba705-a53b-4b3b-b91e-28ef5d55075a-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:19:19.924688 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:19.924663 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/172ba705-a53b-4b3b-b91e-28ef5d55075a-maas-api-xks-ca" pod "172ba705-a53b-4b3b-b91e-28ef5d55075a" (UID: "172ba705-a53b-4b3b-b91e-28ef5d55075a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:19:19.924842 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:19.924752 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/172ba705-a53b-4b3b-b91e-28ef5d55075a-kube-api-access-v8jxl" pod "172ba705-a53b-4b3b-b91e-28ef5d55075a" (UID: "172ba705-a53b-4b3b-b91e-28ef5d55075a"). InnerVolumeSpecName "kube-api-access-v8jxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:19:20.024748 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:20.024719 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/172ba705-a53b-4b3b-b91e-28ef5d55075a-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:19:20.024748 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:20.024747 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8jxl\" (UniqueName: \"kubernetes.io/projected/172ba705-a53b-4b3b-b91e-28ef5d55075a-kube-api-access-v8jxl\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:19:20.618387 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:20.618356 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6acc78-zbdt2" Oct 30 17:19:20.618566 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:20.618361 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6acc78-zbdt2" event={"ID":"172ba705-a53b-4b3b-b91e-28ef5d55075a","Type":"ContainerDied","Data":"192045a465bf7b0d8adb7bbf249adf010632d5e604e7b1c844d45151425adb69"} Oct 30 17:19:20.618566 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:20.618477 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="192045a465bf7b0d8adb7bbf249adf010632d5e604e7b1c844d45151425adb69" Oct 30 17:19:30.351016 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:19:30.350975 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 17:19:30.686954 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:30.686918 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-24acb2/test-model-24acb2-kserve-946ff6bb7-h8cjs" podUID="cd1ff57b-1a55-4ef7-95f6-ee9389a00d99" containerName="main" containerID="cri-o://7025845d11a69595f9dca24bea1b1681eaf5b1f436eecd5dd4f95b6bb52ba2af" gracePeriod=45 Oct 30 17:19:31.093012 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:31.092987 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-24acb2/test-model-24acb2-kserve-946ff6bb7-h8cjs" Oct 30 17:19:31.235582 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:31.235547 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd1ff57b-1a55-4ef7-95f6-ee9389a00d99-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd1ff57b-1a55-4ef7-95f6-ee9389a00d99-tmp-dir\") pod \"cd1ff57b-1a55-4ef7-95f6-ee9389a00d99\" (UID: \"cd1ff57b-1a55-4ef7-95f6-ee9389a00d99\") " Oct 30 17:19:31.235772 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:31.235669 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd1ff57b-1a55-4ef7-95f6-ee9389a00d99-kube-api-access-jq74z\" (UniqueName: \"kubernetes.io/projected/cd1ff57b-1a55-4ef7-95f6-ee9389a00d99-kube-api-access-jq74z\") pod \"cd1ff57b-1a55-4ef7-95f6-ee9389a00d99\" (UID: \"cd1ff57b-1a55-4ef7-95f6-ee9389a00d99\") " Oct 30 17:19:31.235862 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:31.235833 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd1ff57b-1a55-4ef7-95f6-ee9389a00d99-home\" (UniqueName: \"kubernetes.io/empty-dir/cd1ff57b-1a55-4ef7-95f6-ee9389a00d99-home\") pod \"cd1ff57b-1a55-4ef7-95f6-ee9389a00d99\" (UID: \"cd1ff57b-1a55-4ef7-95f6-ee9389a00d99\") " Oct 30 17:19:31.235916 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:31.235887 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd1ff57b-1a55-4ef7-95f6-ee9389a00d99-tmp-dir" pod "cd1ff57b-1a55-4ef7-95f6-ee9389a00d99" (UID: "cd1ff57b-1a55-4ef7-95f6-ee9389a00d99"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:19:31.236040 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:31.236025 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd1ff57b-1a55-4ef7-95f6-ee9389a00d99-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd1ff57b-1a55-4ef7-95f6-ee9389a00d99-model-cache\") pod \"cd1ff57b-1a55-4ef7-95f6-ee9389a00d99\" (UID: \"cd1ff57b-1a55-4ef7-95f6-ee9389a00d99\") " Oct 30 17:19:31.236109 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:31.236048 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd1ff57b-1a55-4ef7-95f6-ee9389a00d99-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd1ff57b-1a55-4ef7-95f6-ee9389a00d99-dshm\") pod \"cd1ff57b-1a55-4ef7-95f6-ee9389a00d99\" (UID: \"cd1ff57b-1a55-4ef7-95f6-ee9389a00d99\") " Oct 30 17:19:31.236155 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:31.236136 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd1ff57b-1a55-4ef7-95f6-ee9389a00d99-home" pod "cd1ff57b-1a55-4ef7-95f6-ee9389a00d99" (UID: "cd1ff57b-1a55-4ef7-95f6-ee9389a00d99"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:19:31.236242 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:31.236209 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd1ff57b-1a55-4ef7-95f6-ee9389a00d99-tls-certs\" (UniqueName: \"kubernetes.io/secret/cd1ff57b-1a55-4ef7-95f6-ee9389a00d99-tls-certs\") pod \"cd1ff57b-1a55-4ef7-95f6-ee9389a00d99\" (UID: \"cd1ff57b-1a55-4ef7-95f6-ee9389a00d99\") " Oct 30 17:19:31.236293 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:31.236273 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd1ff57b-1a55-4ef7-95f6-ee9389a00d99-model-cache" pod "cd1ff57b-1a55-4ef7-95f6-ee9389a00d99" (UID: "cd1ff57b-1a55-4ef7-95f6-ee9389a00d99"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:19:31.237286 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:31.237258 4493 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd1ff57b-1a55-4ef7-95f6-ee9389a00d99-tmp-dir\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:19:31.237420 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:31.237289 4493 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd1ff57b-1a55-4ef7-95f6-ee9389a00d99-home\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:19:31.237420 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:31.237302 4493 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd1ff57b-1a55-4ef7-95f6-ee9389a00d99-model-cache\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:19:31.238371 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:31.238341 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd1ff57b-1a55-4ef7-95f6-ee9389a00d99-kube-api-access-jq74z" pod "cd1ff57b-1a55-4ef7-95f6-ee9389a00d99" (UID: "cd1ff57b-1a55-4ef7-95f6-ee9389a00d99"). InnerVolumeSpecName "kube-api-access-jq74z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:19:31.238679 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:31.238659 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd1ff57b-1a55-4ef7-95f6-ee9389a00d99-dshm" pod "cd1ff57b-1a55-4ef7-95f6-ee9389a00d99" (UID: "cd1ff57b-1a55-4ef7-95f6-ee9389a00d99"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:19:31.238994 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:31.238892 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd1ff57b-1a55-4ef7-95f6-ee9389a00d99-tls-certs" pod "cd1ff57b-1a55-4ef7-95f6-ee9389a00d99" (UID: "cd1ff57b-1a55-4ef7-95f6-ee9389a00d99"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:19:31.338978 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:31.338902 4493 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd1ff57b-1a55-4ef7-95f6-ee9389a00d99-tls-certs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:19:31.338978 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:31.338938 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jq74z\" (UniqueName: \"kubernetes.io/projected/cd1ff57b-1a55-4ef7-95f6-ee9389a00d99-kube-api-access-jq74z\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:19:31.338978 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:31.338951 4493 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd1ff57b-1a55-4ef7-95f6-ee9389a00d99-dshm\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:19:31.833206 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:31.833155 4493 generic.go:356] "Generic (PLEG): container finished" podID="cd1ff57b-1a55-4ef7-95f6-ee9389a00d99" containerID="7025845d11a69595f9dca24bea1b1681eaf5b1f436eecd5dd4f95b6bb52ba2af" exitCode=0 Oct 30 17:19:31.833698 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:31.833219 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-24acb2/test-model-24acb2-kserve-946ff6bb7-h8cjs" Oct 30 17:19:31.833698 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:31.833281 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-24acb2/test-model-24acb2-kserve-946ff6bb7-h8cjs" event={"ID":"cd1ff57b-1a55-4ef7-95f6-ee9389a00d99","Type":"ContainerDied","Data":"7025845d11a69595f9dca24bea1b1681eaf5b1f436eecd5dd4f95b6bb52ba2af"} Oct 30 17:19:31.833698 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:31.833314 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-24acb2/test-model-24acb2-kserve-946ff6bb7-h8cjs" event={"ID":"cd1ff57b-1a55-4ef7-95f6-ee9389a00d99","Type":"ContainerDied","Data":"b814e24362b807a9f41dfac9d23de70d87291ca610d88428f0e344975e5d07ba"} Oct 30 17:19:31.833698 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:31.833336 4493 scope.go:122] "RemoveContainer" containerID="7025845d11a69595f9dca24bea1b1681eaf5b1f436eecd5dd4f95b6bb52ba2af" Oct 30 17:19:31.862056 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:31.862032 4493 scope.go:122] "RemoveContainer" containerID="7025845d11a69595f9dca24bea1b1681eaf5b1f436eecd5dd4f95b6bb52ba2af" Oct 30 17:19:31.862393 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:19:31.862372 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7025845d11a69595f9dca24bea1b1681eaf5b1f436eecd5dd4f95b6bb52ba2af\": container with ID starting with 7025845d11a69595f9dca24bea1b1681eaf5b1f436eecd5dd4f95b6bb52ba2af not found: ID does not exist" containerID="7025845d11a69595f9dca24bea1b1681eaf5b1f436eecd5dd4f95b6bb52ba2af" Oct 30 17:19:31.862454 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:31.862402 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7025845d11a69595f9dca24bea1b1681eaf5b1f436eecd5dd4f95b6bb52ba2af"} err="failed to get container status \"7025845d11a69595f9dca24bea1b1681eaf5b1f436eecd5dd4f95b6bb52ba2af\": rpc error: code = NotFound desc = could not find container \"7025845d11a69595f9dca24bea1b1681eaf5b1f436eecd5dd4f95b6bb52ba2af\": container with ID starting with 7025845d11a69595f9dca24bea1b1681eaf5b1f436eecd5dd4f95b6bb52ba2af not found: ID does not exist" Oct 30 17:19:31.887832 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:31.887804 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-24acb2/test-model-24acb2-kserve-946ff6bb7-h8cjs"] Oct 30 17:19:31.899515 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:31.899480 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-24acb2/test-model-24acb2-kserve-946ff6bb7-h8cjs"] Oct 30 17:19:32.938574 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:32.938541 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd1ff57b-1a55-4ef7-95f6-ee9389a00d99" path="/var/lib/kubelet/pods/cd1ff57b-1a55-4ef7-95f6-ee9389a00d99/volumes" Oct 30 17:19:34.913613 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:34.913576 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q2t9l"] Oct 30 17:19:34.914341 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:34.913876 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q2t9l" podUID="98c9aa43-9e50-4612-9965-4a20f9458deb" containerName="istio-proxy" containerID="cri-o://989d496ae2a43920deeb5428d4c4342649a7a36d786be51b7fab7d6c28a8f2a1" gracePeriod=30 Oct 30 17:19:40.206624 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:40.206557 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q2t9l" Oct 30 17:19:40.304257 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:40.304220 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98c9aa43-9e50-4612-9965-4a20f9458deb-istio-token\" (UniqueName: \"kubernetes.io/projected/98c9aa43-9e50-4612-9965-4a20f9458deb-istio-token\") pod \"98c9aa43-9e50-4612-9965-4a20f9458deb\" (UID: \"98c9aa43-9e50-4612-9965-4a20f9458deb\") " Oct 30 17:19:40.304448 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:40.304291 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98c9aa43-9e50-4612-9965-4a20f9458deb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98c9aa43-9e50-4612-9965-4a20f9458deb-credential-socket\") pod \"98c9aa43-9e50-4612-9965-4a20f9458deb\" (UID: \"98c9aa43-9e50-4612-9965-4a20f9458deb\") " Oct 30 17:19:40.304448 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:40.304356 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98c9aa43-9e50-4612-9965-4a20f9458deb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98c9aa43-9e50-4612-9965-4a20f9458deb-workload-socket\") pod \"98c9aa43-9e50-4612-9965-4a20f9458deb\" (UID: \"98c9aa43-9e50-4612-9965-4a20f9458deb\") " Oct 30 17:19:40.304580 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:40.304468 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98c9aa43-9e50-4612-9965-4a20f9458deb-kube-api-access-6dczl\" (UniqueName: \"kubernetes.io/projected/98c9aa43-9e50-4612-9965-4a20f9458deb-kube-api-access-6dczl\") pod \"98c9aa43-9e50-4612-9965-4a20f9458deb\" (UID: \"98c9aa43-9e50-4612-9965-4a20f9458deb\") " Oct 30 17:19:40.304580 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:40.304511 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/98c9aa43-9e50-4612-9965-4a20f9458deb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98c9aa43-9e50-4612-9965-4a20f9458deb-istio-podinfo\") pod \"98c9aa43-9e50-4612-9965-4a20f9458deb\" (UID: \"98c9aa43-9e50-4612-9965-4a20f9458deb\") " Oct 30 17:19:40.304684 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:40.304638 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98c9aa43-9e50-4612-9965-4a20f9458deb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98c9aa43-9e50-4612-9965-4a20f9458deb-istiod-ca-cert\") pod \"98c9aa43-9e50-4612-9965-4a20f9458deb\" (UID: \"98c9aa43-9e50-4612-9965-4a20f9458deb\") " Oct 30 17:19:40.304745 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:40.304685 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98c9aa43-9e50-4612-9965-4a20f9458deb-credential-socket" pod "98c9aa43-9e50-4612-9965-4a20f9458deb" (UID: "98c9aa43-9e50-4612-9965-4a20f9458deb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:19:40.304745 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:40.304706 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98c9aa43-9e50-4612-9965-4a20f9458deb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98c9aa43-9e50-4612-9965-4a20f9458deb-istio-envoy\") pod \"98c9aa43-9e50-4612-9965-4a20f9458deb\" (UID: \"98c9aa43-9e50-4612-9965-4a20f9458deb\") " Oct 30 17:19:40.304875 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:40.304755 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98c9aa43-9e50-4612-9965-4a20f9458deb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98c9aa43-9e50-4612-9965-4a20f9458deb-workload-certs\") pod \"98c9aa43-9e50-4612-9965-4a20f9458deb\" (UID: \"98c9aa43-9e50-4612-9965-4a20f9458deb\") " Oct 30 17:19:40.304936 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:40.304911 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98c9aa43-9e50-4612-9965-4a20f9458deb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98c9aa43-9e50-4612-9965-4a20f9458deb-istio-data\") pod \"98c9aa43-9e50-4612-9965-4a20f9458deb\" (UID: \"98c9aa43-9e50-4612-9965-4a20f9458deb\") " Oct 30 17:19:40.305154 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:40.305121 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98c9aa43-9e50-4612-9965-4a20f9458deb-istiod-ca-cert" pod "98c9aa43-9e50-4612-9965-4a20f9458deb" (UID: "98c9aa43-9e50-4612-9965-4a20f9458deb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:19:40.306081 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:40.305770 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98c9aa43-9e50-4612-9965-4a20f9458deb-workload-certs" pod "98c9aa43-9e50-4612-9965-4a20f9458deb" (UID: "98c9aa43-9e50-4612-9965-4a20f9458deb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:19:40.306081 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:40.306056 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98c9aa43-9e50-4612-9965-4a20f9458deb-istio-data" pod "98c9aa43-9e50-4612-9965-4a20f9458deb" (UID: "98c9aa43-9e50-4612-9965-4a20f9458deb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:19:40.306318 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:40.306294 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98c9aa43-9e50-4612-9965-4a20f9458deb-workload-socket" pod "98c9aa43-9e50-4612-9965-4a20f9458deb" (UID: "98c9aa43-9e50-4612-9965-4a20f9458deb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:19:40.306535 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:40.306409 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98c9aa43-9e50-4612-9965-4a20f9458deb-istio-data\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:19:40.306535 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:40.306435 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98c9aa43-9e50-4612-9965-4a20f9458deb-credential-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:19:40.306535 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:40.306450 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98c9aa43-9e50-4612-9965-4a20f9458deb-istiod-ca-cert\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:19:40.306535 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:40.306464 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98c9aa43-9e50-4612-9965-4a20f9458deb-workload-certs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:19:40.308814 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:40.308761 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98c9aa43-9e50-4612-9965-4a20f9458deb-istio-envoy" pod "98c9aa43-9e50-4612-9965-4a20f9458deb" (UID: "98c9aa43-9e50-4612-9965-4a20f9458deb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:19:40.309572 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:40.309547 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/98c9aa43-9e50-4612-9965-4a20f9458deb-istio-podinfo" pod "98c9aa43-9e50-4612-9965-4a20f9458deb" (UID: "98c9aa43-9e50-4612-9965-4a20f9458deb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:19:40.316835 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:40.316767 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98c9aa43-9e50-4612-9965-4a20f9458deb-istio-token" pod "98c9aa43-9e50-4612-9965-4a20f9458deb" (UID: "98c9aa43-9e50-4612-9965-4a20f9458deb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:19:40.319755 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:40.319728 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98c9aa43-9e50-4612-9965-4a20f9458deb-kube-api-access-6dczl" pod "98c9aa43-9e50-4612-9965-4a20f9458deb" (UID: "98c9aa43-9e50-4612-9965-4a20f9458deb"). InnerVolumeSpecName "kube-api-access-6dczl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:19:40.409030 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:40.408987 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dczl\" (UniqueName: \"kubernetes.io/projected/98c9aa43-9e50-4612-9965-4a20f9458deb-kube-api-access-6dczl\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:19:40.409030 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:40.409023 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98c9aa43-9e50-4612-9965-4a20f9458deb-istio-podinfo\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:19:40.409247 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:40.409038 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98c9aa43-9e50-4612-9965-4a20f9458deb-istio-envoy\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:19:40.409247 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:40.409052 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98c9aa43-9e50-4612-9965-4a20f9458deb-istio-token\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:19:40.409247 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:40.409068 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98c9aa43-9e50-4612-9965-4a20f9458deb-workload-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:19:40.920424 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:40.920388 4493 generic.go:356] "Generic (PLEG): container finished" podID="98c9aa43-9e50-4612-9965-4a20f9458deb" containerID="989d496ae2a43920deeb5428d4c4342649a7a36d786be51b7fab7d6c28a8f2a1" exitCode=0 Oct 30 17:19:40.920624 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:40.920438 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q2t9l" event={"ID":"98c9aa43-9e50-4612-9965-4a20f9458deb","Type":"ContainerDied","Data":"989d496ae2a43920deeb5428d4c4342649a7a36d786be51b7fab7d6c28a8f2a1"} Oct 30 17:19:40.920624 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:40.920460 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q2t9l" Oct 30 17:19:40.920624 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:40.920498 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q2t9l" event={"ID":"98c9aa43-9e50-4612-9965-4a20f9458deb","Type":"ContainerDied","Data":"fe82f4bc1e48b20ebe6abee23c91700f058f0aeec06dcf4032feb0a02a6bf225"} Oct 30 17:19:40.920624 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:40.920522 4493 scope.go:122] "RemoveContainer" containerID="989d496ae2a43920deeb5428d4c4342649a7a36d786be51b7fab7d6c28a8f2a1" Oct 30 17:19:40.935137 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:40.935110 4493 scope.go:122] "RemoveContainer" containerID="989d496ae2a43920deeb5428d4c4342649a7a36d786be51b7fab7d6c28a8f2a1" Oct 30 17:19:40.935460 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:19:40.935434 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"989d496ae2a43920deeb5428d4c4342649a7a36d786be51b7fab7d6c28a8f2a1\": container with ID starting with 989d496ae2a43920deeb5428d4c4342649a7a36d786be51b7fab7d6c28a8f2a1 not found: ID does not exist" containerID="989d496ae2a43920deeb5428d4c4342649a7a36d786be51b7fab7d6c28a8f2a1" Oct 30 17:19:40.935549 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:40.935471 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"989d496ae2a43920deeb5428d4c4342649a7a36d786be51b7fab7d6c28a8f2a1"} err="failed to get container status \"989d496ae2a43920deeb5428d4c4342649a7a36d786be51b7fab7d6c28a8f2a1\": rpc error: code = NotFound desc = could not find container \"989d496ae2a43920deeb5428d4c4342649a7a36d786be51b7fab7d6c28a8f2a1\": container with ID starting with 989d496ae2a43920deeb5428d4c4342649a7a36d786be51b7fab7d6c28a8f2a1 not found: ID does not exist" Oct 30 17:19:40.987726 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:40.987570 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q2t9l"] Oct 30 17:19:41.000683 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:41.000654 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-q2t9l"] Oct 30 17:19:42.942021 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:42.941582 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98c9aa43-9e50-4612-9965-4a20f9458deb" path="/var/lib/kubelet/pods/98c9aa43-9e50-4612-9965-4a20f9458deb/volumes" Oct 30 17:19:50.937722 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:50.937682 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk"] Oct 30 17:19:50.938229 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:50.937967 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk" podUID="26356b33-b847-4ba2-8b73-cfa37b2b14a4" containerName="istio-proxy" containerID="cri-o://46e37164664f969bb2947cf53a46c36d20379cd52502957c2a5c3ad4ac767fe4" gracePeriod=30 Oct 30 17:19:56.099296 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:56.099259 4493 generic.go:356] "Generic (PLEG): container finished" podID="26356b33-b847-4ba2-8b73-cfa37b2b14a4" containerID="46e37164664f969bb2947cf53a46c36d20379cd52502957c2a5c3ad4ac767fe4" exitCode=0 Oct 30 17:19:56.099714 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:56.099330 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk" event={"ID":"26356b33-b847-4ba2-8b73-cfa37b2b14a4","Type":"ContainerDied","Data":"46e37164664f969bb2947cf53a46c36d20379cd52502957c2a5c3ad4ac767fe4"} Oct 30 17:19:56.264058 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:56.264029 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk" Oct 30 17:19:56.412808 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:56.400064 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26356b33-b847-4ba2-8b73-cfa37b2b14a4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26356b33-b847-4ba2-8b73-cfa37b2b14a4-istiod-ca-cert\") pod \"26356b33-b847-4ba2-8b73-cfa37b2b14a4\" (UID: \"26356b33-b847-4ba2-8b73-cfa37b2b14a4\") " Oct 30 17:19:56.412808 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:56.400219 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26356b33-b847-4ba2-8b73-cfa37b2b14a4-istio-token\" (UniqueName: \"kubernetes.io/projected/26356b33-b847-4ba2-8b73-cfa37b2b14a4-istio-token\") pod \"26356b33-b847-4ba2-8b73-cfa37b2b14a4\" (UID: \"26356b33-b847-4ba2-8b73-cfa37b2b14a4\") " Oct 30 17:19:56.412808 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:56.400283 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26356b33-b847-4ba2-8b73-cfa37b2b14a4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26356b33-b847-4ba2-8b73-cfa37b2b14a4-workload-socket\") pod \"26356b33-b847-4ba2-8b73-cfa37b2b14a4\" (UID: \"26356b33-b847-4ba2-8b73-cfa37b2b14a4\") " Oct 30 17:19:56.412808 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:56.400383 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26356b33-b847-4ba2-8b73-cfa37b2b14a4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26356b33-b847-4ba2-8b73-cfa37b2b14a4-istio-envoy\") pod \"26356b33-b847-4ba2-8b73-cfa37b2b14a4\" (UID: \"26356b33-b847-4ba2-8b73-cfa37b2b14a4\") " Oct 30 17:19:56.412808 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:56.402181 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26356b33-b847-4ba2-8b73-cfa37b2b14a4-workload-socket" pod "26356b33-b847-4ba2-8b73-cfa37b2b14a4" (UID: "26356b33-b847-4ba2-8b73-cfa37b2b14a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:19:56.412808 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:56.403039 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26356b33-b847-4ba2-8b73-cfa37b2b14a4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26356b33-b847-4ba2-8b73-cfa37b2b14a4-istio-data\") pod \"26356b33-b847-4ba2-8b73-cfa37b2b14a4\" (UID: \"26356b33-b847-4ba2-8b73-cfa37b2b14a4\") " Oct 30 17:19:56.412808 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:56.403237 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26356b33-b847-4ba2-8b73-cfa37b2b14a4-kube-api-access-qpgqj\" (UniqueName: \"kubernetes.io/projected/26356b33-b847-4ba2-8b73-cfa37b2b14a4-kube-api-access-qpgqj\") pod \"26356b33-b847-4ba2-8b73-cfa37b2b14a4\" (UID: \"26356b33-b847-4ba2-8b73-cfa37b2b14a4\") " Oct 30 17:19:56.412808 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:56.403284 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26356b33-b847-4ba2-8b73-cfa37b2b14a4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26356b33-b847-4ba2-8b73-cfa37b2b14a4-workload-certs\") pod \"26356b33-b847-4ba2-8b73-cfa37b2b14a4\" (UID: \"26356b33-b847-4ba2-8b73-cfa37b2b14a4\") " Oct 30 17:19:56.412808 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:56.403328 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26356b33-b847-4ba2-8b73-cfa37b2b14a4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26356b33-b847-4ba2-8b73-cfa37b2b14a4-credential-socket\") pod \"26356b33-b847-4ba2-8b73-cfa37b2b14a4\" (UID: \"26356b33-b847-4ba2-8b73-cfa37b2b14a4\") " Oct 30 17:19:56.412808 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:56.403362 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/26356b33-b847-4ba2-8b73-cfa37b2b14a4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26356b33-b847-4ba2-8b73-cfa37b2b14a4-istio-podinfo\") pod \"26356b33-b847-4ba2-8b73-cfa37b2b14a4\" (UID: \"26356b33-b847-4ba2-8b73-cfa37b2b14a4\") " Oct 30 17:19:56.412808 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:56.403453 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26356b33-b847-4ba2-8b73-cfa37b2b14a4-istiod-ca-cert" pod "26356b33-b847-4ba2-8b73-cfa37b2b14a4" (UID: "26356b33-b847-4ba2-8b73-cfa37b2b14a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:19:56.412808 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:56.404096 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26356b33-b847-4ba2-8b73-cfa37b2b14a4-istio-data" pod "26356b33-b847-4ba2-8b73-cfa37b2b14a4" (UID: "26356b33-b847-4ba2-8b73-cfa37b2b14a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:19:56.412808 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:56.404122 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26356b33-b847-4ba2-8b73-cfa37b2b14a4-istiod-ca-cert\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:19:56.412808 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:56.404141 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26356b33-b847-4ba2-8b73-cfa37b2b14a4-workload-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:19:56.412808 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:56.404304 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26356b33-b847-4ba2-8b73-cfa37b2b14a4-workload-certs" pod "26356b33-b847-4ba2-8b73-cfa37b2b14a4" (UID: "26356b33-b847-4ba2-8b73-cfa37b2b14a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:19:56.412808 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:56.404352 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26356b33-b847-4ba2-8b73-cfa37b2b14a4-credential-socket" pod "26356b33-b847-4ba2-8b73-cfa37b2b14a4" (UID: "26356b33-b847-4ba2-8b73-cfa37b2b14a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:19:56.412808 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:56.410709 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/26356b33-b847-4ba2-8b73-cfa37b2b14a4-istio-podinfo" pod "26356b33-b847-4ba2-8b73-cfa37b2b14a4" (UID: "26356b33-b847-4ba2-8b73-cfa37b2b14a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:19:56.425396 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:56.420409 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26356b33-b847-4ba2-8b73-cfa37b2b14a4-istio-token" pod "26356b33-b847-4ba2-8b73-cfa37b2b14a4" (UID: "26356b33-b847-4ba2-8b73-cfa37b2b14a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:19:56.425396 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:56.420976 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26356b33-b847-4ba2-8b73-cfa37b2b14a4-kube-api-access-qpgqj" pod "26356b33-b847-4ba2-8b73-cfa37b2b14a4" (UID: "26356b33-b847-4ba2-8b73-cfa37b2b14a4"). InnerVolumeSpecName "kube-api-access-qpgqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:19:56.428684 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:56.427269 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26356b33-b847-4ba2-8b73-cfa37b2b14a4-istio-envoy" pod "26356b33-b847-4ba2-8b73-cfa37b2b14a4" (UID: "26356b33-b847-4ba2-8b73-cfa37b2b14a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:19:56.505484 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:56.505394 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26356b33-b847-4ba2-8b73-cfa37b2b14a4-istio-token\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:19:56.505484 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:56.505422 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26356b33-b847-4ba2-8b73-cfa37b2b14a4-istio-envoy\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:19:56.505484 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:56.505432 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26356b33-b847-4ba2-8b73-cfa37b2b14a4-istio-data\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:19:56.505484 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:56.505440 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpgqj\" (UniqueName: \"kubernetes.io/projected/26356b33-b847-4ba2-8b73-cfa37b2b14a4-kube-api-access-qpgqj\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:19:56.505484 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:56.505452 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26356b33-b847-4ba2-8b73-cfa37b2b14a4-workload-certs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:19:56.505484 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:56.505460 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26356b33-b847-4ba2-8b73-cfa37b2b14a4-credential-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:19:56.505484 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:56.505470 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26356b33-b847-4ba2-8b73-cfa37b2b14a4-istio-podinfo\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:19:57.110369 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:57.110331 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk" Oct 30 17:19:57.110827 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:57.110339 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk" event={"ID":"26356b33-b847-4ba2-8b73-cfa37b2b14a4","Type":"ContainerDied","Data":"fda5147535928243b6065999b9b49a723c0d1923df82dfa3107fbfb7af0c36f0"} Oct 30 17:19:57.110827 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:57.110440 4493 scope.go:122] "RemoveContainer" containerID="46e37164664f969bb2947cf53a46c36d20379cd52502957c2a5c3ad4ac767fe4" Oct 30 17:19:57.176924 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:57.176888 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk"] Oct 30 17:19:57.188021 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:57.187994 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-rvcmk"] Oct 30 17:19:57.660248 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:57.660218 4493 scope.go:122] "RemoveContainer" containerID="61557f4679b0e265b57f611a179bd705cc85b3baa5904e363f88b609179820f9" Oct 30 17:19:57.671503 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:57.671481 4493 scope.go:122] "RemoveContainer" containerID="b82046b32be1ba0aff8189faba1a0520f8aa3c793af33125a4e621e370149644" Oct 30 17:19:57.685685 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:57.685664 4493 scope.go:122] "RemoveContainer" containerID="cd6f261f02ee181f14b0484da034be39ac5174909da70cc4bcc581c547c93979" Oct 30 17:19:57.695604 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:57.695577 4493 scope.go:122] "RemoveContainer" containerID="671265411c99153246976c6497e300737c6ae9d34dc235183f0964a5c971b61b" Oct 30 17:19:57.705378 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:57.705355 4493 scope.go:122] "RemoveContainer" containerID="b7686aa2eecd9fbf06ebb07c5c620d95347aeeb4b1576053d78546518d9a6cbb" Oct 30 17:19:57.715872 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:57.715851 4493 scope.go:122] "RemoveContainer" containerID="85f85e13ee8b9bcaa6953c50189126b749b1fa31ec160862c8103c167d6263cc" Oct 30 17:19:58.936286 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:19:58.936251 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26356b33-b847-4ba2-8b73-cfa37b2b14a4" path="/var/lib/kubelet/pods/26356b33-b847-4ba2-8b73-cfa37b2b14a4/volumes" Oct 30 17:20:00.346222 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:20:00.346185 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 17:20:07.652115 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:07.652073 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6acc78-zbdt2"] Oct 30 17:20:07.658965 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:07.658932 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6acc78-zbdt2"] Oct 30 17:20:08.936532 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:08.936495 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="172ba705-a53b-4b3b-b91e-28ef5d55075a" path="/var/lib/kubelet/pods/172ba705-a53b-4b3b-b91e-28ef5d55075a/volumes" Oct 30 17:20:10.137810 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:10.136902 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 17:20:14.755175 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:14.755123 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg"] Oct 30 17:20:14.756418 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:14.756385 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg" podUID="c5146890-4f44-4e48-9147-dc215b8195d9" containerName="istio-proxy" containerID="cri-o://f552982c48d33c757781f7056c3edbdf024c507caac153eda2ce0f0e5a06df51" gracePeriod=30 Oct 30 17:20:15.375758 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:15.372353 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24acb2-w8svg"] Oct 30 17:20:15.375758 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:15.375662 4493 cpu_manager.go:414] "containerMap: removing container" podUID="cd1ff57b-1a55-4ef7-95f6-ee9389a00d99" containerName="main" Oct 30 17:20:15.375758 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:15.375682 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd1ff57b-1a55-4ef7-95f6-ee9389a00d99" containerName="main" Oct 30 17:20:15.375758 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:15.375714 4493 cpu_manager.go:414] "containerMap: removing container" podUID="98c9aa43-9e50-4612-9965-4a20f9458deb" containerName="istio-proxy" Oct 30 17:20:15.375758 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:15.375722 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98c9aa43-9e50-4612-9965-4a20f9458deb" containerName="istio-proxy" Oct 30 17:20:15.375758 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:15.375746 4493 cpu_manager.go:414] "containerMap: removing container" podUID="172ba705-a53b-4b3b-b91e-28ef5d55075a" containerName="revoke-keys" Oct 30 17:20:15.375758 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:15.375753 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="172ba705-a53b-4b3b-b91e-28ef5d55075a" containerName="revoke-keys" Oct 30 17:20:15.376365 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:15.375810 4493 cpu_manager.go:414] "containerMap: removing container" podUID="26356b33-b847-4ba2-8b73-cfa37b2b14a4" containerName="istio-proxy" Oct 30 17:20:15.376365 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:15.375819 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26356b33-b847-4ba2-8b73-cfa37b2b14a4" containerName="istio-proxy" Oct 30 17:20:15.376365 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:15.376016 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd1ff57b-1a55-4ef7-95f6-ee9389a00d99" containerName="main" Oct 30 17:20:15.376365 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:15.376036 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="98c9aa43-9e50-4612-9965-4a20f9458deb" containerName="istio-proxy" Oct 30 17:20:15.376365 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:15.376049 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="172ba705-a53b-4b3b-b91e-28ef5d55075a" containerName="revoke-keys" Oct 30 17:20:15.376365 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:15.376066 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="26356b33-b847-4ba2-8b73-cfa37b2b14a4" containerName="istio-proxy" Oct 30 17:20:15.382057 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:15.382026 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24acb2-w8svg" Oct 30 17:20:15.399924 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:15.399900 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:20:15.400188 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:15.400169 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:20:15.400421 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:15.400388 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24acb2-w8svg"] Oct 30 17:20:15.434811 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:15.426151 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 17:20:15.451809 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:15.442950 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:20:15.530236 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:15.516335 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfbzh\" (UniqueName: \"kubernetes.io/projected/866c22fc-a2d3-47db-bac5-8ff5eb84cd93-kube-api-access-tfbzh\") pod \"maas-api-revoke-keys-e2e-inf-b-24acb2-w8svg\" (UID: \"866c22fc-a2d3-47db-bac5-8ff5eb84cd93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24acb2-w8svg" Oct 30 17:20:15.530236 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:15.516462 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/866c22fc-a2d3-47db-bac5-8ff5eb84cd93-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-24acb2-w8svg\" (UID: \"866c22fc-a2d3-47db-bac5-8ff5eb84cd93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24acb2-w8svg" Oct 30 17:20:15.530236 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:15.516577 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/866c22fc-a2d3-47db-bac5-8ff5eb84cd93-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-24acb2-w8svg\" (UID: \"866c22fc-a2d3-47db-bac5-8ff5eb84cd93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24acb2-w8svg" Oct 30 17:20:15.630016 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:15.629918 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfbzh\" (UniqueName: \"kubernetes.io/projected/866c22fc-a2d3-47db-bac5-8ff5eb84cd93-kube-api-access-tfbzh\") pod \"maas-api-revoke-keys-e2e-inf-b-24acb2-w8svg\" (UID: \"866c22fc-a2d3-47db-bac5-8ff5eb84cd93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24acb2-w8svg" Oct 30 17:20:15.635108 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:15.635075 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/866c22fc-a2d3-47db-bac5-8ff5eb84cd93-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-24acb2-w8svg\" (UID: \"866c22fc-a2d3-47db-bac5-8ff5eb84cd93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24acb2-w8svg" Oct 30 17:20:15.635332 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:15.635310 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/866c22fc-a2d3-47db-bac5-8ff5eb84cd93-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-24acb2-w8svg\" (UID: \"866c22fc-a2d3-47db-bac5-8ff5eb84cd93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24acb2-w8svg" Oct 30 17:20:15.638286 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:15.638261 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/866c22fc-a2d3-47db-bac5-8ff5eb84cd93-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-24acb2-w8svg\" (UID: \"866c22fc-a2d3-47db-bac5-8ff5eb84cd93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24acb2-w8svg" Oct 30 17:20:15.642517 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:15.642495 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/866c22fc-a2d3-47db-bac5-8ff5eb84cd93-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-24acb2-w8svg\" (UID: \"866c22fc-a2d3-47db-bac5-8ff5eb84cd93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24acb2-w8svg" Oct 30 17:20:15.659793 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:15.659755 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfbzh\" (UniqueName: \"kubernetes.io/projected/866c22fc-a2d3-47db-bac5-8ff5eb84cd93-kube-api-access-tfbzh\") pod \"maas-api-revoke-keys-e2e-inf-b-24acb2-w8svg\" (UID: \"866c22fc-a2d3-47db-bac5-8ff5eb84cd93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24acb2-w8svg" Oct 30 17:20:15.701314 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:15.701295 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24acb2-w8svg" Oct 30 17:20:16.042874 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:16.042841 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24acb2-w8svg"] Oct 30 17:20:16.047689 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:20:16.047659 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod866c22fc_a2d3_47db_bac5_8ff5eb84cd93.slice/crio-c2a7b8e22bee0705ffbcf5b8e9590d48b0cdd498a406e00acc9d23cc52846225 WatchSource:0}: Error finding container c2a7b8e22bee0705ffbcf5b8e9590d48b0cdd498a406e00acc9d23cc52846225: Status 404 returned error can't find the container with id c2a7b8e22bee0705ffbcf5b8e9590d48b0cdd498a406e00acc9d23cc52846225 Oct 30 17:20:16.862085 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:16.861990 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24acb2-w8svg" event={"ID":"866c22fc-a2d3-47db-bac5-8ff5eb84cd93","Type":"ContainerStarted","Data":"5ea80a0432b9205a6792942a4cf028de036e131eb9095b515c22ec5a1dd58111"} Oct 30 17:20:16.862085 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:16.862060 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24acb2-w8svg" event={"ID":"866c22fc-a2d3-47db-bac5-8ff5eb84cd93","Type":"ContainerStarted","Data":"c2a7b8e22bee0705ffbcf5b8e9590d48b0cdd498a406e00acc9d23cc52846225"} Oct 30 17:20:16.902760 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:16.902701 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24acb2-w8svg" podStartSLOduration=1.9026833509999999 podStartE2EDuration="1.902683351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:20:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:20:16.901255742 +0000 UTC m=+3444.128430841" watchObservedRunningTime="2026-10-30 17:20:16.902683351 +0000 UTC m=+3444.129858453" Oct 30 17:20:17.873200 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:17.873158 4493 generic.go:356] "Generic (PLEG): container finished" podID="866c22fc-a2d3-47db-bac5-8ff5eb84cd93" containerID="5ea80a0432b9205a6792942a4cf028de036e131eb9095b515c22ec5a1dd58111" exitCode=0 Oct 30 17:20:17.873709 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:17.873236 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24acb2-w8svg" event={"ID":"866c22fc-a2d3-47db-bac5-8ff5eb84cd93","Type":"ContainerDied","Data":"5ea80a0432b9205a6792942a4cf028de036e131eb9095b515c22ec5a1dd58111"} Oct 30 17:20:19.091682 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:19.091650 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24acb2-w8svg" Oct 30 17:20:19.216452 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:19.216412 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/866c22fc-a2d3-47db-bac5-8ff5eb84cd93-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/866c22fc-a2d3-47db-bac5-8ff5eb84cd93-maas-api-xks-ca\") pod \"866c22fc-a2d3-47db-bac5-8ff5eb84cd93\" (UID: \"866c22fc-a2d3-47db-bac5-8ff5eb84cd93\") " Oct 30 17:20:19.216682 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:19.216592 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/866c22fc-a2d3-47db-bac5-8ff5eb84cd93-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/866c22fc-a2d3-47db-bac5-8ff5eb84cd93-maas-api-openshift-ca\") pod \"866c22fc-a2d3-47db-bac5-8ff5eb84cd93\" (UID: \"866c22fc-a2d3-47db-bac5-8ff5eb84cd93\") " Oct 30 17:20:19.216682 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:19.216656 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/866c22fc-a2d3-47db-bac5-8ff5eb84cd93-kube-api-access-tfbzh\" (UniqueName: \"kubernetes.io/projected/866c22fc-a2d3-47db-bac5-8ff5eb84cd93-kube-api-access-tfbzh\") pod \"866c22fc-a2d3-47db-bac5-8ff5eb84cd93\" (UID: \"866c22fc-a2d3-47db-bac5-8ff5eb84cd93\") " Oct 30 17:20:19.217284 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:19.217180 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/866c22fc-a2d3-47db-bac5-8ff5eb84cd93-maas-api-openshift-ca" pod "866c22fc-a2d3-47db-bac5-8ff5eb84cd93" (UID: "866c22fc-a2d3-47db-bac5-8ff5eb84cd93"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:20:19.217629 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:19.217602 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/866c22fc-a2d3-47db-bac5-8ff5eb84cd93-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:20:19.219480 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:19.219450 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/866c22fc-a2d3-47db-bac5-8ff5eb84cd93-maas-api-xks-ca" pod "866c22fc-a2d3-47db-bac5-8ff5eb84cd93" (UID: "866c22fc-a2d3-47db-bac5-8ff5eb84cd93"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:20:19.219741 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:19.219711 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/866c22fc-a2d3-47db-bac5-8ff5eb84cd93-kube-api-access-tfbzh" pod "866c22fc-a2d3-47db-bac5-8ff5eb84cd93" (UID: "866c22fc-a2d3-47db-bac5-8ff5eb84cd93"). InnerVolumeSpecName "kube-api-access-tfbzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:20:19.319802 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:19.319743 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfbzh\" (UniqueName: \"kubernetes.io/projected/866c22fc-a2d3-47db-bac5-8ff5eb84cd93-kube-api-access-tfbzh\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:20:19.319802 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:19.319803 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/866c22fc-a2d3-47db-bac5-8ff5eb84cd93-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:20:19.892757 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:19.892723 4493 generic.go:356] "Generic (PLEG): container finished" podID="c5146890-4f44-4e48-9147-dc215b8195d9" containerID="f552982c48d33c757781f7056c3edbdf024c507caac153eda2ce0f0e5a06df51" exitCode=0 Oct 30 17:20:19.892937 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:19.892807 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg" event={"ID":"c5146890-4f44-4e48-9147-dc215b8195d9","Type":"ContainerDied","Data":"f552982c48d33c757781f7056c3edbdf024c507caac153eda2ce0f0e5a06df51"} Oct 30 17:20:19.892937 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:19.892870 4493 scope.go:122] "RemoveContainer" containerID="f3d935c69434a834596a8b60e289ddca19d0a6132e8ffc572e33f8952aa1ae38" Oct 30 17:20:19.894399 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:19.894382 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24acb2-w8svg" Oct 30 17:20:19.894518 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:19.894416 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24acb2-w8svg" event={"ID":"866c22fc-a2d3-47db-bac5-8ff5eb84cd93","Type":"ContainerDied","Data":"c2a7b8e22bee0705ffbcf5b8e9590d48b0cdd498a406e00acc9d23cc52846225"} Oct 30 17:20:19.894518 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:19.894459 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2a7b8e22bee0705ffbcf5b8e9590d48b0cdd498a406e00acc9d23cc52846225" Oct 30 17:20:20.181187 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:20.181158 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg" Oct 30 17:20:20.246500 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:20.246466 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c5146890-4f44-4e48-9147-dc215b8195d9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5146890-4f44-4e48-9147-dc215b8195d9-istio-podinfo\") pod \"c5146890-4f44-4e48-9147-dc215b8195d9\" (UID: \"c5146890-4f44-4e48-9147-dc215b8195d9\") " Oct 30 17:20:20.246711 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:20.246542 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5146890-4f44-4e48-9147-dc215b8195d9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5146890-4f44-4e48-9147-dc215b8195d9-istio-data\") pod \"c5146890-4f44-4e48-9147-dc215b8195d9\" (UID: \"c5146890-4f44-4e48-9147-dc215b8195d9\") " Oct 30 17:20:20.246711 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:20.246584 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5146890-4f44-4e48-9147-dc215b8195d9-kube-api-access-jdznp\" (UniqueName: \"kubernetes.io/projected/c5146890-4f44-4e48-9147-dc215b8195d9-kube-api-access-jdznp\") pod \"c5146890-4f44-4e48-9147-dc215b8195d9\" (UID: \"c5146890-4f44-4e48-9147-dc215b8195d9\") " Oct 30 17:20:20.246711 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:20.246623 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5146890-4f44-4e48-9147-dc215b8195d9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5146890-4f44-4e48-9147-dc215b8195d9-workload-socket\") pod \"c5146890-4f44-4e48-9147-dc215b8195d9\" (UID: \"c5146890-4f44-4e48-9147-dc215b8195d9\") " Oct 30 17:20:20.246711 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:20.246659 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5146890-4f44-4e48-9147-dc215b8195d9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5146890-4f44-4e48-9147-dc215b8195d9-workload-certs\") pod \"c5146890-4f44-4e48-9147-dc215b8195d9\" (UID: \"c5146890-4f44-4e48-9147-dc215b8195d9\") " Oct 30 17:20:20.247020 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:20.246761 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5146890-4f44-4e48-9147-dc215b8195d9-istio-token\" (UniqueName: \"kubernetes.io/projected/c5146890-4f44-4e48-9147-dc215b8195d9-istio-token\") pod \"c5146890-4f44-4e48-9147-dc215b8195d9\" (UID: \"c5146890-4f44-4e48-9147-dc215b8195d9\") " Oct 30 17:20:20.247020 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:20.246928 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5146890-4f44-4e48-9147-dc215b8195d9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5146890-4f44-4e48-9147-dc215b8195d9-istio-envoy\") pod \"c5146890-4f44-4e48-9147-dc215b8195d9\" (UID: \"c5146890-4f44-4e48-9147-dc215b8195d9\") " Oct 30 17:20:20.247020 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:20.247001 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5146890-4f44-4e48-9147-dc215b8195d9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5146890-4f44-4e48-9147-dc215b8195d9-istiod-ca-cert\") pod \"c5146890-4f44-4e48-9147-dc215b8195d9\" (UID: \"c5146890-4f44-4e48-9147-dc215b8195d9\") " Oct 30 17:20:20.247197 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:20.247073 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5146890-4f44-4e48-9147-dc215b8195d9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5146890-4f44-4e48-9147-dc215b8195d9-credential-socket\") pod \"c5146890-4f44-4e48-9147-dc215b8195d9\" (UID: \"c5146890-4f44-4e48-9147-dc215b8195d9\") " Oct 30 17:20:20.247320 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:20.247296 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5146890-4f44-4e48-9147-dc215b8195d9-workload-certs" pod "c5146890-4f44-4e48-9147-dc215b8195d9" (UID: "c5146890-4f44-4e48-9147-dc215b8195d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:20:20.247550 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:20.247404 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5146890-4f44-4e48-9147-dc215b8195d9-istio-data" pod "c5146890-4f44-4e48-9147-dc215b8195d9" (UID: "c5146890-4f44-4e48-9147-dc215b8195d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:20:20.247739 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:20.247709 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5146890-4f44-4e48-9147-dc215b8195d9-credential-socket" pod "c5146890-4f44-4e48-9147-dc215b8195d9" (UID: "c5146890-4f44-4e48-9147-dc215b8195d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:20:20.248014 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:20.247990 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5146890-4f44-4e48-9147-dc215b8195d9-istiod-ca-cert" pod "c5146890-4f44-4e48-9147-dc215b8195d9" (UID: "c5146890-4f44-4e48-9147-dc215b8195d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:20:20.248652 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:20.248626 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5146890-4f44-4e48-9147-dc215b8195d9-istiod-ca-cert\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:20:20.248751 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:20.248657 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5146890-4f44-4e48-9147-dc215b8195d9-credential-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:20:20.248751 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:20.248673 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5146890-4f44-4e48-9147-dc215b8195d9-istio-data\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:20:20.248751 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:20.248687 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5146890-4f44-4e48-9147-dc215b8195d9-workload-certs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:20:20.248954 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:20.248937 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5146890-4f44-4e48-9147-dc215b8195d9-workload-socket" pod "c5146890-4f44-4e48-9147-dc215b8195d9" (UID: "c5146890-4f44-4e48-9147-dc215b8195d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:20:20.249908 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:20.249874 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5146890-4f44-4e48-9147-dc215b8195d9-istio-token" pod "c5146890-4f44-4e48-9147-dc215b8195d9" (UID: "c5146890-4f44-4e48-9147-dc215b8195d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:20:20.251031 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:20.251006 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5146890-4f44-4e48-9147-dc215b8195d9-istio-envoy" pod "c5146890-4f44-4e48-9147-dc215b8195d9" (UID: "c5146890-4f44-4e48-9147-dc215b8195d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:20:20.251251 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:20.251216 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5146890-4f44-4e48-9147-dc215b8195d9-istio-podinfo" pod "c5146890-4f44-4e48-9147-dc215b8195d9" (UID: "c5146890-4f44-4e48-9147-dc215b8195d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:20:20.251705 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:20.251660 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5146890-4f44-4e48-9147-dc215b8195d9-kube-api-access-jdznp" pod "c5146890-4f44-4e48-9147-dc215b8195d9" (UID: "c5146890-4f44-4e48-9147-dc215b8195d9"). InnerVolumeSpecName "kube-api-access-jdznp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:20:20.350569 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:20.350529 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5146890-4f44-4e48-9147-dc215b8195d9-istio-token\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:20:20.350569 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:20.350558 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5146890-4f44-4e48-9147-dc215b8195d9-istio-envoy\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:20:20.350569 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:20.350568 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5146890-4f44-4e48-9147-dc215b8195d9-istio-podinfo\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:20:20.350569 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:20.350578 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdznp\" (UniqueName: \"kubernetes.io/projected/c5146890-4f44-4e48-9147-dc215b8195d9-kube-api-access-jdznp\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:20:20.350914 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:20.350590 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5146890-4f44-4e48-9147-dc215b8195d9-workload-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:20:20.904491 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:20.904453 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg" event={"ID":"c5146890-4f44-4e48-9147-dc215b8195d9","Type":"ContainerDied","Data":"19a1bbcda04783c7159eb2a3bea75cd2602c8d274034d3507fc04f0d760ebe0b"} Oct 30 17:20:20.904677 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:20.904528 4493 scope.go:122] "RemoveContainer" containerID="f552982c48d33c757781f7056c3edbdf024c507caac153eda2ce0f0e5a06df51" Oct 30 17:20:20.904677 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:20.904462 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg" Oct 30 17:20:20.972654 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:20.972573 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg"] Oct 30 17:20:20.982346 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:20.982315 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-6acc78-openshift-default-6c4ccdc59f-jkkmg"] Oct 30 17:20:21.760923 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:21.760892 4493 patch_prober.go:29] interesting pod/e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:20:21.761351 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:21.760942 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4" podUID="a18c02c8-1a51-4dcf-aeef-696acb2d2280" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:20:22.935922 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:22.935887 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5146890-4f44-4e48-9147-dc215b8195d9" path="/var/lib/kubelet/pods/c5146890-4f44-4e48-9147-dc215b8195d9/volumes" Oct 30 17:20:28.039019 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:28.038969 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67607d5a-hnkqk"] Oct 30 17:20:28.050616 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:28.050591 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67607d5a-hnkqk"] Oct 30 17:20:28.938223 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:28.938186 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05b43e44-eef6-4fd8-99d3-8ffebc0c7370" path="/var/lib/kubelet/pods/05b43e44-eef6-4fd8-99d3-8ffebc0c7370/volumes" Oct 30 17:20:30.351009 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:20:30.350970 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 17:20:32.094582 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:32.094556 4493 generic.go:356] "Generic (PLEG): container finished" podID="a18c02c8-1a51-4dcf-aeef-696acb2d2280" containerID="49cc2b2335dd4d05149323f545e7f7c0c490009af8f91c804dfc19053918d7fb" exitCode=0 Oct 30 17:20:32.094965 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:32.094622 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4" event={"ID":"a18c02c8-1a51-4dcf-aeef-696acb2d2280","Type":"ContainerDied","Data":"49cc2b2335dd4d05149323f545e7f7c0c490009af8f91c804dfc19053918d7fb"} Oct 30 17:20:32.095080 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:32.095063 4493 scope.go:122] "RemoveContainer" containerID="49cc2b2335dd4d05149323f545e7f7c0c490009af8f91c804dfc19053918d7fb" Oct 30 17:20:32.097207 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:32.097172 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:20:32.097295 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:32.097249 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:20:32.097295 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:32.097290 4493 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892016Ki","pods":"250"} Oct 30 17:20:32.940521 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:32.940488 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-d80ebb/test-model-d80ebb-kserve-6f594b78b5-pg9wm"] Oct 30 17:20:32.941492 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:32.941468 4493 cpu_manager.go:414] "containerMap: removing container" podUID="c5146890-4f44-4e48-9147-dc215b8195d9" containerName="istio-proxy" Oct 30 17:20:32.941591 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:32.941494 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5146890-4f44-4e48-9147-dc215b8195d9" containerName="istio-proxy" Oct 30 17:20:32.941591 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:32.941521 4493 cpu_manager.go:414] "containerMap: removing container" podUID="c5146890-4f44-4e48-9147-dc215b8195d9" containerName="istio-proxy" Oct 30 17:20:32.941591 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:32.941529 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5146890-4f44-4e48-9147-dc215b8195d9" containerName="istio-proxy" Oct 30 17:20:32.941591 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:32.941564 4493 cpu_manager.go:414] "containerMap: removing container" podUID="866c22fc-a2d3-47db-bac5-8ff5eb84cd93" containerName="revoke-keys" Oct 30 17:20:32.941591 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:32.941572 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="866c22fc-a2d3-47db-bac5-8ff5eb84cd93" containerName="revoke-keys" Oct 30 17:20:32.941827 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:32.941797 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5146890-4f44-4e48-9147-dc215b8195d9" containerName="istio-proxy" Oct 30 17:20:32.941827 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:32.941816 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5146890-4f44-4e48-9147-dc215b8195d9" containerName="istio-proxy" Oct 30 17:20:32.941910 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:32.941830 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="866c22fc-a2d3-47db-bac5-8ff5eb84cd93" containerName="revoke-keys" Oct 30 17:20:32.944929 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:32.944909 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d80ebb/test-model-d80ebb-kserve-6f594b78b5-pg9wm" Oct 30 17:20:32.960681 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:32.960535 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-d80ebb\"/\"openshift-service-ca.crt\"" Oct 30 17:20:32.966962 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:32.966931 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-d80ebb/test-model-d80ebb-kserve-6f594b78b5-pg9wm"] Oct 30 17:20:32.971870 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:32.971837 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-d80ebb\"/\"default-dockercfg-wvxvs\"" Oct 30 17:20:32.972269 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:32.972240 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-d80ebb\"/\"test-model-d80ebb-kserve-self-signed-certs\"" Oct 30 17:20:32.972426 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:32.972370 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-d80ebb\"/\"kube-root-ca.crt\"" Oct 30 17:20:33.102518 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:33.102483 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/442ab715-fe12-4e4c-8ca3-c452a3739cbe-model-cache\") pod \"test-model-d80ebb-kserve-6f594b78b5-pg9wm\" (UID: \"442ab715-fe12-4e4c-8ca3-c452a3739cbe\") " pod="ai-tenant-e2e-inf-a-d80ebb/test-model-d80ebb-kserve-6f594b78b5-pg9wm" Oct 30 17:20:33.103037 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:33.102642 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/442ab715-fe12-4e4c-8ca3-c452a3739cbe-tls-certs\") pod \"test-model-d80ebb-kserve-6f594b78b5-pg9wm\" (UID: \"442ab715-fe12-4e4c-8ca3-c452a3739cbe\") " pod="ai-tenant-e2e-inf-a-d80ebb/test-model-d80ebb-kserve-6f594b78b5-pg9wm" Oct 30 17:20:33.103037 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:33.102696 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/442ab715-fe12-4e4c-8ca3-c452a3739cbe-tmp-dir\") pod \"test-model-d80ebb-kserve-6f594b78b5-pg9wm\" (UID: \"442ab715-fe12-4e4c-8ca3-c452a3739cbe\") " pod="ai-tenant-e2e-inf-a-d80ebb/test-model-d80ebb-kserve-6f594b78b5-pg9wm" Oct 30 17:20:33.103037 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:33.102944 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/442ab715-fe12-4e4c-8ca3-c452a3739cbe-home\") pod \"test-model-d80ebb-kserve-6f594b78b5-pg9wm\" (UID: \"442ab715-fe12-4e4c-8ca3-c452a3739cbe\") " pod="ai-tenant-e2e-inf-a-d80ebb/test-model-d80ebb-kserve-6f594b78b5-pg9wm" Oct 30 17:20:33.103244 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:33.103145 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4wj7\" (UniqueName: \"kubernetes.io/projected/442ab715-fe12-4e4c-8ca3-c452a3739cbe-kube-api-access-q4wj7\") pod \"test-model-d80ebb-kserve-6f594b78b5-pg9wm\" (UID: \"442ab715-fe12-4e4c-8ca3-c452a3739cbe\") " pod="ai-tenant-e2e-inf-a-d80ebb/test-model-d80ebb-kserve-6f594b78b5-pg9wm" Oct 30 17:20:33.103244 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:33.103185 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/442ab715-fe12-4e4c-8ca3-c452a3739cbe-dshm\") pod \"test-model-d80ebb-kserve-6f594b78b5-pg9wm\" (UID: \"442ab715-fe12-4e4c-8ca3-c452a3739cbe\") " pod="ai-tenant-e2e-inf-a-d80ebb/test-model-d80ebb-kserve-6f594b78b5-pg9wm" Oct 30 17:20:33.135698 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:33.129085 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4" event={"ID":"a18c02c8-1a51-4dcf-aeef-696acb2d2280","Type":"ContainerStarted","Data":"0a51574a70b4dc00ccd73e8dccdb4d1ad094ccbedaae44605fb8ad39482e97e1"} Oct 30 17:20:33.247861 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:33.243732 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/442ab715-fe12-4e4c-8ca3-c452a3739cbe-model-cache\") pod \"test-model-d80ebb-kserve-6f594b78b5-pg9wm\" (UID: \"442ab715-fe12-4e4c-8ca3-c452a3739cbe\") " pod="ai-tenant-e2e-inf-a-d80ebb/test-model-d80ebb-kserve-6f594b78b5-pg9wm" Oct 30 17:20:33.251804 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:33.248416 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/442ab715-fe12-4e4c-8ca3-c452a3739cbe-tls-certs\") pod \"test-model-d80ebb-kserve-6f594b78b5-pg9wm\" (UID: \"442ab715-fe12-4e4c-8ca3-c452a3739cbe\") " pod="ai-tenant-e2e-inf-a-d80ebb/test-model-d80ebb-kserve-6f594b78b5-pg9wm" Oct 30 17:20:33.251804 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:33.248520 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/442ab715-fe12-4e4c-8ca3-c452a3739cbe-tmp-dir\") pod \"test-model-d80ebb-kserve-6f594b78b5-pg9wm\" (UID: \"442ab715-fe12-4e4c-8ca3-c452a3739cbe\") " pod="ai-tenant-e2e-inf-a-d80ebb/test-model-d80ebb-kserve-6f594b78b5-pg9wm" Oct 30 17:20:33.251804 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:33.248653 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/442ab715-fe12-4e4c-8ca3-c452a3739cbe-home\") pod \"test-model-d80ebb-kserve-6f594b78b5-pg9wm\" (UID: \"442ab715-fe12-4e4c-8ca3-c452a3739cbe\") " pod="ai-tenant-e2e-inf-a-d80ebb/test-model-d80ebb-kserve-6f594b78b5-pg9wm" Oct 30 17:20:33.251804 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:33.248872 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4wj7\" (UniqueName: \"kubernetes.io/projected/442ab715-fe12-4e4c-8ca3-c452a3739cbe-kube-api-access-q4wj7\") pod \"test-model-d80ebb-kserve-6f594b78b5-pg9wm\" (UID: \"442ab715-fe12-4e4c-8ca3-c452a3739cbe\") " pod="ai-tenant-e2e-inf-a-d80ebb/test-model-d80ebb-kserve-6f594b78b5-pg9wm" Oct 30 17:20:33.251804 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:33.248925 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/442ab715-fe12-4e4c-8ca3-c452a3739cbe-dshm\") pod \"test-model-d80ebb-kserve-6f594b78b5-pg9wm\" (UID: \"442ab715-fe12-4e4c-8ca3-c452a3739cbe\") " pod="ai-tenant-e2e-inf-a-d80ebb/test-model-d80ebb-kserve-6f594b78b5-pg9wm" Oct 30 17:20:33.269810 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:33.261665 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/442ab715-fe12-4e4c-8ca3-c452a3739cbe-model-cache\") pod \"test-model-d80ebb-kserve-6f594b78b5-pg9wm\" (UID: \"442ab715-fe12-4e4c-8ca3-c452a3739cbe\") " pod="ai-tenant-e2e-inf-a-d80ebb/test-model-d80ebb-kserve-6f594b78b5-pg9wm" Oct 30 17:20:33.269810 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:33.262727 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/442ab715-fe12-4e4c-8ca3-c452a3739cbe-home\") pod \"test-model-d80ebb-kserve-6f594b78b5-pg9wm\" (UID: \"442ab715-fe12-4e4c-8ca3-c452a3739cbe\") " pod="ai-tenant-e2e-inf-a-d80ebb/test-model-d80ebb-kserve-6f594b78b5-pg9wm" Oct 30 17:20:33.269810 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:33.262997 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/442ab715-fe12-4e4c-8ca3-c452a3739cbe-tmp-dir\") pod \"test-model-d80ebb-kserve-6f594b78b5-pg9wm\" (UID: \"442ab715-fe12-4e4c-8ca3-c452a3739cbe\") " pod="ai-tenant-e2e-inf-a-d80ebb/test-model-d80ebb-kserve-6f594b78b5-pg9wm" Oct 30 17:20:33.303011 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:33.302925 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4" Oct 30 17:20:33.330807 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:33.308109 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4wj7\" (UniqueName: \"kubernetes.io/projected/442ab715-fe12-4e4c-8ca3-c452a3739cbe-kube-api-access-q4wj7\") pod \"test-model-d80ebb-kserve-6f594b78b5-pg9wm\" (UID: \"442ab715-fe12-4e4c-8ca3-c452a3739cbe\") " pod="ai-tenant-e2e-inf-a-d80ebb/test-model-d80ebb-kserve-6f594b78b5-pg9wm" Oct 30 17:20:33.331477 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:33.331136 4493 patch_prober.go:29] interesting pod/e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused" start-of-body= Oct 30 17:20:33.331477 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:33.331194 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4" podUID="a18c02c8-1a51-4dcf-aeef-696acb2d2280" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused" Oct 30 17:20:33.335667 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:33.335642 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/442ab715-fe12-4e4c-8ca3-c452a3739cbe-dshm\") pod \"test-model-d80ebb-kserve-6f594b78b5-pg9wm\" (UID: \"442ab715-fe12-4e4c-8ca3-c452a3739cbe\") " pod="ai-tenant-e2e-inf-a-d80ebb/test-model-d80ebb-kserve-6f594b78b5-pg9wm" Oct 30 17:20:33.353810 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:33.342439 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/442ab715-fe12-4e4c-8ca3-c452a3739cbe-tls-certs\") pod \"test-model-d80ebb-kserve-6f594b78b5-pg9wm\" (UID: \"442ab715-fe12-4e4c-8ca3-c452a3739cbe\") " pod="ai-tenant-e2e-inf-a-d80ebb/test-model-d80ebb-kserve-6f594b78b5-pg9wm" Oct 30 17:20:33.601178 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:33.601078 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d80ebb/test-model-d80ebb-kserve-6f594b78b5-pg9wm" Oct 30 17:20:34.041811 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:34.024663 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-d80ebb/test-model-d80ebb-kserve-6f594b78b5-pg9wm"] Oct 30 17:20:34.168966 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:34.168939 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d80ebb/test-model-d80ebb-kserve-6f594b78b5-pg9wm" event={"ID":"442ab715-fe12-4e4c-8ca3-c452a3739cbe","Type":"ContainerStarted","Data":"9ac5a9d3e9b4bf48d92f58c9c06dba4b4579b10e951237f6c8ebd8f3df0345c4"} Oct 30 17:20:34.302980 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:34.302907 4493 patch_prober.go:29] interesting pod/e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused" start-of-body= Oct 30 17:20:34.302980 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:34.302960 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4" podUID="a18c02c8-1a51-4dcf-aeef-696acb2d2280" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused" Oct 30 17:20:34.601453 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:34.601376 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-d80ebb/test-model-d80ebb-kserve-6f594b78b5-pg9wm" Oct 30 17:20:35.177700 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:35.177657 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d80ebb/test-model-d80ebb-kserve-6f594b78b5-pg9wm" event={"ID":"442ab715-fe12-4e4c-8ca3-c452a3739cbe","Type":"ContainerStarted","Data":"2ca03535acc16d82933a03da51cecd8827b2874ccd997ef62583b20e76161597"} Oct 30 17:20:35.302771 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:35.302734 4493 patch_prober.go:29] interesting pod/e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused" start-of-body= Oct 30 17:20:35.303013 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:35.302810 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4" podUID="a18c02c8-1a51-4dcf-aeef-696acb2d2280" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused" Oct 30 17:20:35.730809 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:35.719872 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4" Oct 30 17:20:36.307811 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:36.304280 4493 patch_prober.go:29] interesting pod/e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused" start-of-body= Oct 30 17:20:36.307811 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:36.304334 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4" podUID="a18c02c8-1a51-4dcf-aeef-696acb2d2280" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused" Oct 30 17:20:37.303574 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:37.303539 4493 patch_prober.go:29] interesting pod/e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused" start-of-body= Oct 30 17:20:37.303761 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:37.303601 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4" podUID="a18c02c8-1a51-4dcf-aeef-696acb2d2280" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused" Oct 30 17:20:38.303504 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:38.303471 4493 patch_prober.go:29] interesting pod/e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused" start-of-body= Oct 30 17:20:38.304082 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:38.303529 4493 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4" podUID="a18c02c8-1a51-4dcf-aeef-696acb2d2280" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused" Oct 30 17:20:39.309833 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:39.309778 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4" Oct 30 17:20:39.310936 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:39.310916 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4" Oct 30 17:20:39.368493 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:39.368442 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-d80ebb/test-model-d80ebb-kserve-6f594b78b5-pg9wm" podStartSLOduration=7.368428245 podStartE2EDuration="7.368428245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:20:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:20:34.220058538 +0000 UTC m=+3461.447233633" watchObservedRunningTime="2026-10-30 17:20:39.368428245 +0000 UTC m=+3466.595603344" Oct 30 17:20:41.040006 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:41.039971 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71a240d6-npfcw"] Oct 30 17:20:41.061543 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:41.061506 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71a240d6-npfcw"] Oct 30 17:20:43.058807 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:43.043087 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b851d0f-5c42-4e0a-8dbf-0428b92f90e2" path="/var/lib/kubelet/pods/2b851d0f-5c42-4e0a-8dbf-0428b92f90e2/volumes" Oct 30 17:20:43.602204 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:43.602167 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-d80ebb/test-model-d80ebb-kserve-6f594b78b5-pg9wm" Oct 30 17:20:43.622734 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:43.622696 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-d80ebb/test-model-d80ebb-kserve-6f594b78b5-pg9wm" Oct 30 17:20:44.304911 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:44.304881 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-d80ebb/test-model-d80ebb-kserve-6f594b78b5-pg9wm" Oct 30 17:20:53.037886 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:53.037850 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7b49f59-nhvb5"] Oct 30 17:20:53.051522 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:53.051491 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7b49f59-nhvb5"] Oct 30 17:20:54.936556 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:20:54.936513 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8863e836-3d23-44ee-971c-2fbf9ca21310" path="/var/lib/kubelet/pods/8863e836-3d23-44ee-971c-2fbf9ca21310/volumes" Oct 30 17:21:00.348426 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:21:00.348359 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 17:21:00.379681 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:00.379649 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-0b09d3/test-model-0b09d3-kserve-64fc7fb899-wsjbz"] Oct 30 17:21:00.386551 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:00.386526 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0b09d3/test-model-0b09d3-kserve-64fc7fb899-wsjbz" Oct 30 17:21:00.398405 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:00.398371 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97c8ed56-7059-4f2e-b508-cc011621264c-home\") pod \"test-model-0b09d3-kserve-64fc7fb899-wsjbz\" (UID: \"97c8ed56-7059-4f2e-b508-cc011621264c\") " pod="ai-tenant-e2e-inf-b-0b09d3/test-model-0b09d3-kserve-64fc7fb899-wsjbz" Oct 30 17:21:00.398587 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:00.398442 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s9fc\" (UniqueName: \"kubernetes.io/projected/97c8ed56-7059-4f2e-b508-cc011621264c-kube-api-access-2s9fc\") pod \"test-model-0b09d3-kserve-64fc7fb899-wsjbz\" (UID: \"97c8ed56-7059-4f2e-b508-cc011621264c\") " pod="ai-tenant-e2e-inf-b-0b09d3/test-model-0b09d3-kserve-64fc7fb899-wsjbz" Oct 30 17:21:00.398587 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:00.398522 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97c8ed56-7059-4f2e-b508-cc011621264c-dshm\") pod \"test-model-0b09d3-kserve-64fc7fb899-wsjbz\" (UID: \"97c8ed56-7059-4f2e-b508-cc011621264c\") " pod="ai-tenant-e2e-inf-b-0b09d3/test-model-0b09d3-kserve-64fc7fb899-wsjbz" Oct 30 17:21:00.399021 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:00.398595 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97c8ed56-7059-4f2e-b508-cc011621264c-model-cache\") pod \"test-model-0b09d3-kserve-64fc7fb899-wsjbz\" (UID: \"97c8ed56-7059-4f2e-b508-cc011621264c\") " pod="ai-tenant-e2e-inf-b-0b09d3/test-model-0b09d3-kserve-64fc7fb899-wsjbz" Oct 30 17:21:00.399021 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:00.398671 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97c8ed56-7059-4f2e-b508-cc011621264c-tmp-dir\") pod \"test-model-0b09d3-kserve-64fc7fb899-wsjbz\" (UID: \"97c8ed56-7059-4f2e-b508-cc011621264c\") " pod="ai-tenant-e2e-inf-b-0b09d3/test-model-0b09d3-kserve-64fc7fb899-wsjbz" Oct 30 17:21:00.399021 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:00.398775 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97c8ed56-7059-4f2e-b508-cc011621264c-tls-certs\") pod \"test-model-0b09d3-kserve-64fc7fb899-wsjbz\" (UID: \"97c8ed56-7059-4f2e-b508-cc011621264c\") " pod="ai-tenant-e2e-inf-b-0b09d3/test-model-0b09d3-kserve-64fc7fb899-wsjbz" Oct 30 17:21:00.416143 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:00.415273 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-0b09d3\"/\"openshift-service-ca.crt\"" Oct 30 17:21:00.416143 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:00.415350 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-0b09d3\"/\"test-model-0b09d3-kserve-self-signed-certs\"" Oct 30 17:21:00.416143 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:00.415420 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-0b09d3\"/\"default-dockercfg-629xd\"" Oct 30 17:21:00.416143 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:00.416104 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-0b09d3\"/\"kube-root-ca.crt\"" Oct 30 17:21:00.422417 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:00.422394 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-0b09d3/test-model-0b09d3-kserve-64fc7fb899-wsjbz"] Oct 30 17:21:00.500289 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:00.500256 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97c8ed56-7059-4f2e-b508-cc011621264c-home\") pod \"test-model-0b09d3-kserve-64fc7fb899-wsjbz\" (UID: \"97c8ed56-7059-4f2e-b508-cc011621264c\") " pod="ai-tenant-e2e-inf-b-0b09d3/test-model-0b09d3-kserve-64fc7fb899-wsjbz" Oct 30 17:21:00.500508 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:00.500303 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s9fc\" (UniqueName: \"kubernetes.io/projected/97c8ed56-7059-4f2e-b508-cc011621264c-kube-api-access-2s9fc\") pod \"test-model-0b09d3-kserve-64fc7fb899-wsjbz\" (UID: \"97c8ed56-7059-4f2e-b508-cc011621264c\") " pod="ai-tenant-e2e-inf-b-0b09d3/test-model-0b09d3-kserve-64fc7fb899-wsjbz" Oct 30 17:21:00.500508 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:00.500360 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97c8ed56-7059-4f2e-b508-cc011621264c-dshm\") pod \"test-model-0b09d3-kserve-64fc7fb899-wsjbz\" (UID: \"97c8ed56-7059-4f2e-b508-cc011621264c\") " pod="ai-tenant-e2e-inf-b-0b09d3/test-model-0b09d3-kserve-64fc7fb899-wsjbz" Oct 30 17:21:00.500508 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:00.500416 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97c8ed56-7059-4f2e-b508-cc011621264c-model-cache\") pod \"test-model-0b09d3-kserve-64fc7fb899-wsjbz\" (UID: \"97c8ed56-7059-4f2e-b508-cc011621264c\") " pod="ai-tenant-e2e-inf-b-0b09d3/test-model-0b09d3-kserve-64fc7fb899-wsjbz" Oct 30 17:21:00.500508 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:00.500485 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97c8ed56-7059-4f2e-b508-cc011621264c-tmp-dir\") pod \"test-model-0b09d3-kserve-64fc7fb899-wsjbz\" (UID: \"97c8ed56-7059-4f2e-b508-cc011621264c\") " pod="ai-tenant-e2e-inf-b-0b09d3/test-model-0b09d3-kserve-64fc7fb899-wsjbz" Oct 30 17:21:00.500757 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:00.500576 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97c8ed56-7059-4f2e-b508-cc011621264c-tls-certs\") pod \"test-model-0b09d3-kserve-64fc7fb899-wsjbz\" (UID: \"97c8ed56-7059-4f2e-b508-cc011621264c\") " pod="ai-tenant-e2e-inf-b-0b09d3/test-model-0b09d3-kserve-64fc7fb899-wsjbz" Oct 30 17:21:00.500757 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:00.500620 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97c8ed56-7059-4f2e-b508-cc011621264c-home\") pod \"test-model-0b09d3-kserve-64fc7fb899-wsjbz\" (UID: \"97c8ed56-7059-4f2e-b508-cc011621264c\") " pod="ai-tenant-e2e-inf-b-0b09d3/test-model-0b09d3-kserve-64fc7fb899-wsjbz" Oct 30 17:21:00.501034 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:00.501013 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97c8ed56-7059-4f2e-b508-cc011621264c-tmp-dir\") pod \"test-model-0b09d3-kserve-64fc7fb899-wsjbz\" (UID: \"97c8ed56-7059-4f2e-b508-cc011621264c\") " pod="ai-tenant-e2e-inf-b-0b09d3/test-model-0b09d3-kserve-64fc7fb899-wsjbz" Oct 30 17:21:00.501100 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:00.501075 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97c8ed56-7059-4f2e-b508-cc011621264c-model-cache\") pod \"test-model-0b09d3-kserve-64fc7fb899-wsjbz\" (UID: \"97c8ed56-7059-4f2e-b508-cc011621264c\") " pod="ai-tenant-e2e-inf-b-0b09d3/test-model-0b09d3-kserve-64fc7fb899-wsjbz" Oct 30 17:21:00.503380 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:00.503354 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97c8ed56-7059-4f2e-b508-cc011621264c-dshm\") pod \"test-model-0b09d3-kserve-64fc7fb899-wsjbz\" (UID: \"97c8ed56-7059-4f2e-b508-cc011621264c\") " pod="ai-tenant-e2e-inf-b-0b09d3/test-model-0b09d3-kserve-64fc7fb899-wsjbz" Oct 30 17:21:00.503512 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:00.503381 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97c8ed56-7059-4f2e-b508-cc011621264c-tls-certs\") pod \"test-model-0b09d3-kserve-64fc7fb899-wsjbz\" (UID: \"97c8ed56-7059-4f2e-b508-cc011621264c\") " pod="ai-tenant-e2e-inf-b-0b09d3/test-model-0b09d3-kserve-64fc7fb899-wsjbz" Oct 30 17:21:00.527413 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:00.527387 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s9fc\" (UniqueName: \"kubernetes.io/projected/97c8ed56-7059-4f2e-b508-cc011621264c-kube-api-access-2s9fc\") pod \"test-model-0b09d3-kserve-64fc7fb899-wsjbz\" (UID: \"97c8ed56-7059-4f2e-b508-cc011621264c\") " pod="ai-tenant-e2e-inf-b-0b09d3/test-model-0b09d3-kserve-64fc7fb899-wsjbz" Oct 30 17:21:00.710514 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:00.710400 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0b09d3/test-model-0b09d3-kserve-64fc7fb899-wsjbz" Oct 30 17:21:01.278272 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:01.278243 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-0b09d3/test-model-0b09d3-kserve-64fc7fb899-wsjbz"] Oct 30 17:21:01.470807 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:01.466572 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0b09d3/test-model-0b09d3-kserve-64fc7fb899-wsjbz" event={"ID":"97c8ed56-7059-4f2e-b508-cc011621264c","Type":"ContainerStarted","Data":"c4fb1b55f948c5deca548a10ec362e7738079ff7970b9006833d7252e096803a"} Oct 30 17:21:01.470807 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:01.466628 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0b09d3/test-model-0b09d3-kserve-64fc7fb899-wsjbz" event={"ID":"97c8ed56-7059-4f2e-b508-cc011621264c","Type":"ContainerStarted","Data":"59f079b9609d36969129fb7cabbf3d2b0871180236622cd03a84de30d2c7e1c2"} Oct 30 17:21:01.534860 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:01.530679 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-0b09d3/test-model-0b09d3-kserve-64fc7fb899-wsjbz" podStartSLOduration=1.530657331 podStartE2EDuration="1.530657331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:21:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:21:01.529011855 +0000 UTC m=+3488.756186956" watchObservedRunningTime="2026-10-30 17:21:01.530657331 +0000 UTC m=+3488.757832433" Oct 30 17:21:01.713833 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:01.711420 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-0b09d3/test-model-0b09d3-kserve-64fc7fb899-wsjbz" Oct 30 17:21:07.556246 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:07.556204 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 17:21:10.710875 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:10.710837 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-0b09d3/test-model-0b09d3-kserve-64fc7fb899-wsjbz" Oct 30 17:21:10.726196 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:10.726169 4493 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-0b09d3/test-model-0b09d3-kserve-64fc7fb899-wsjbz" Oct 30 17:21:11.685465 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:11.685434 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-0b09d3/test-model-0b09d3-kserve-64fc7fb899-wsjbz" Oct 30 17:21:14.560117 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:14.560079 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx"] Oct 30 17:21:14.560641 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:14.560287 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx" podUID="1d917fe1-dd87-48a4-94ff-75ece269728f" containerName="istio-proxy" containerID="cri-o://1d8d4bce72ac31f56e7d8737dd1fe0aee76c7d4bf7318aea6eca64f90c53f9a8" gracePeriod=30 Oct 30 17:21:20.376598 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:20.375920 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24acb2-w8svg"] Oct 30 17:21:20.386587 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:20.386553 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24acb2-w8svg"] Oct 30 17:21:20.438260 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:20.438224 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx" Oct 30 17:21:20.541864 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:20.538724 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1d917fe1-dd87-48a4-94ff-75ece269728f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d917fe1-dd87-48a4-94ff-75ece269728f-istio-podinfo\") pod \"1d917fe1-dd87-48a4-94ff-75ece269728f\" (UID: \"1d917fe1-dd87-48a4-94ff-75ece269728f\") " Oct 30 17:21:20.541864 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:20.538873 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d917fe1-dd87-48a4-94ff-75ece269728f-istio-token\" (UniqueName: \"kubernetes.io/projected/1d917fe1-dd87-48a4-94ff-75ece269728f-istio-token\") pod \"1d917fe1-dd87-48a4-94ff-75ece269728f\" (UID: \"1d917fe1-dd87-48a4-94ff-75ece269728f\") " Oct 30 17:21:20.541864 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:20.538964 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d917fe1-dd87-48a4-94ff-75ece269728f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d917fe1-dd87-48a4-94ff-75ece269728f-istiod-ca-cert\") pod \"1d917fe1-dd87-48a4-94ff-75ece269728f\" (UID: \"1d917fe1-dd87-48a4-94ff-75ece269728f\") " Oct 30 17:21:20.541864 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:20.539003 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d917fe1-dd87-48a4-94ff-75ece269728f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d917fe1-dd87-48a4-94ff-75ece269728f-istio-data\") pod \"1d917fe1-dd87-48a4-94ff-75ece269728f\" (UID: \"1d917fe1-dd87-48a4-94ff-75ece269728f\") " Oct 30 17:21:20.541864 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:20.539055 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d917fe1-dd87-48a4-94ff-75ece269728f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d917fe1-dd87-48a4-94ff-75ece269728f-istio-envoy\") pod \"1d917fe1-dd87-48a4-94ff-75ece269728f\" (UID: \"1d917fe1-dd87-48a4-94ff-75ece269728f\") " Oct 30 17:21:20.541864 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:20.539106 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d917fe1-dd87-48a4-94ff-75ece269728f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d917fe1-dd87-48a4-94ff-75ece269728f-credential-socket\") pod \"1d917fe1-dd87-48a4-94ff-75ece269728f\" (UID: \"1d917fe1-dd87-48a4-94ff-75ece269728f\") " Oct 30 17:21:20.541864 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:20.539155 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d917fe1-dd87-48a4-94ff-75ece269728f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d917fe1-dd87-48a4-94ff-75ece269728f-workload-certs\") pod \"1d917fe1-dd87-48a4-94ff-75ece269728f\" (UID: \"1d917fe1-dd87-48a4-94ff-75ece269728f\") " Oct 30 17:21:20.541864 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:20.539206 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d917fe1-dd87-48a4-94ff-75ece269728f-kube-api-access-5x8sd\" (UniqueName: \"kubernetes.io/projected/1d917fe1-dd87-48a4-94ff-75ece269728f-kube-api-access-5x8sd\") pod \"1d917fe1-dd87-48a4-94ff-75ece269728f\" (UID: \"1d917fe1-dd87-48a4-94ff-75ece269728f\") " Oct 30 17:21:20.541864 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:20.539256 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d917fe1-dd87-48a4-94ff-75ece269728f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d917fe1-dd87-48a4-94ff-75ece269728f-workload-socket\") pod \"1d917fe1-dd87-48a4-94ff-75ece269728f\" (UID: \"1d917fe1-dd87-48a4-94ff-75ece269728f\") " Oct 30 17:21:20.565824 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:20.555941 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d917fe1-dd87-48a4-94ff-75ece269728f-istio-data" pod "1d917fe1-dd87-48a4-94ff-75ece269728f" (UID: "1d917fe1-dd87-48a4-94ff-75ece269728f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:20.565824 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:20.561530 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d917fe1-dd87-48a4-94ff-75ece269728f-credential-socket" pod "1d917fe1-dd87-48a4-94ff-75ece269728f" (UID: "1d917fe1-dd87-48a4-94ff-75ece269728f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:20.565824 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:20.561948 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d917fe1-dd87-48a4-94ff-75ece269728f-istiod-ca-cert" pod "1d917fe1-dd87-48a4-94ff-75ece269728f" (UID: "1d917fe1-dd87-48a4-94ff-75ece269728f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:21:20.565824 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:20.562271 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d917fe1-dd87-48a4-94ff-75ece269728f-workload-certs" pod "1d917fe1-dd87-48a4-94ff-75ece269728f" (UID: "1d917fe1-dd87-48a4-94ff-75ece269728f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:20.574927 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:20.574882 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d917fe1-dd87-48a4-94ff-75ece269728f-workload-socket" pod "1d917fe1-dd87-48a4-94ff-75ece269728f" (UID: "1d917fe1-dd87-48a4-94ff-75ece269728f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:20.575946 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:20.575892 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1d917fe1-dd87-48a4-94ff-75ece269728f-istio-podinfo" pod "1d917fe1-dd87-48a4-94ff-75ece269728f" (UID: "1d917fe1-dd87-48a4-94ff-75ece269728f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:21:20.581121 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:20.576356 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d917fe1-dd87-48a4-94ff-75ece269728f-istio-envoy" pod "1d917fe1-dd87-48a4-94ff-75ece269728f" (UID: "1d917fe1-dd87-48a4-94ff-75ece269728f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:20.581121 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:20.578859 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d917fe1-dd87-48a4-94ff-75ece269728f-istio-token" pod "1d917fe1-dd87-48a4-94ff-75ece269728f" (UID: "1d917fe1-dd87-48a4-94ff-75ece269728f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:21:20.583410 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:20.583121 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d917fe1-dd87-48a4-94ff-75ece269728f-kube-api-access-5x8sd" pod "1d917fe1-dd87-48a4-94ff-75ece269728f" (UID: "1d917fe1-dd87-48a4-94ff-75ece269728f"). InnerVolumeSpecName "kube-api-access-5x8sd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:21:20.650552 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:20.650512 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d917fe1-dd87-48a4-94ff-75ece269728f-istiod-ca-cert\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:21:20.650552 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:20.650543 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d917fe1-dd87-48a4-94ff-75ece269728f-istio-data\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:21:20.650552 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:20.650552 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d917fe1-dd87-48a4-94ff-75ece269728f-istio-envoy\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:21:20.651003 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:20.650562 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d917fe1-dd87-48a4-94ff-75ece269728f-credential-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:21:20.651003 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:20.650572 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d917fe1-dd87-48a4-94ff-75ece269728f-workload-certs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:21:20.651003 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:20.650581 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5x8sd\" (UniqueName: \"kubernetes.io/projected/1d917fe1-dd87-48a4-94ff-75ece269728f-kube-api-access-5x8sd\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:21:20.651003 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:20.650588 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d917fe1-dd87-48a4-94ff-75ece269728f-workload-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:21:20.651003 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:20.650597 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d917fe1-dd87-48a4-94ff-75ece269728f-istio-podinfo\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:21:20.651003 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:20.650605 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d917fe1-dd87-48a4-94ff-75ece269728f-istio-token\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:21:20.793815 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:20.793718 4493 generic.go:356] "Generic (PLEG): container finished" podID="1d917fe1-dd87-48a4-94ff-75ece269728f" containerID="1d8d4bce72ac31f56e7d8737dd1fe0aee76c7d4bf7318aea6eca64f90c53f9a8" exitCode=0 Oct 30 17:21:20.793815 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:20.793799 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx" Oct 30 17:21:20.794041 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:20.793818 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx" event={"ID":"1d917fe1-dd87-48a4-94ff-75ece269728f","Type":"ContainerDied","Data":"1d8d4bce72ac31f56e7d8737dd1fe0aee76c7d4bf7318aea6eca64f90c53f9a8"} Oct 30 17:21:20.794041 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:20.793931 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx" event={"ID":"1d917fe1-dd87-48a4-94ff-75ece269728f","Type":"ContainerDied","Data":"1cf0c8fe6675a5a6d908c0851fe6bf683df53b657e25a8dbb7aaf1078fee8a83"} Oct 30 17:21:20.794041 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:20.793962 4493 scope.go:122] "RemoveContainer" containerID="1d8d4bce72ac31f56e7d8737dd1fe0aee76c7d4bf7318aea6eca64f90c53f9a8" Oct 30 17:21:20.806558 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:20.806500 4493 scope.go:122] "RemoveContainer" containerID="9c9355ed96a93693e74a7eb26b026e4646934012d696b37e1e19030aafe04913" Oct 30 17:21:20.817722 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:20.817699 4493 scope.go:122] "RemoveContainer" containerID="1d8d4bce72ac31f56e7d8737dd1fe0aee76c7d4bf7318aea6eca64f90c53f9a8" Oct 30 17:21:20.818104 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:21:20.818085 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d8d4bce72ac31f56e7d8737dd1fe0aee76c7d4bf7318aea6eca64f90c53f9a8\": container with ID starting with 1d8d4bce72ac31f56e7d8737dd1fe0aee76c7d4bf7318aea6eca64f90c53f9a8 not found: ID does not exist" containerID="1d8d4bce72ac31f56e7d8737dd1fe0aee76c7d4bf7318aea6eca64f90c53f9a8" Oct 30 17:21:20.818166 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:20.818113 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d8d4bce72ac31f56e7d8737dd1fe0aee76c7d4bf7318aea6eca64f90c53f9a8"} err="failed to get container status \"1d8d4bce72ac31f56e7d8737dd1fe0aee76c7d4bf7318aea6eca64f90c53f9a8\": rpc error: code = NotFound desc = could not find container \"1d8d4bce72ac31f56e7d8737dd1fe0aee76c7d4bf7318aea6eca64f90c53f9a8\": container with ID starting with 1d8d4bce72ac31f56e7d8737dd1fe0aee76c7d4bf7318aea6eca64f90c53f9a8 not found: ID does not exist" Oct 30 17:21:20.818166 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:20.818129 4493 scope.go:122] "RemoveContainer" containerID="9c9355ed96a93693e74a7eb26b026e4646934012d696b37e1e19030aafe04913" Oct 30 17:21:20.818394 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:21:20.818373 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c9355ed96a93693e74a7eb26b026e4646934012d696b37e1e19030aafe04913\": container with ID starting with 9c9355ed96a93693e74a7eb26b026e4646934012d696b37e1e19030aafe04913 not found: ID does not exist" containerID="9c9355ed96a93693e74a7eb26b026e4646934012d696b37e1e19030aafe04913" Oct 30 17:21:20.818444 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:20.818401 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c9355ed96a93693e74a7eb26b026e4646934012d696b37e1e19030aafe04913"} err="failed to get container status \"9c9355ed96a93693e74a7eb26b026e4646934012d696b37e1e19030aafe04913\": rpc error: code = NotFound desc = could not find container \"9c9355ed96a93693e74a7eb26b026e4646934012d696b37e1e19030aafe04913\": container with ID starting with 9c9355ed96a93693e74a7eb26b026e4646934012d696b37e1e19030aafe04913 not found: ID does not exist" Oct 30 17:21:20.849685 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:20.849652 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx"] Oct 30 17:21:20.860135 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:20.860101 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-smcvx"] Oct 30 17:21:20.935331 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:20.935299 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d917fe1-dd87-48a4-94ff-75ece269728f" path="/var/lib/kubelet/pods/1d917fe1-dd87-48a4-94ff-75ece269728f/volumes" Oct 30 17:21:20.935869 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:20.935853 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="866c22fc-a2d3-47db-bac5-8ff5eb84cd93" path="/var/lib/kubelet/pods/866c22fc-a2d3-47db-bac5-8ff5eb84cd93/volumes" Oct 30 17:21:27.868976 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:27.868938 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt"] Oct 30 17:21:27.872591 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:27.872549 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt" podUID="f06b3212-a348-4185-b625-530f85fa91bf" containerName="istio-proxy" containerID="cri-o://2a33aab82a525ef6f843527938c8af09dca8fe5f10791fc2e3639424747e59a9" gracePeriod=30 Oct 30 17:21:27.875805 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:27.875762 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw"] Oct 30 17:21:27.876126 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:27.876074 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw" podUID="66c82e49-cc58-4f4c-b090-78dbcbdff588" containerName="istio-proxy" containerID="cri-o://8aa60a7cf4135b49980a4a40788696424a55f27cc66f1a1db3c113a34ca1fe17" gracePeriod=30 Oct 30 17:21:28.408397 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:28.408361 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d5b798-26fc4"] Oct 30 17:21:28.409437 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:28.409398 4493 cpu_manager.go:414] "containerMap: removing container" podUID="1d917fe1-dd87-48a4-94ff-75ece269728f" containerName="istio-proxy" Oct 30 17:21:28.409437 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:28.409425 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d917fe1-dd87-48a4-94ff-75ece269728f" containerName="istio-proxy" Oct 30 17:21:28.409652 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:28.409451 4493 cpu_manager.go:414] "containerMap: removing container" podUID="1d917fe1-dd87-48a4-94ff-75ece269728f" containerName="istio-proxy" Oct 30 17:21:28.409652 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:28.409459 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d917fe1-dd87-48a4-94ff-75ece269728f" containerName="istio-proxy" Oct 30 17:21:28.409725 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:28.409654 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d917fe1-dd87-48a4-94ff-75ece269728f" containerName="istio-proxy" Oct 30 17:21:28.414093 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:28.414074 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d5b798-26fc4" Oct 30 17:21:28.428919 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:28.428874 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:21:28.428919 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:28.428935 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:21:28.434402 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:28.434378 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d5b798-26fc4"] Oct 30 17:21:28.475967 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:28.475938 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fffa98d4-f627-438b-8f08-c5ecc86813c1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-d5b798-26fc4\" (UID: \"fffa98d4-f627-438b-8f08-c5ecc86813c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d5b798-26fc4" Oct 30 17:21:28.476123 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:28.475987 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fffa98d4-f627-438b-8f08-c5ecc86813c1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-d5b798-26fc4\" (UID: \"fffa98d4-f627-438b-8f08-c5ecc86813c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d5b798-26fc4" Oct 30 17:21:28.476316 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:28.476293 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5h9d\" (UniqueName: \"kubernetes.io/projected/fffa98d4-f627-438b-8f08-c5ecc86813c1-kube-api-access-w5h9d\") pod \"maas-api-revoke-keys-e2e-extmod-a-d5b798-26fc4\" (UID: \"fffa98d4-f627-438b-8f08-c5ecc86813c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d5b798-26fc4" Oct 30 17:21:28.505633 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:28.505605 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:21:28.505826 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:28.505806 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 17:21:28.578135 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:28.578098 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5h9d\" (UniqueName: \"kubernetes.io/projected/fffa98d4-f627-438b-8f08-c5ecc86813c1-kube-api-access-w5h9d\") pod \"maas-api-revoke-keys-e2e-extmod-a-d5b798-26fc4\" (UID: \"fffa98d4-f627-438b-8f08-c5ecc86813c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d5b798-26fc4" Oct 30 17:21:28.578297 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:28.578236 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fffa98d4-f627-438b-8f08-c5ecc86813c1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-d5b798-26fc4\" (UID: \"fffa98d4-f627-438b-8f08-c5ecc86813c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d5b798-26fc4" Oct 30 17:21:28.578297 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:28.578257 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fffa98d4-f627-438b-8f08-c5ecc86813c1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-d5b798-26fc4\" (UID: \"fffa98d4-f627-438b-8f08-c5ecc86813c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d5b798-26fc4" Oct 30 17:21:28.579036 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:28.578966 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fffa98d4-f627-438b-8f08-c5ecc86813c1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-d5b798-26fc4\" (UID: \"fffa98d4-f627-438b-8f08-c5ecc86813c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d5b798-26fc4" Oct 30 17:21:28.580757 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:28.580733 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fffa98d4-f627-438b-8f08-c5ecc86813c1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-d5b798-26fc4\" (UID: \"fffa98d4-f627-438b-8f08-c5ecc86813c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d5b798-26fc4" Oct 30 17:21:28.604931 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:28.604893 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5h9d\" (UniqueName: \"kubernetes.io/projected/fffa98d4-f627-438b-8f08-c5ecc86813c1-kube-api-access-w5h9d\") pod \"maas-api-revoke-keys-e2e-extmod-a-d5b798-26fc4\" (UID: \"fffa98d4-f627-438b-8f08-c5ecc86813c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d5b798-26fc4" Oct 30 17:21:28.732164 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:28.732074 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d5b798-26fc4" Oct 30 17:21:29.045711 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:29.045683 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d5b798-26fc4"] Oct 30 17:21:29.898506 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:29.898465 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d5b798-26fc4" event={"ID":"fffa98d4-f627-438b-8f08-c5ecc86813c1","Type":"ContainerStarted","Data":"93f19a6d71688b2a68119cdcc4d935dc27976fbbfc73b7a63f448f64a002d167"} Oct 30 17:21:29.898693 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:29.898549 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d5b798-26fc4" event={"ID":"fffa98d4-f627-438b-8f08-c5ecc86813c1","Type":"ContainerStarted","Data":"8840c65d40e7b135211a94a15edd06ea75e1ca472e27049c972080109eec43d1"} Oct 30 17:21:29.940107 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:29.940054 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d5b798-26fc4" podStartSLOduration=1.940040749 podStartE2EDuration="1.940040749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:21:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:21:29.938651729 +0000 UTC m=+3517.165826830" watchObservedRunningTime="2026-10-30 17:21:29.940040749 +0000 UTC m=+3517.167215897" Oct 30 17:21:30.347517 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:21:30.347467 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 17:21:30.908184 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:30.908149 4493 generic.go:356] "Generic (PLEG): container finished" podID="fffa98d4-f627-438b-8f08-c5ecc86813c1" containerID="93f19a6d71688b2a68119cdcc4d935dc27976fbbfc73b7a63f448f64a002d167" exitCode=0 Oct 30 17:21:30.908407 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:30.908221 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d5b798-26fc4" event={"ID":"fffa98d4-f627-438b-8f08-c5ecc86813c1","Type":"ContainerDied","Data":"93f19a6d71688b2a68119cdcc4d935dc27976fbbfc73b7a63f448f64a002d167"} Oct 30 17:21:31.807733 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:31.807702 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw"] Oct 30 17:21:31.808270 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:31.807938 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw" podUID="27945ccd-4a29-4f22-a0a2-e1f663869939" containerName="istio-proxy" containerID="cri-o://00877711eaa6547d40d1869bd176550476daecb42e3b523593e66bd1aee01e77" gracePeriod=30 Oct 30 17:21:32.143487 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:32.143460 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d5b798-26fc4" Oct 30 17:21:32.247803 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:32.242928 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fffa98d4-f627-438b-8f08-c5ecc86813c1-kube-api-access-w5h9d\" (UniqueName: \"kubernetes.io/projected/fffa98d4-f627-438b-8f08-c5ecc86813c1-kube-api-access-w5h9d\") pod \"fffa98d4-f627-438b-8f08-c5ecc86813c1\" (UID: \"fffa98d4-f627-438b-8f08-c5ecc86813c1\") " Oct 30 17:21:32.247803 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:32.243102 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fffa98d4-f627-438b-8f08-c5ecc86813c1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fffa98d4-f627-438b-8f08-c5ecc86813c1-maas-api-openshift-ca\") pod \"fffa98d4-f627-438b-8f08-c5ecc86813c1\" (UID: \"fffa98d4-f627-438b-8f08-c5ecc86813c1\") " Oct 30 17:21:32.247803 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:32.243186 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fffa98d4-f627-438b-8f08-c5ecc86813c1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fffa98d4-f627-438b-8f08-c5ecc86813c1-maas-api-xks-ca\") pod \"fffa98d4-f627-438b-8f08-c5ecc86813c1\" (UID: \"fffa98d4-f627-438b-8f08-c5ecc86813c1\") " Oct 30 17:21:32.247803 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:32.243651 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fffa98d4-f627-438b-8f08-c5ecc86813c1-maas-api-openshift-ca" pod "fffa98d4-f627-438b-8f08-c5ecc86813c1" (UID: "fffa98d4-f627-438b-8f08-c5ecc86813c1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:21:32.247803 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:32.245165 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fffa98d4-f627-438b-8f08-c5ecc86813c1-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:21:32.248810 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:32.248673 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fffa98d4-f627-438b-8f08-c5ecc86813c1-maas-api-xks-ca" pod "fffa98d4-f627-438b-8f08-c5ecc86813c1" (UID: "fffa98d4-f627-438b-8f08-c5ecc86813c1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:21:32.258997 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:32.258973 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fffa98d4-f627-438b-8f08-c5ecc86813c1-kube-api-access-w5h9d" pod "fffa98d4-f627-438b-8f08-c5ecc86813c1" (UID: "fffa98d4-f627-438b-8f08-c5ecc86813c1"). InnerVolumeSpecName "kube-api-access-w5h9d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:21:32.346707 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:32.346671 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5h9d\" (UniqueName: \"kubernetes.io/projected/fffa98d4-f627-438b-8f08-c5ecc86813c1-kube-api-access-w5h9d\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:21:32.346707 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:32.346710 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fffa98d4-f627-438b-8f08-c5ecc86813c1-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:21:32.948705 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:32.948674 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d5b798-26fc4" event={"ID":"fffa98d4-f627-438b-8f08-c5ecc86813c1","Type":"ContainerDied","Data":"8840c65d40e7b135211a94a15edd06ea75e1ca472e27049c972080109eec43d1"} Oct 30 17:21:32.948705 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:32.948691 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d5b798-26fc4" Oct 30 17:21:32.949124 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:32.948719 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8840c65d40e7b135211a94a15edd06ea75e1ca472e27049c972080109eec43d1" Oct 30 17:21:33.221953 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:33.221912 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt" Oct 30 17:21:33.291218 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:33.291188 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw" Oct 30 17:21:33.364026 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:33.363991 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f06b3212-a348-4185-b625-530f85fa91bf-istio-token\" (UniqueName: \"kubernetes.io/projected/f06b3212-a348-4185-b625-530f85fa91bf-istio-token\") pod \"f06b3212-a348-4185-b625-530f85fa91bf\" (UID: \"f06b3212-a348-4185-b625-530f85fa91bf\") " Oct 30 17:21:33.364224 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:33.364045 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f06b3212-a348-4185-b625-530f85fa91bf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f06b3212-a348-4185-b625-530f85fa91bf-istiod-ca-cert\") pod \"f06b3212-a348-4185-b625-530f85fa91bf\" (UID: \"f06b3212-a348-4185-b625-530f85fa91bf\") " Oct 30 17:21:33.364224 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:33.364111 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66c82e49-cc58-4f4c-b090-78dbcbdff588-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66c82e49-cc58-4f4c-b090-78dbcbdff588-workload-socket\") pod \"66c82e49-cc58-4f4c-b090-78dbcbdff588\" (UID: \"66c82e49-cc58-4f4c-b090-78dbcbdff588\") " Oct 30 17:21:33.364224 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:33.364141 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f06b3212-a348-4185-b625-530f85fa91bf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f06b3212-a348-4185-b625-530f85fa91bf-istio-envoy\") pod \"f06b3212-a348-4185-b625-530f85fa91bf\" (UID: \"f06b3212-a348-4185-b625-530f85fa91bf\") " Oct 30 17:21:33.364224 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:33.364204 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f06b3212-a348-4185-b625-530f85fa91bf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f06b3212-a348-4185-b625-530f85fa91bf-credential-socket\") pod \"f06b3212-a348-4185-b625-530f85fa91bf\" (UID: \"f06b3212-a348-4185-b625-530f85fa91bf\") " Oct 30 17:21:33.364463 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:33.364303 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f06b3212-a348-4185-b625-530f85fa91bf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f06b3212-a348-4185-b625-530f85fa91bf-workload-socket\") pod \"f06b3212-a348-4185-b625-530f85fa91bf\" (UID: \"f06b3212-a348-4185-b625-530f85fa91bf\") " Oct 30 17:21:33.364463 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:33.364403 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66c82e49-cc58-4f4c-b090-78dbcbdff588-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66c82e49-cc58-4f4c-b090-78dbcbdff588-istio-envoy\") pod \"66c82e49-cc58-4f4c-b090-78dbcbdff588\" (UID: \"66c82e49-cc58-4f4c-b090-78dbcbdff588\") " Oct 30 17:21:33.364463 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:33.364413 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f06b3212-a348-4185-b625-530f85fa91bf-istiod-ca-cert" pod "f06b3212-a348-4185-b625-530f85fa91bf" (UID: "f06b3212-a348-4185-b625-530f85fa91bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:21:33.364618 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:33.364512 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f06b3212-a348-4185-b625-530f85fa91bf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f06b3212-a348-4185-b625-530f85fa91bf-istio-data\") pod \"f06b3212-a348-4185-b625-530f85fa91bf\" (UID: \"f06b3212-a348-4185-b625-530f85fa91bf\") " Oct 30 17:21:33.364618 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:33.364546 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66c82e49-cc58-4f4c-b090-78dbcbdff588-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66c82e49-cc58-4f4c-b090-78dbcbdff588-istiod-ca-cert\") pod \"66c82e49-cc58-4f4c-b090-78dbcbdff588\" (UID: \"66c82e49-cc58-4f4c-b090-78dbcbdff588\") " Oct 30 17:21:33.364732 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:33.364623 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66c82e49-cc58-4f4c-b090-78dbcbdff588-kube-api-access-th87s\" (UniqueName: \"kubernetes.io/projected/66c82e49-cc58-4f4c-b090-78dbcbdff588-kube-api-access-th87s\") pod \"66c82e49-cc58-4f4c-b090-78dbcbdff588\" (UID: \"66c82e49-cc58-4f4c-b090-78dbcbdff588\") " Oct 30 17:21:33.364732 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:33.364662 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f06b3212-a348-4185-b625-530f85fa91bf-kube-api-access-p9z9l\" (UniqueName: \"kubernetes.io/projected/f06b3212-a348-4185-b625-530f85fa91bf-kube-api-access-p9z9l\") pod \"f06b3212-a348-4185-b625-530f85fa91bf\" (UID: \"f06b3212-a348-4185-b625-530f85fa91bf\") " Oct 30 17:21:33.364732 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:33.364691 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66c82e49-cc58-4f4c-b090-78dbcbdff588-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66c82e49-cc58-4f4c-b090-78dbcbdff588-credential-socket\") pod \"66c82e49-cc58-4f4c-b090-78dbcbdff588\" (UID: \"66c82e49-cc58-4f4c-b090-78dbcbdff588\") " Oct 30 17:21:33.364732 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:33.364724 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/66c82e49-cc58-4f4c-b090-78dbcbdff588-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66c82e49-cc58-4f4c-b090-78dbcbdff588-istio-podinfo\") pod \"66c82e49-cc58-4f4c-b090-78dbcbdff588\" (UID: \"66c82e49-cc58-4f4c-b090-78dbcbdff588\") " Oct 30 17:21:33.366152 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:33.364851 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f06b3212-a348-4185-b625-530f85fa91bf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f06b3212-a348-4185-b625-530f85fa91bf-istio-podinfo\") pod \"f06b3212-a348-4185-b625-530f85fa91bf\" (UID: \"f06b3212-a348-4185-b625-530f85fa91bf\") " Oct 30 17:21:33.366152 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:33.364947 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66c82e49-cc58-4f4c-b090-78dbcbdff588-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66c82e49-cc58-4f4c-b090-78dbcbdff588-workload-certs\") pod \"66c82e49-cc58-4f4c-b090-78dbcbdff588\" (UID: \"66c82e49-cc58-4f4c-b090-78dbcbdff588\") " Oct 30 17:21:33.366152 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:33.364983 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f06b3212-a348-4185-b625-530f85fa91bf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f06b3212-a348-4185-b625-530f85fa91bf-workload-certs\") pod \"f06b3212-a348-4185-b625-530f85fa91bf\" (UID: \"f06b3212-a348-4185-b625-530f85fa91bf\") " Oct 30 17:21:33.366152 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:33.365029 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66c82e49-cc58-4f4c-b090-78dbcbdff588-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66c82e49-cc58-4f4c-b090-78dbcbdff588-istio-data\") pod \"66c82e49-cc58-4f4c-b090-78dbcbdff588\" (UID: \"66c82e49-cc58-4f4c-b090-78dbcbdff588\") " Oct 30 17:21:33.366152 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:33.365027 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66c82e49-cc58-4f4c-b090-78dbcbdff588-workload-socket" pod "66c82e49-cc58-4f4c-b090-78dbcbdff588" (UID: "66c82e49-cc58-4f4c-b090-78dbcbdff588"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:33.366152 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:33.365150 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66c82e49-cc58-4f4c-b090-78dbcbdff588-istio-token\" (UniqueName: \"kubernetes.io/projected/66c82e49-cc58-4f4c-b090-78dbcbdff588-istio-token\") pod \"66c82e49-cc58-4f4c-b090-78dbcbdff588\" (UID: \"66c82e49-cc58-4f4c-b090-78dbcbdff588\") " Oct 30 17:21:33.366152 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:33.365246 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f06b3212-a348-4185-b625-530f85fa91bf-workload-socket" pod "f06b3212-a348-4185-b625-530f85fa91bf" (UID: "f06b3212-a348-4185-b625-530f85fa91bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:33.366152 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:33.365416 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f06b3212-a348-4185-b625-530f85fa91bf-credential-socket" pod "f06b3212-a348-4185-b625-530f85fa91bf" (UID: "f06b3212-a348-4185-b625-530f85fa91bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:33.366152 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:33.365498 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f06b3212-a348-4185-b625-530f85fa91bf-istio-data" pod "f06b3212-a348-4185-b625-530f85fa91bf" (UID: "f06b3212-a348-4185-b625-530f85fa91bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:33.366152 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:33.365593 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66c82e49-cc58-4f4c-b090-78dbcbdff588-credential-socket" pod "66c82e49-cc58-4f4c-b090-78dbcbdff588" (UID: "66c82e49-cc58-4f4c-b090-78dbcbdff588"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:33.366152 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:33.365604 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66c82e49-cc58-4f4c-b090-78dbcbdff588-workload-certs" pod "66c82e49-cc58-4f4c-b090-78dbcbdff588" (UID: "66c82e49-cc58-4f4c-b090-78dbcbdff588"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:33.367374 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:33.367241 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f06b3212-a348-4185-b625-530f85fa91bf-istio-envoy" pod "f06b3212-a348-4185-b625-530f85fa91bf" (UID: "f06b3212-a348-4185-b625-530f85fa91bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:33.367678 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:33.367656 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f06b3212-a348-4185-b625-530f85fa91bf-workload-certs" pod "f06b3212-a348-4185-b625-530f85fa91bf" (UID: "f06b3212-a348-4185-b625-530f85fa91bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:33.367861 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:33.367840 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66c82e49-cc58-4f4c-b090-78dbcbdff588-istio-data" pod "66c82e49-cc58-4f4c-b090-78dbcbdff588" (UID: "66c82e49-cc58-4f4c-b090-78dbcbdff588"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:33.368128 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:33.368110 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66c82e49-cc58-4f4c-b090-78dbcbdff588-istiod-ca-cert" pod "66c82e49-cc58-4f4c-b090-78dbcbdff588" (UID: "66c82e49-cc58-4f4c-b090-78dbcbdff588"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:21:33.368650 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:33.368635 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66c82e49-cc58-4f4c-b090-78dbcbdff588-workload-certs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:21:33.368695 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:33.368656 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f06b3212-a348-4185-b625-530f85fa91bf-workload-certs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:21:33.368695 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:33.368685 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66c82e49-cc58-4f4c-b090-78dbcbdff588-istio-data\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:21:33.368762 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:33.368698 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f06b3212-a348-4185-b625-530f85fa91bf-istiod-ca-cert\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:21:33.368762 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:33.368711 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66c82e49-cc58-4f4c-b090-78dbcbdff588-workload-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:21:33.368762 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:33.368725 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f06b3212-a348-4185-b625-530f85fa91bf-istio-envoy\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:21:33.368762 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:33.368738 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f06b3212-a348-4185-b625-530f85fa91bf-credential-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:21:33.368762 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:33.368753 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f06b3212-a348-4185-b625-530f85fa91bf-workload-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:21:33.369093 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:33.368767 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f06b3212-a348-4185-b625-530f85fa91bf-istio-data\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:21:33.369093 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:33.368799 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66c82e49-cc58-4f4c-b090-78dbcbdff588-istiod-ca-cert\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:21:33.369093 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:33.368813 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66c82e49-cc58-4f4c-b090-78dbcbdff588-credential-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:21:33.370462 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:33.370438 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f06b3212-a348-4185-b625-530f85fa91bf-istio-podinfo" pod "f06b3212-a348-4185-b625-530f85fa91bf" (UID: "f06b3212-a348-4185-b625-530f85fa91bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:21:33.370756 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:33.370735 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66c82e49-cc58-4f4c-b090-78dbcbdff588-istio-envoy" pod "66c82e49-cc58-4f4c-b090-78dbcbdff588" (UID: "66c82e49-cc58-4f4c-b090-78dbcbdff588"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:33.370869 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:33.370836 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f06b3212-a348-4185-b625-530f85fa91bf-kube-api-access-p9z9l" pod "f06b3212-a348-4185-b625-530f85fa91bf" (UID: "f06b3212-a348-4185-b625-530f85fa91bf"). InnerVolumeSpecName "kube-api-access-p9z9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:21:33.371997 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:33.371965 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66c82e49-cc58-4f4c-b090-78dbcbdff588-istio-token" pod "66c82e49-cc58-4f4c-b090-78dbcbdff588" (UID: "66c82e49-cc58-4f4c-b090-78dbcbdff588"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:21:33.372607 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:33.372583 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/66c82e49-cc58-4f4c-b090-78dbcbdff588-istio-podinfo" pod "66c82e49-cc58-4f4c-b090-78dbcbdff588" (UID: "66c82e49-cc58-4f4c-b090-78dbcbdff588"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:21:33.372911 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:33.372885 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66c82e49-cc58-4f4c-b090-78dbcbdff588-kube-api-access-th87s" pod "66c82e49-cc58-4f4c-b090-78dbcbdff588" (UID: "66c82e49-cc58-4f4c-b090-78dbcbdff588"). InnerVolumeSpecName "kube-api-access-th87s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:21:33.373102 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:33.373084 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f06b3212-a348-4185-b625-530f85fa91bf-istio-token" pod "f06b3212-a348-4185-b625-530f85fa91bf" (UID: "f06b3212-a348-4185-b625-530f85fa91bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:21:33.488898 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:33.483500 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f06b3212-a348-4185-b625-530f85fa91bf-istio-podinfo\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:21:33.488898 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:33.483532 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66c82e49-cc58-4f4c-b090-78dbcbdff588-istio-token\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:21:33.488898 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:33.483542 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f06b3212-a348-4185-b625-530f85fa91bf-istio-token\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:21:33.488898 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:33.483550 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66c82e49-cc58-4f4c-b090-78dbcbdff588-istio-envoy\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:21:33.488898 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:33.483558 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-th87s\" (UniqueName: \"kubernetes.io/projected/66c82e49-cc58-4f4c-b090-78dbcbdff588-kube-api-access-th87s\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:21:33.488898 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:33.483568 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9z9l\" (UniqueName: \"kubernetes.io/projected/f06b3212-a348-4185-b625-530f85fa91bf-kube-api-access-p9z9l\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:21:33.488898 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:33.483578 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66c82e49-cc58-4f4c-b090-78dbcbdff588-istio-podinfo\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:21:33.961163 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:33.961126 4493 generic.go:356] "Generic (PLEG): container finished" podID="f06b3212-a348-4185-b625-530f85fa91bf" containerID="2a33aab82a525ef6f843527938c8af09dca8fe5f10791fc2e3639424747e59a9" exitCode=0 Oct 30 17:21:33.961707 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:33.961172 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt" Oct 30 17:21:33.961707 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:33.961214 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt" event={"ID":"f06b3212-a348-4185-b625-530f85fa91bf","Type":"ContainerDied","Data":"2a33aab82a525ef6f843527938c8af09dca8fe5f10791fc2e3639424747e59a9"} Oct 30 17:21:33.961707 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:33.961282 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt" event={"ID":"f06b3212-a348-4185-b625-530f85fa91bf","Type":"ContainerDied","Data":"7cf66eeb74645492a4db642eaf829f5ffc3caf08bfc3cea28b984baf0eb47ad8"} Oct 30 17:21:33.961707 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:33.961315 4493 scope.go:122] "RemoveContainer" containerID="2a33aab82a525ef6f843527938c8af09dca8fe5f10791fc2e3639424747e59a9" Oct 30 17:21:33.963564 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:33.963538 4493 generic.go:356] "Generic (PLEG): container finished" podID="66c82e49-cc58-4f4c-b090-78dbcbdff588" containerID="8aa60a7cf4135b49980a4a40788696424a55f27cc66f1a1db3c113a34ca1fe17" exitCode=0 Oct 30 17:21:33.963677 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:33.963589 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw" event={"ID":"66c82e49-cc58-4f4c-b090-78dbcbdff588","Type":"ContainerDied","Data":"8aa60a7cf4135b49980a4a40788696424a55f27cc66f1a1db3c113a34ca1fe17"} Oct 30 17:21:33.963677 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:33.963631 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw" event={"ID":"66c82e49-cc58-4f4c-b090-78dbcbdff588","Type":"ContainerDied","Data":"1cabd735f05d364aa6672fb8d27fa6cfb309c531b3ce3d158a93111bfe0f3439"} Oct 30 17:21:33.963677 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:33.963640 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw" Oct 30 17:21:33.976879 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:33.976775 4493 scope.go:122] "RemoveContainer" containerID="2a33aab82a525ef6f843527938c8af09dca8fe5f10791fc2e3639424747e59a9" Oct 30 17:21:33.977139 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:21:33.977115 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a33aab82a525ef6f843527938c8af09dca8fe5f10791fc2e3639424747e59a9\": container with ID starting with 2a33aab82a525ef6f843527938c8af09dca8fe5f10791fc2e3639424747e59a9 not found: ID does not exist" containerID="2a33aab82a525ef6f843527938c8af09dca8fe5f10791fc2e3639424747e59a9" Oct 30 17:21:33.977232 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:33.977147 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a33aab82a525ef6f843527938c8af09dca8fe5f10791fc2e3639424747e59a9"} err="failed to get container status \"2a33aab82a525ef6f843527938c8af09dca8fe5f10791fc2e3639424747e59a9\": rpc error: code = NotFound desc = could not find container \"2a33aab82a525ef6f843527938c8af09dca8fe5f10791fc2e3639424747e59a9\": container with ID starting with 2a33aab82a525ef6f843527938c8af09dca8fe5f10791fc2e3639424747e59a9 not found: ID does not exist" Oct 30 17:21:33.977232 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:33.977163 4493 scope.go:122] "RemoveContainer" containerID="8aa60a7cf4135b49980a4a40788696424a55f27cc66f1a1db3c113a34ca1fe17" Oct 30 17:21:33.990731 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:33.990710 4493 scope.go:122] "RemoveContainer" containerID="847c48a76d37d99d788a15fb5702dfb7e31d66d156572538fbc95c640f105702" Oct 30 17:21:34.001541 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:34.001386 4493 scope.go:122] "RemoveContainer" containerID="8aa60a7cf4135b49980a4a40788696424a55f27cc66f1a1db3c113a34ca1fe17" Oct 30 17:21:34.001666 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:21:34.001644 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8aa60a7cf4135b49980a4a40788696424a55f27cc66f1a1db3c113a34ca1fe17\": container with ID starting with 8aa60a7cf4135b49980a4a40788696424a55f27cc66f1a1db3c113a34ca1fe17 not found: ID does not exist" containerID="8aa60a7cf4135b49980a4a40788696424a55f27cc66f1a1db3c113a34ca1fe17" Oct 30 17:21:34.001711 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:34.001672 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8aa60a7cf4135b49980a4a40788696424a55f27cc66f1a1db3c113a34ca1fe17"} err="failed to get container status \"8aa60a7cf4135b49980a4a40788696424a55f27cc66f1a1db3c113a34ca1fe17\": rpc error: code = NotFound desc = could not find container \"8aa60a7cf4135b49980a4a40788696424a55f27cc66f1a1db3c113a34ca1fe17\": container with ID starting with 8aa60a7cf4135b49980a4a40788696424a55f27cc66f1a1db3c113a34ca1fe17 not found: ID does not exist" Oct 30 17:21:34.001711 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:34.001688 4493 scope.go:122] "RemoveContainer" containerID="847c48a76d37d99d788a15fb5702dfb7e31d66d156572538fbc95c640f105702" Oct 30 17:21:34.001934 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:21:34.001917 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"847c48a76d37d99d788a15fb5702dfb7e31d66d156572538fbc95c640f105702\": container with ID starting with 847c48a76d37d99d788a15fb5702dfb7e31d66d156572538fbc95c640f105702 not found: ID does not exist" containerID="847c48a76d37d99d788a15fb5702dfb7e31d66d156572538fbc95c640f105702" Oct 30 17:21:34.001984 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:34.001940 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"847c48a76d37d99d788a15fb5702dfb7e31d66d156572538fbc95c640f105702"} err="failed to get container status \"847c48a76d37d99d788a15fb5702dfb7e31d66d156572538fbc95c640f105702\": rpc error: code = NotFound desc = could not find container \"847c48a76d37d99d788a15fb5702dfb7e31d66d156572538fbc95c640f105702\": container with ID starting with 847c48a76d37d99d788a15fb5702dfb7e31d66d156572538fbc95c640f105702 not found: ID does not exist" Oct 30 17:21:34.015684 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:34.015655 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw"] Oct 30 17:21:34.029359 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:34.029331 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-24acb2-openshift-default-7d5d746d89-v26pw"] Oct 30 17:21:34.056731 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:34.056702 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt"] Oct 30 17:21:34.061761 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:34.061736 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-d5b798-openshift-default-688df7b64f-c5kqt"] Oct 30 17:21:34.947708 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:34.947668 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66c82e49-cc58-4f4c-b090-78dbcbdff588" path="/var/lib/kubelet/pods/66c82e49-cc58-4f4c-b090-78dbcbdff588/volumes" Oct 30 17:21:34.948596 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:34.948570 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f06b3212-a348-4185-b625-530f85fa91bf" path="/var/lib/kubelet/pods/f06b3212-a348-4185-b625-530f85fa91bf/volumes" Oct 30 17:21:35.954629 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:35.954596 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl"] Oct 30 17:21:35.955187 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:35.954834 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl" podUID="35f0aae9-952b-4239-ae42-f9dfb3dd94be" containerName="istio-proxy" containerID="cri-o://e2ebfae1618e58a6b5da158c8d3200e7837c63201fe3e315a5d8090d34581620" gracePeriod=30 Oct 30 17:21:35.965313 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:35.965263 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb"] Oct 30 17:21:35.965517 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:35.965485 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb" podUID="84f45fe0-793d-4f0d-98d5-f78b9a287c5a" containerName="istio-proxy" containerID="cri-o://844da8f5d8e17772fdbe3f484dba17d3a764e5e1d2b796bc7a07dcadd2294c70" gracePeriod=30 Oct 30 17:21:37.050535 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:37.050493 4493 generic.go:356] "Generic (PLEG): container finished" podID="27945ccd-4a29-4f22-a0a2-e1f663869939" containerID="00877711eaa6547d40d1869bd176550476daecb42e3b523593e66bd1aee01e77" exitCode=0 Oct 30 17:21:37.051062 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:37.050710 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw" event={"ID":"27945ccd-4a29-4f22-a0a2-e1f663869939","Type":"ContainerDied","Data":"00877711eaa6547d40d1869bd176550476daecb42e3b523593e66bd1aee01e77"} Oct 30 17:21:37.193358 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:37.193329 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw" Oct 30 17:21:37.251403 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:37.251370 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27945ccd-4a29-4f22-a0a2-e1f663869939-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27945ccd-4a29-4f22-a0a2-e1f663869939-istio-data\") pod \"27945ccd-4a29-4f22-a0a2-e1f663869939\" (UID: \"27945ccd-4a29-4f22-a0a2-e1f663869939\") " Oct 30 17:21:37.251556 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:37.251459 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27945ccd-4a29-4f22-a0a2-e1f663869939-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27945ccd-4a29-4f22-a0a2-e1f663869939-istiod-ca-cert\") pod \"27945ccd-4a29-4f22-a0a2-e1f663869939\" (UID: \"27945ccd-4a29-4f22-a0a2-e1f663869939\") " Oct 30 17:21:37.251556 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:37.251522 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27945ccd-4a29-4f22-a0a2-e1f663869939-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27945ccd-4a29-4f22-a0a2-e1f663869939-workload-socket\") pod \"27945ccd-4a29-4f22-a0a2-e1f663869939\" (UID: \"27945ccd-4a29-4f22-a0a2-e1f663869939\") " Oct 30 17:21:37.251670 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:37.251562 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27945ccd-4a29-4f22-a0a2-e1f663869939-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27945ccd-4a29-4f22-a0a2-e1f663869939-credential-socket\") pod \"27945ccd-4a29-4f22-a0a2-e1f663869939\" (UID: \"27945ccd-4a29-4f22-a0a2-e1f663869939\") " Oct 30 17:21:37.251670 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:37.251618 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/27945ccd-4a29-4f22-a0a2-e1f663869939-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27945ccd-4a29-4f22-a0a2-e1f663869939-istio-podinfo\") pod \"27945ccd-4a29-4f22-a0a2-e1f663869939\" (UID: \"27945ccd-4a29-4f22-a0a2-e1f663869939\") " Oct 30 17:21:37.252307 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:37.252081 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27945ccd-4a29-4f22-a0a2-e1f663869939-istiod-ca-cert" pod "27945ccd-4a29-4f22-a0a2-e1f663869939" (UID: "27945ccd-4a29-4f22-a0a2-e1f663869939"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:21:37.252454 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:37.252341 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27945ccd-4a29-4f22-a0a2-e1f663869939-istio-data" pod "27945ccd-4a29-4f22-a0a2-e1f663869939" (UID: "27945ccd-4a29-4f22-a0a2-e1f663869939"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:37.252532 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:37.252503 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27945ccd-4a29-4f22-a0a2-e1f663869939-workload-socket" pod "27945ccd-4a29-4f22-a0a2-e1f663869939" (UID: "27945ccd-4a29-4f22-a0a2-e1f663869939"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:37.253759 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:37.252650 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27945ccd-4a29-4f22-a0a2-e1f663869939-credential-socket" pod "27945ccd-4a29-4f22-a0a2-e1f663869939" (UID: "27945ccd-4a29-4f22-a0a2-e1f663869939"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:37.253759 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:37.252771 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27945ccd-4a29-4f22-a0a2-e1f663869939-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27945ccd-4a29-4f22-a0a2-e1f663869939-istio-envoy\") pod \"27945ccd-4a29-4f22-a0a2-e1f663869939\" (UID: \"27945ccd-4a29-4f22-a0a2-e1f663869939\") " Oct 30 17:21:37.253759 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:37.252899 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27945ccd-4a29-4f22-a0a2-e1f663869939-istio-token\" (UniqueName: \"kubernetes.io/projected/27945ccd-4a29-4f22-a0a2-e1f663869939-istio-token\") pod \"27945ccd-4a29-4f22-a0a2-e1f663869939\" (UID: \"27945ccd-4a29-4f22-a0a2-e1f663869939\") " Oct 30 17:21:37.253759 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:37.252971 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27945ccd-4a29-4f22-a0a2-e1f663869939-kube-api-access-4cnjb\" (UniqueName: \"kubernetes.io/projected/27945ccd-4a29-4f22-a0a2-e1f663869939-kube-api-access-4cnjb\") pod \"27945ccd-4a29-4f22-a0a2-e1f663869939\" (UID: \"27945ccd-4a29-4f22-a0a2-e1f663869939\") " Oct 30 17:21:37.253759 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:37.253007 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27945ccd-4a29-4f22-a0a2-e1f663869939-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27945ccd-4a29-4f22-a0a2-e1f663869939-workload-certs\") pod \"27945ccd-4a29-4f22-a0a2-e1f663869939\" (UID: \"27945ccd-4a29-4f22-a0a2-e1f663869939\") " Oct 30 17:21:37.255378 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:37.255211 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27945ccd-4a29-4f22-a0a2-e1f663869939-workload-certs" pod "27945ccd-4a29-4f22-a0a2-e1f663869939" (UID: "27945ccd-4a29-4f22-a0a2-e1f663869939"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:37.262807 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:37.257334 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27945ccd-4a29-4f22-a0a2-e1f663869939-istiod-ca-cert\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:21:37.262807 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:37.257366 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27945ccd-4a29-4f22-a0a2-e1f663869939-workload-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:21:37.262807 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:37.257380 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27945ccd-4a29-4f22-a0a2-e1f663869939-credential-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:21:37.262807 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:37.257396 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27945ccd-4a29-4f22-a0a2-e1f663869939-workload-certs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:21:37.262807 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:37.257410 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27945ccd-4a29-4f22-a0a2-e1f663869939-istio-data\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:21:37.262807 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:37.257914 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27945ccd-4a29-4f22-a0a2-e1f663869939-istio-envoy" pod "27945ccd-4a29-4f22-a0a2-e1f663869939" (UID: "27945ccd-4a29-4f22-a0a2-e1f663869939"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:37.262807 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:37.261474 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27945ccd-4a29-4f22-a0a2-e1f663869939-istio-token" pod "27945ccd-4a29-4f22-a0a2-e1f663869939" (UID: "27945ccd-4a29-4f22-a0a2-e1f663869939"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:21:37.262807 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:37.261931 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27945ccd-4a29-4f22-a0a2-e1f663869939-kube-api-access-4cnjb" pod "27945ccd-4a29-4f22-a0a2-e1f663869939" (UID: "27945ccd-4a29-4f22-a0a2-e1f663869939"). InnerVolumeSpecName "kube-api-access-4cnjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:21:37.264660 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:37.264633 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/27945ccd-4a29-4f22-a0a2-e1f663869939-istio-podinfo" pod "27945ccd-4a29-4f22-a0a2-e1f663869939" (UID: "27945ccd-4a29-4f22-a0a2-e1f663869939"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:21:37.359525 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:37.359488 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27945ccd-4a29-4f22-a0a2-e1f663869939-istio-envoy\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:21:37.359525 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:37.359523 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27945ccd-4a29-4f22-a0a2-e1f663869939-istio-token\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:21:37.359827 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:37.359536 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cnjb\" (UniqueName: \"kubernetes.io/projected/27945ccd-4a29-4f22-a0a2-e1f663869939-kube-api-access-4cnjb\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:21:37.359827 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:37.359549 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27945ccd-4a29-4f22-a0a2-e1f663869939-istio-podinfo\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:21:38.061332 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:38.061304 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw" Oct 30 17:21:38.061813 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:38.061304 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw" event={"ID":"27945ccd-4a29-4f22-a0a2-e1f663869939","Type":"ContainerDied","Data":"063de8b1639358a36464a2abf015b37c3ad1d03aff6836911666e295e3183e57"} Oct 30 17:21:38.061813 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:38.061440 4493 scope.go:122] "RemoveContainer" containerID="00877711eaa6547d40d1869bd176550476daecb42e3b523593e66bd1aee01e77" Oct 30 17:21:38.130597 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:38.130564 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw"] Oct 30 17:21:38.140706 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:38.140678 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-2589b5-openshift-default-67d875dc85-f5jxw"] Oct 30 17:21:38.942436 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:38.942387 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27945ccd-4a29-4f22-a0a2-e1f663869939" path="/var/lib/kubelet/pods/27945ccd-4a29-4f22-a0a2-e1f663869939/volumes" Oct 30 17:21:41.106916 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:21:41.106880 4493 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35f0aae9_952b_4239_ae42_f9dfb3dd94be.slice/crio-conmon-e2ebfae1618e58a6b5da158c8d3200e7837c63201fe3e315a5d8090d34581620.scope\": RecentStats: unable to find data in memory cache]" Oct 30 17:21:41.148850 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:41.147939 4493 generic.go:356] "Generic (PLEG): container finished" podID="35f0aae9-952b-4239-ae42-f9dfb3dd94be" containerID="e2ebfae1618e58a6b5da158c8d3200e7837c63201fe3e315a5d8090d34581620" exitCode=0 Oct 30 17:21:41.148850 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:41.148086 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl" event={"ID":"35f0aae9-952b-4239-ae42-f9dfb3dd94be","Type":"ContainerDied","Data":"e2ebfae1618e58a6b5da158c8d3200e7837c63201fe3e315a5d8090d34581620"} Oct 30 17:21:41.150510 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:41.150457 4493 generic.go:356] "Generic (PLEG): container finished" podID="84f45fe0-793d-4f0d-98d5-f78b9a287c5a" containerID="844da8f5d8e17772fdbe3f484dba17d3a764e5e1d2b796bc7a07dcadd2294c70" exitCode=0 Oct 30 17:21:41.150639 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:41.150523 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb" event={"ID":"84f45fe0-793d-4f0d-98d5-f78b9a287c5a","Type":"ContainerDied","Data":"844da8f5d8e17772fdbe3f484dba17d3a764e5e1d2b796bc7a07dcadd2294c70"} Oct 30 17:21:41.375754 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:41.375415 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb" Oct 30 17:21:41.422263 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:41.422235 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl" Oct 30 17:21:41.430496 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:41.430454 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84f45fe0-793d-4f0d-98d5-f78b9a287c5a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84f45fe0-793d-4f0d-98d5-f78b9a287c5a-workload-socket\") pod \"84f45fe0-793d-4f0d-98d5-f78b9a287c5a\" (UID: \"84f45fe0-793d-4f0d-98d5-f78b9a287c5a\") " Oct 30 17:21:41.430629 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:41.430510 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35f0aae9-952b-4239-ae42-f9dfb3dd94be-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35f0aae9-952b-4239-ae42-f9dfb3dd94be-istio-envoy\") pod \"35f0aae9-952b-4239-ae42-f9dfb3dd94be\" (UID: \"35f0aae9-952b-4239-ae42-f9dfb3dd94be\") " Oct 30 17:21:41.430629 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:41.430554 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35f0aae9-952b-4239-ae42-f9dfb3dd94be-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35f0aae9-952b-4239-ae42-f9dfb3dd94be-istiod-ca-cert\") pod \"35f0aae9-952b-4239-ae42-f9dfb3dd94be\" (UID: \"35f0aae9-952b-4239-ae42-f9dfb3dd94be\") " Oct 30 17:21:41.430629 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:41.430591 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84f45fe0-793d-4f0d-98d5-f78b9a287c5a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84f45fe0-793d-4f0d-98d5-f78b9a287c5a-credential-socket\") pod \"84f45fe0-793d-4f0d-98d5-f78b9a287c5a\" (UID: \"84f45fe0-793d-4f0d-98d5-f78b9a287c5a\") " Oct 30 17:21:41.430629 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:41.430614 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84f45fe0-793d-4f0d-98d5-f78b9a287c5a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84f45fe0-793d-4f0d-98d5-f78b9a287c5a-istio-envoy\") pod \"84f45fe0-793d-4f0d-98d5-f78b9a287c5a\" (UID: \"84f45fe0-793d-4f0d-98d5-f78b9a287c5a\") " Oct 30 17:21:41.430805 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:41.430661 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84f45fe0-793d-4f0d-98d5-f78b9a287c5a-workload-socket" pod "84f45fe0-793d-4f0d-98d5-f78b9a287c5a" (UID: "84f45fe0-793d-4f0d-98d5-f78b9a287c5a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:41.430937 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:41.430919 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35f0aae9-952b-4239-ae42-f9dfb3dd94be-istiod-ca-cert" pod "35f0aae9-952b-4239-ae42-f9dfb3dd94be" (UID: "35f0aae9-952b-4239-ae42-f9dfb3dd94be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:21:41.430976 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:41.430957 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35f0aae9-952b-4239-ae42-f9dfb3dd94be-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35f0aae9-952b-4239-ae42-f9dfb3dd94be-workload-certs\") pod \"35f0aae9-952b-4239-ae42-f9dfb3dd94be\" (UID: \"35f0aae9-952b-4239-ae42-f9dfb3dd94be\") " Oct 30 17:21:41.431025 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:41.431012 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35f0aae9-952b-4239-ae42-f9dfb3dd94be-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f0aae9-952b-4239-ae42-f9dfb3dd94be-workload-socket\") pod \"35f0aae9-952b-4239-ae42-f9dfb3dd94be\" (UID: \"35f0aae9-952b-4239-ae42-f9dfb3dd94be\") " Oct 30 17:21:41.431088 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:41.431032 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35f0aae9-952b-4239-ae42-f9dfb3dd94be-kube-api-access-rpztg\" (UniqueName: \"kubernetes.io/projected/35f0aae9-952b-4239-ae42-f9dfb3dd94be-kube-api-access-rpztg\") pod \"35f0aae9-952b-4239-ae42-f9dfb3dd94be\" (UID: \"35f0aae9-952b-4239-ae42-f9dfb3dd94be\") " Oct 30 17:21:41.431088 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:41.431066 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84f45fe0-793d-4f0d-98d5-f78b9a287c5a-istio-token\" (UniqueName: \"kubernetes.io/projected/84f45fe0-793d-4f0d-98d5-f78b9a287c5a-istio-token\") pod \"84f45fe0-793d-4f0d-98d5-f78b9a287c5a\" (UID: \"84f45fe0-793d-4f0d-98d5-f78b9a287c5a\") " Oct 30 17:21:41.431185 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:41.431142 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84f45fe0-793d-4f0d-98d5-f78b9a287c5a-credential-socket" pod "84f45fe0-793d-4f0d-98d5-f78b9a287c5a" (UID: "84f45fe0-793d-4f0d-98d5-f78b9a287c5a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:41.431185 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:41.431156 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35f0aae9-952b-4239-ae42-f9dfb3dd94be-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f0aae9-952b-4239-ae42-f9dfb3dd94be-credential-socket\") pod \"35f0aae9-952b-4239-ae42-f9dfb3dd94be\" (UID: \"35f0aae9-952b-4239-ae42-f9dfb3dd94be\") " Oct 30 17:21:41.431297 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:41.431190 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35f0aae9-952b-4239-ae42-f9dfb3dd94be-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35f0aae9-952b-4239-ae42-f9dfb3dd94be-istio-data\") pod \"35f0aae9-952b-4239-ae42-f9dfb3dd94be\" (UID: \"35f0aae9-952b-4239-ae42-f9dfb3dd94be\") " Oct 30 17:21:41.431297 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:41.431211 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/84f45fe0-793d-4f0d-98d5-f78b9a287c5a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84f45fe0-793d-4f0d-98d5-f78b9a287c5a-istio-podinfo\") pod \"84f45fe0-793d-4f0d-98d5-f78b9a287c5a\" (UID: \"84f45fe0-793d-4f0d-98d5-f78b9a287c5a\") " Oct 30 17:21:41.431297 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:41.431253 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84f45fe0-793d-4f0d-98d5-f78b9a287c5a-kube-api-access-54pw5\" (UniqueName: \"kubernetes.io/projected/84f45fe0-793d-4f0d-98d5-f78b9a287c5a-kube-api-access-54pw5\") pod \"84f45fe0-793d-4f0d-98d5-f78b9a287c5a\" (UID: \"84f45fe0-793d-4f0d-98d5-f78b9a287c5a\") " Oct 30 17:21:41.431638 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:41.431298 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84f45fe0-793d-4f0d-98d5-f78b9a287c5a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84f45fe0-793d-4f0d-98d5-f78b9a287c5a-workload-certs\") pod \"84f45fe0-793d-4f0d-98d5-f78b9a287c5a\" (UID: \"84f45fe0-793d-4f0d-98d5-f78b9a287c5a\") " Oct 30 17:21:41.431638 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:41.431307 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35f0aae9-952b-4239-ae42-f9dfb3dd94be-workload-socket" pod "35f0aae9-952b-4239-ae42-f9dfb3dd94be" (UID: "35f0aae9-952b-4239-ae42-f9dfb3dd94be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:41.431638 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:41.431317 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35f0aae9-952b-4239-ae42-f9dfb3dd94be-istio-token\" (UniqueName: \"kubernetes.io/projected/35f0aae9-952b-4239-ae42-f9dfb3dd94be-istio-token\") pod \"35f0aae9-952b-4239-ae42-f9dfb3dd94be\" (UID: \"35f0aae9-952b-4239-ae42-f9dfb3dd94be\") " Oct 30 17:21:41.431638 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:41.431357 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84f45fe0-793d-4f0d-98d5-f78b9a287c5a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84f45fe0-793d-4f0d-98d5-f78b9a287c5a-istiod-ca-cert\") pod \"84f45fe0-793d-4f0d-98d5-f78b9a287c5a\" (UID: \"84f45fe0-793d-4f0d-98d5-f78b9a287c5a\") " Oct 30 17:21:41.431638 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:41.431410 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84f45fe0-793d-4f0d-98d5-f78b9a287c5a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84f45fe0-793d-4f0d-98d5-f78b9a287c5a-istio-data\") pod \"84f45fe0-793d-4f0d-98d5-f78b9a287c5a\" (UID: \"84f45fe0-793d-4f0d-98d5-f78b9a287c5a\") " Oct 30 17:21:41.431638 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:41.431438 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/35f0aae9-952b-4239-ae42-f9dfb3dd94be-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35f0aae9-952b-4239-ae42-f9dfb3dd94be-istio-podinfo\") pod \"35f0aae9-952b-4239-ae42-f9dfb3dd94be\" (UID: \"35f0aae9-952b-4239-ae42-f9dfb3dd94be\") " Oct 30 17:21:41.432078 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:41.432055 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35f0aae9-952b-4239-ae42-f9dfb3dd94be-workload-certs" pod "35f0aae9-952b-4239-ae42-f9dfb3dd94be" (UID: "35f0aae9-952b-4239-ae42-f9dfb3dd94be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:41.432206 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:41.432187 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84f45fe0-793d-4f0d-98d5-f78b9a287c5a-workload-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:21:41.432284 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:41.432213 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35f0aae9-952b-4239-ae42-f9dfb3dd94be-istiod-ca-cert\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:21:41.432284 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:41.432229 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84f45fe0-793d-4f0d-98d5-f78b9a287c5a-credential-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:21:41.432284 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:41.432244 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35f0aae9-952b-4239-ae42-f9dfb3dd94be-workload-certs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:21:41.432284 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:41.432258 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f0aae9-952b-4239-ae42-f9dfb3dd94be-workload-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:21:41.432498 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:41.432479 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35f0aae9-952b-4239-ae42-f9dfb3dd94be-credential-socket" pod "35f0aae9-952b-4239-ae42-f9dfb3dd94be" (UID: "35f0aae9-952b-4239-ae42-f9dfb3dd94be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:41.434035 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:41.434008 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84f45fe0-793d-4f0d-98d5-f78b9a287c5a-istio-envoy" pod "84f45fe0-793d-4f0d-98d5-f78b9a287c5a" (UID: "84f45fe0-793d-4f0d-98d5-f78b9a287c5a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:41.434153 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:41.434048 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35f0aae9-952b-4239-ae42-f9dfb3dd94be-istio-envoy" pod "35f0aae9-952b-4239-ae42-f9dfb3dd94be" (UID: "35f0aae9-952b-4239-ae42-f9dfb3dd94be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:41.434264 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:41.434243 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84f45fe0-793d-4f0d-98d5-f78b9a287c5a-workload-certs" pod "84f45fe0-793d-4f0d-98d5-f78b9a287c5a" (UID: "84f45fe0-793d-4f0d-98d5-f78b9a287c5a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:41.434264 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:41.434252 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35f0aae9-952b-4239-ae42-f9dfb3dd94be-istio-data" pod "35f0aae9-952b-4239-ae42-f9dfb3dd94be" (UID: "35f0aae9-952b-4239-ae42-f9dfb3dd94be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:41.434426 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:41.434410 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84f45fe0-793d-4f0d-98d5-f78b9a287c5a-istio-data" pod "84f45fe0-793d-4f0d-98d5-f78b9a287c5a" (UID: "84f45fe0-793d-4f0d-98d5-f78b9a287c5a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:41.434596 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:41.434565 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84f45fe0-793d-4f0d-98d5-f78b9a287c5a-istiod-ca-cert" pod "84f45fe0-793d-4f0d-98d5-f78b9a287c5a" (UID: "84f45fe0-793d-4f0d-98d5-f78b9a287c5a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:21:41.435038 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:41.434935 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/84f45fe0-793d-4f0d-98d5-f78b9a287c5a-istio-podinfo" pod "84f45fe0-793d-4f0d-98d5-f78b9a287c5a" (UID: "84f45fe0-793d-4f0d-98d5-f78b9a287c5a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:21:41.435038 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:41.435001 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84f45fe0-793d-4f0d-98d5-f78b9a287c5a-istio-token" pod "84f45fe0-793d-4f0d-98d5-f78b9a287c5a" (UID: "84f45fe0-793d-4f0d-98d5-f78b9a287c5a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:21:41.435292 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:41.435270 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35f0aae9-952b-4239-ae42-f9dfb3dd94be-istio-token" pod "35f0aae9-952b-4239-ae42-f9dfb3dd94be" (UID: "35f0aae9-952b-4239-ae42-f9dfb3dd94be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:21:41.435379 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:41.435303 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84f45fe0-793d-4f0d-98d5-f78b9a287c5a-kube-api-access-54pw5" pod "84f45fe0-793d-4f0d-98d5-f78b9a287c5a" (UID: "84f45fe0-793d-4f0d-98d5-f78b9a287c5a"). InnerVolumeSpecName "kube-api-access-54pw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:21:41.436296 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:41.436269 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35f0aae9-952b-4239-ae42-f9dfb3dd94be-istio-podinfo" pod "35f0aae9-952b-4239-ae42-f9dfb3dd94be" (UID: "35f0aae9-952b-4239-ae42-f9dfb3dd94be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:21:41.436542 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:41.436502 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35f0aae9-952b-4239-ae42-f9dfb3dd94be-kube-api-access-rpztg" pod "35f0aae9-952b-4239-ae42-f9dfb3dd94be" (UID: "35f0aae9-952b-4239-ae42-f9dfb3dd94be"). InnerVolumeSpecName "kube-api-access-rpztg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:21:41.534037 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:41.533995 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54pw5\" (UniqueName: \"kubernetes.io/projected/84f45fe0-793d-4f0d-98d5-f78b9a287c5a-kube-api-access-54pw5\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:21:41.534037 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:41.534029 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84f45fe0-793d-4f0d-98d5-f78b9a287c5a-workload-certs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:21:41.534037 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:41.534043 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35f0aae9-952b-4239-ae42-f9dfb3dd94be-istio-token\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:21:41.534310 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:41.534055 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84f45fe0-793d-4f0d-98d5-f78b9a287c5a-istiod-ca-cert\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:21:41.534310 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:41.534069 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84f45fe0-793d-4f0d-98d5-f78b9a287c5a-istio-data\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:21:41.534310 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:41.534079 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35f0aae9-952b-4239-ae42-f9dfb3dd94be-istio-podinfo\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:21:41.534310 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:41.534090 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35f0aae9-952b-4239-ae42-f9dfb3dd94be-istio-envoy\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:21:41.534310 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:41.534102 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84f45fe0-793d-4f0d-98d5-f78b9a287c5a-istio-envoy\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:21:41.534310 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:41.534112 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpztg\" (UniqueName: \"kubernetes.io/projected/35f0aae9-952b-4239-ae42-f9dfb3dd94be-kube-api-access-rpztg\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:21:41.534310 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:41.534124 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84f45fe0-793d-4f0d-98d5-f78b9a287c5a-istio-token\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:21:41.534310 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:41.534135 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35f0aae9-952b-4239-ae42-f9dfb3dd94be-credential-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:21:41.534310 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:41.534147 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35f0aae9-952b-4239-ae42-f9dfb3dd94be-istio-data\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:21:41.534310 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:41.534158 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84f45fe0-793d-4f0d-98d5-f78b9a287c5a-istio-podinfo\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:21:42.159926 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:42.159896 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl" Oct 30 17:21:42.159926 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:42.159916 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl" event={"ID":"35f0aae9-952b-4239-ae42-f9dfb3dd94be","Type":"ContainerDied","Data":"06ce16c10916a495d5a2f292c27d3623f976a24e79b7a0a41384b323c2c2b543"} Oct 30 17:21:42.160464 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:42.159980 4493 scope.go:122] "RemoveContainer" containerID="e2ebfae1618e58a6b5da158c8d3200e7837c63201fe3e315a5d8090d34581620" Oct 30 17:21:42.161547 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:42.161529 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb" Oct 30 17:21:42.161650 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:42.161575 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb" event={"ID":"84f45fe0-793d-4f0d-98d5-f78b9a287c5a","Type":"ContainerDied","Data":"0557abb52e80988ea44898ecdfde455af7aa05b41c9ca3badfef60f24e5cdc99"} Oct 30 17:21:42.174136 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:42.174111 4493 scope.go:122] "RemoveContainer" containerID="844da8f5d8e17772fdbe3f484dba17d3a764e5e1d2b796bc7a07dcadd2294c70" Oct 30 17:21:42.217697 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:42.217665 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl"] Oct 30 17:21:42.224315 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:42.224290 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-3106d6-openshift-default-5876659797-86nhl"] Oct 30 17:21:42.252248 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:42.252136 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb"] Oct 30 17:21:42.258831 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:42.258803 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-cb6251-openshift-default-666cb7988f-x8wmb"] Oct 30 17:21:42.936370 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:42.936330 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35f0aae9-952b-4239-ae42-f9dfb3dd94be" path="/var/lib/kubelet/pods/35f0aae9-952b-4239-ae42-f9dfb3dd94be/volumes" Oct 30 17:21:42.937003 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:42.936980 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84f45fe0-793d-4f0d-98d5-f78b9a287c5a" path="/var/lib/kubelet/pods/84f45fe0-793d-4f0d-98d5-f78b9a287c5a/volumes" Oct 30 17:21:46.697565 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:46.696665 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d5b798-26fc4"] Oct 30 17:21:46.699523 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:46.699496 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-d5b798-26fc4"] Oct 30 17:21:46.936253 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:46.936215 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fffa98d4-f627-438b-8f08-c5ecc86813c1" path="/var/lib/kubelet/pods/fffa98d4-f627-438b-8f08-c5ecc86813c1/volumes" Oct 30 17:21:50.938074 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:50.938041 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g"] Oct 30 17:21:50.938643 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:50.938246 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g" podUID="5a4af0cd-c986-4b31-b123-545551673476" containerName="istio-proxy" containerID="cri-o://aa7da326256c6a80b6c838b35b061c7ac6712e8ff7a185f4408c6392edcaed72" gracePeriod=30 Oct 30 17:21:50.976420 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:50.976389 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8"] Oct 30 17:21:50.976618 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:50.976565 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8" podUID="56700de4-8e2f-4ede-b163-f152add8e6fa" containerName="istio-proxy" containerID="cri-o://40aea16c219909e45a0fcac30a7f966c94b874ae718a7470de494597b5afbb69" gracePeriod=30 Oct 30 17:21:54.479718 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:54.479678 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-abda73-d96hh"] Oct 30 17:21:54.480342 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:54.480326 4493 cpu_manager.go:414] "containerMap: removing container" podUID="84f45fe0-793d-4f0d-98d5-f78b9a287c5a" containerName="istio-proxy" Oct 30 17:21:54.480342 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:54.480342 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84f45fe0-793d-4f0d-98d5-f78b9a287c5a" containerName="istio-proxy" Oct 30 17:21:54.480493 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:54.480379 4493 cpu_manager.go:414] "containerMap: removing container" podUID="35f0aae9-952b-4239-ae42-f9dfb3dd94be" containerName="istio-proxy" Oct 30 17:21:54.480493 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:54.480385 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35f0aae9-952b-4239-ae42-f9dfb3dd94be" containerName="istio-proxy" Oct 30 17:21:54.480493 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:54.480397 4493 cpu_manager.go:414] "containerMap: removing container" podUID="27945ccd-4a29-4f22-a0a2-e1f663869939" containerName="istio-proxy" Oct 30 17:21:54.480493 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:54.480402 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27945ccd-4a29-4f22-a0a2-e1f663869939" containerName="istio-proxy" Oct 30 17:21:54.480493 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:54.480412 4493 cpu_manager.go:414] "containerMap: removing container" podUID="66c82e49-cc58-4f4c-b090-78dbcbdff588" containerName="istio-proxy" Oct 30 17:21:54.480493 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:54.480419 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66c82e49-cc58-4f4c-b090-78dbcbdff588" containerName="istio-proxy" Oct 30 17:21:54.480493 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:54.480433 4493 cpu_manager.go:414] "containerMap: removing container" podUID="f06b3212-a348-4185-b625-530f85fa91bf" containerName="istio-proxy" Oct 30 17:21:54.480493 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:54.480440 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f06b3212-a348-4185-b625-530f85fa91bf" containerName="istio-proxy" Oct 30 17:21:54.480493 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:54.480465 4493 cpu_manager.go:414] "containerMap: removing container" podUID="fffa98d4-f627-438b-8f08-c5ecc86813c1" containerName="revoke-keys" Oct 30 17:21:54.480493 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:54.480469 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fffa98d4-f627-438b-8f08-c5ecc86813c1" containerName="revoke-keys" Oct 30 17:21:54.481029 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:54.480581 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="84f45fe0-793d-4f0d-98d5-f78b9a287c5a" containerName="istio-proxy" Oct 30 17:21:54.481029 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:54.480594 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="35f0aae9-952b-4239-ae42-f9dfb3dd94be" containerName="istio-proxy" Oct 30 17:21:54.481029 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:54.480601 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="27945ccd-4a29-4f22-a0a2-e1f663869939" containerName="istio-proxy" Oct 30 17:21:54.481029 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:54.480607 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="66c82e49-cc58-4f4c-b090-78dbcbdff588" containerName="istio-proxy" Oct 30 17:21:54.481029 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:54.480613 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="f06b3212-a348-4185-b625-530f85fa91bf" containerName="istio-proxy" Oct 30 17:21:54.481029 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:54.480621 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d917fe1-dd87-48a4-94ff-75ece269728f" containerName="istio-proxy" Oct 30 17:21:54.481029 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:54.480628 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="fffa98d4-f627-438b-8f08-c5ecc86813c1" containerName="revoke-keys" Oct 30 17:21:54.485953 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:54.485933 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-abda73-d96hh" Oct 30 17:21:54.512988 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:54.512963 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:21:54.513184 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:54.513058 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:21:54.519771 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:54.519743 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-abda73-d96hh"] Oct 30 17:21:54.521326 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:54.521303 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:21:54.523142 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:54.523122 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 17:21:54.581884 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:54.581849 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60f8eca9-6103-45f2-809e-cf45f72a7d5b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-abda73-d96hh\" (UID: \"60f8eca9-6103-45f2-809e-cf45f72a7d5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-abda73-d96hh" Oct 30 17:21:54.582059 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:54.581908 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4hbk\" (UniqueName: \"kubernetes.io/projected/60f8eca9-6103-45f2-809e-cf45f72a7d5b-kube-api-access-q4hbk\") pod \"maas-api-revoke-keys-e2e-extmod-b-abda73-d96hh\" (UID: \"60f8eca9-6103-45f2-809e-cf45f72a7d5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-abda73-d96hh" Oct 30 17:21:54.582059 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:54.582030 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60f8eca9-6103-45f2-809e-cf45f72a7d5b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-abda73-d96hh\" (UID: \"60f8eca9-6103-45f2-809e-cf45f72a7d5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-abda73-d96hh" Oct 30 17:21:54.683059 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:54.683023 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60f8eca9-6103-45f2-809e-cf45f72a7d5b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-abda73-d96hh\" (UID: \"60f8eca9-6103-45f2-809e-cf45f72a7d5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-abda73-d96hh" Oct 30 17:21:54.683273 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:54.683089 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4hbk\" (UniqueName: \"kubernetes.io/projected/60f8eca9-6103-45f2-809e-cf45f72a7d5b-kube-api-access-q4hbk\") pod \"maas-api-revoke-keys-e2e-extmod-b-abda73-d96hh\" (UID: \"60f8eca9-6103-45f2-809e-cf45f72a7d5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-abda73-d96hh" Oct 30 17:21:54.683273 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:54.683183 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60f8eca9-6103-45f2-809e-cf45f72a7d5b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-abda73-d96hh\" (UID: \"60f8eca9-6103-45f2-809e-cf45f72a7d5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-abda73-d96hh" Oct 30 17:21:54.683903 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:54.683832 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60f8eca9-6103-45f2-809e-cf45f72a7d5b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-abda73-d96hh\" (UID: \"60f8eca9-6103-45f2-809e-cf45f72a7d5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-abda73-d96hh" Oct 30 17:21:54.685638 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:54.685615 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60f8eca9-6103-45f2-809e-cf45f72a7d5b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-abda73-d96hh\" (UID: \"60f8eca9-6103-45f2-809e-cf45f72a7d5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-abda73-d96hh" Oct 30 17:21:54.706193 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:54.706158 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4hbk\" (UniqueName: \"kubernetes.io/projected/60f8eca9-6103-45f2-809e-cf45f72a7d5b-kube-api-access-q4hbk\") pod \"maas-api-revoke-keys-e2e-extmod-b-abda73-d96hh\" (UID: \"60f8eca9-6103-45f2-809e-cf45f72a7d5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-abda73-d96hh" Oct 30 17:21:54.800669 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:54.800591 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-abda73-d96hh" Oct 30 17:21:55.115861 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:21:55.115827 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60f8eca9_6103_45f2_809e_cf45f72a7d5b.slice/crio-dfa95f7ff48a91fe968565ed815dbff6ba9535c9f709a06ddfc9eee3406bdba3 WatchSource:0}: Error finding container dfa95f7ff48a91fe968565ed815dbff6ba9535c9f709a06ddfc9eee3406bdba3: Status 404 returned error can't find the container with id dfa95f7ff48a91fe968565ed815dbff6ba9535c9f709a06ddfc9eee3406bdba3 Oct 30 17:21:55.116061 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:55.115966 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-abda73-d96hh"] Oct 30 17:21:55.544253 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:55.544211 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-abda73-d96hh" event={"ID":"60f8eca9-6103-45f2-809e-cf45f72a7d5b","Type":"ContainerStarted","Data":"60a32d3a3f764df7376ec313753e3141af20e73724baea11cb6a4e6f215178f5"} Oct 30 17:21:55.544707 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:55.544268 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-abda73-d96hh" event={"ID":"60f8eca9-6103-45f2-809e-cf45f72a7d5b","Type":"ContainerStarted","Data":"dfa95f7ff48a91fe968565ed815dbff6ba9535c9f709a06ddfc9eee3406bdba3"} Oct 30 17:21:55.584248 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:55.584201 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-abda73-d96hh" podStartSLOduration=1.584187455 podStartE2EDuration="1.584187455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:21:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:21:55.583508027 +0000 UTC m=+3542.810683126" watchObservedRunningTime="2026-10-30 17:21:55.584187455 +0000 UTC m=+3542.811362577" Oct 30 17:21:56.283948 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:56.283916 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g" Oct 30 17:21:56.308106 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:56.308074 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a4af0cd-c986-4b31-b123-545551673476-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a4af0cd-c986-4b31-b123-545551673476-workload-certs\") pod \"5a4af0cd-c986-4b31-b123-545551673476\" (UID: \"5a4af0cd-c986-4b31-b123-545551673476\") " Oct 30 17:21:56.308272 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:56.308156 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a4af0cd-c986-4b31-b123-545551673476-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a4af0cd-c986-4b31-b123-545551673476-istio-data\") pod \"5a4af0cd-c986-4b31-b123-545551673476\" (UID: \"5a4af0cd-c986-4b31-b123-545551673476\") " Oct 30 17:21:56.308342 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:56.308305 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a4af0cd-c986-4b31-b123-545551673476-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a4af0cd-c986-4b31-b123-545551673476-istio-envoy\") pod \"5a4af0cd-c986-4b31-b123-545551673476\" (UID: \"5a4af0cd-c986-4b31-b123-545551673476\") " Oct 30 17:21:56.308427 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:56.308411 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a4af0cd-c986-4b31-b123-545551673476-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a4af0cd-c986-4b31-b123-545551673476-credential-socket\") pod \"5a4af0cd-c986-4b31-b123-545551673476\" (UID: \"5a4af0cd-c986-4b31-b123-545551673476\") " Oct 30 17:21:56.308491 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:56.308480 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a4af0cd-c986-4b31-b123-545551673476-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a4af0cd-c986-4b31-b123-545551673476-workload-socket\") pod \"5a4af0cd-c986-4b31-b123-545551673476\" (UID: \"5a4af0cd-c986-4b31-b123-545551673476\") " Oct 30 17:21:56.308551 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:56.308537 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5a4af0cd-c986-4b31-b123-545551673476-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a4af0cd-c986-4b31-b123-545551673476-istio-podinfo\") pod \"5a4af0cd-c986-4b31-b123-545551673476\" (UID: \"5a4af0cd-c986-4b31-b123-545551673476\") " Oct 30 17:21:56.308631 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:56.308617 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a4af0cd-c986-4b31-b123-545551673476-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a4af0cd-c986-4b31-b123-545551673476-istiod-ca-cert\") pod \"5a4af0cd-c986-4b31-b123-545551673476\" (UID: \"5a4af0cd-c986-4b31-b123-545551673476\") " Oct 30 17:21:56.308688 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:56.308674 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a4af0cd-c986-4b31-b123-545551673476-kube-api-access-5txdt\" (UniqueName: \"kubernetes.io/projected/5a4af0cd-c986-4b31-b123-545551673476-kube-api-access-5txdt\") pod \"5a4af0cd-c986-4b31-b123-545551673476\" (UID: \"5a4af0cd-c986-4b31-b123-545551673476\") " Oct 30 17:21:56.308741 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:56.308707 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a4af0cd-c986-4b31-b123-545551673476-istio-token\" (UniqueName: \"kubernetes.io/projected/5a4af0cd-c986-4b31-b123-545551673476-istio-token\") pod \"5a4af0cd-c986-4b31-b123-545551673476\" (UID: \"5a4af0cd-c986-4b31-b123-545551673476\") " Oct 30 17:21:56.310159 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:56.310130 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a4af0cd-c986-4b31-b123-545551673476-credential-socket" pod "5a4af0cd-c986-4b31-b123-545551673476" (UID: "5a4af0cd-c986-4b31-b123-545551673476"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:56.310331 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:56.310222 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a4af0cd-c986-4b31-b123-545551673476-istio-data" pod "5a4af0cd-c986-4b31-b123-545551673476" (UID: "5a4af0cd-c986-4b31-b123-545551673476"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:56.310387 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:56.310362 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a4af0cd-c986-4b31-b123-545551673476-workload-certs" pod "5a4af0cd-c986-4b31-b123-545551673476" (UID: "5a4af0cd-c986-4b31-b123-545551673476"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:56.312853 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:56.311330 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a4af0cd-c986-4b31-b123-545551673476-workload-socket" pod "5a4af0cd-c986-4b31-b123-545551673476" (UID: "5a4af0cd-c986-4b31-b123-545551673476"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:56.312853 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:56.311918 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a4af0cd-c986-4b31-b123-545551673476-istiod-ca-cert" pod "5a4af0cd-c986-4b31-b123-545551673476" (UID: "5a4af0cd-c986-4b31-b123-545551673476"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:21:56.313080 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:56.313055 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a4af0cd-c986-4b31-b123-545551673476-istio-envoy" pod "5a4af0cd-c986-4b31-b123-545551673476" (UID: "5a4af0cd-c986-4b31-b123-545551673476"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:56.316240 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:56.315528 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a4af0cd-c986-4b31-b123-545551673476-istio-podinfo" pod "5a4af0cd-c986-4b31-b123-545551673476" (UID: "5a4af0cd-c986-4b31-b123-545551673476"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:21:56.316240 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:56.315765 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a4af0cd-c986-4b31-b123-545551673476-kube-api-access-5txdt" pod "5a4af0cd-c986-4b31-b123-545551673476" (UID: "5a4af0cd-c986-4b31-b123-545551673476"). InnerVolumeSpecName "kube-api-access-5txdt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:21:56.318831 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:56.318572 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a4af0cd-c986-4b31-b123-545551673476-istio-token" pod "5a4af0cd-c986-4b31-b123-545551673476" (UID: "5a4af0cd-c986-4b31-b123-545551673476"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:21:56.326819 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:56.326756 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8" Oct 30 17:21:56.410731 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:56.410690 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/56700de4-8e2f-4ede-b163-f152add8e6fa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56700de4-8e2f-4ede-b163-f152add8e6fa-istio-podinfo\") pod \"56700de4-8e2f-4ede-b163-f152add8e6fa\" (UID: \"56700de4-8e2f-4ede-b163-f152add8e6fa\") " Oct 30 17:21:56.411402 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:56.411374 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56700de4-8e2f-4ede-b163-f152add8e6fa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56700de4-8e2f-4ede-b163-f152add8e6fa-istio-data\") pod \"56700de4-8e2f-4ede-b163-f152add8e6fa\" (UID: \"56700de4-8e2f-4ede-b163-f152add8e6fa\") " Oct 30 17:21:56.411562 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:56.411499 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56700de4-8e2f-4ede-b163-f152add8e6fa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56700de4-8e2f-4ede-b163-f152add8e6fa-workload-socket\") pod \"56700de4-8e2f-4ede-b163-f152add8e6fa\" (UID: \"56700de4-8e2f-4ede-b163-f152add8e6fa\") " Oct 30 17:21:56.411629 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:56.411588 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56700de4-8e2f-4ede-b163-f152add8e6fa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56700de4-8e2f-4ede-b163-f152add8e6fa-workload-certs\") pod \"56700de4-8e2f-4ede-b163-f152add8e6fa\" (UID: \"56700de4-8e2f-4ede-b163-f152add8e6fa\") " Oct 30 17:21:56.411629 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:56.411616 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56700de4-8e2f-4ede-b163-f152add8e6fa-istio-token\" (UniqueName: \"kubernetes.io/projected/56700de4-8e2f-4ede-b163-f152add8e6fa-istio-token\") pod \"56700de4-8e2f-4ede-b163-f152add8e6fa\" (UID: \"56700de4-8e2f-4ede-b163-f152add8e6fa\") " Oct 30 17:21:56.411753 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:56.411673 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56700de4-8e2f-4ede-b163-f152add8e6fa-kube-api-access-dktss\" (UniqueName: \"kubernetes.io/projected/56700de4-8e2f-4ede-b163-f152add8e6fa-kube-api-access-dktss\") pod \"56700de4-8e2f-4ede-b163-f152add8e6fa\" (UID: \"56700de4-8e2f-4ede-b163-f152add8e6fa\") " Oct 30 17:21:56.411753 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:56.411742 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56700de4-8e2f-4ede-b163-f152add8e6fa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56700de4-8e2f-4ede-b163-f152add8e6fa-credential-socket\") pod \"56700de4-8e2f-4ede-b163-f152add8e6fa\" (UID: \"56700de4-8e2f-4ede-b163-f152add8e6fa\") " Oct 30 17:21:56.412253 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:56.411916 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56700de4-8e2f-4ede-b163-f152add8e6fa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56700de4-8e2f-4ede-b163-f152add8e6fa-istio-envoy\") pod \"56700de4-8e2f-4ede-b163-f152add8e6fa\" (UID: \"56700de4-8e2f-4ede-b163-f152add8e6fa\") " Oct 30 17:21:56.412253 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:56.411973 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56700de4-8e2f-4ede-b163-f152add8e6fa-workload-socket" pod "56700de4-8e2f-4ede-b163-f152add8e6fa" (UID: "56700de4-8e2f-4ede-b163-f152add8e6fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:56.412253 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:56.411987 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56700de4-8e2f-4ede-b163-f152add8e6fa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56700de4-8e2f-4ede-b163-f152add8e6fa-istiod-ca-cert\") pod \"56700de4-8e2f-4ede-b163-f152add8e6fa\" (UID: \"56700de4-8e2f-4ede-b163-f152add8e6fa\") " Oct 30 17:21:56.413116 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:56.412278 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56700de4-8e2f-4ede-b163-f152add8e6fa-istio-data" pod "56700de4-8e2f-4ede-b163-f152add8e6fa" (UID: "56700de4-8e2f-4ede-b163-f152add8e6fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:56.413116 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:56.412454 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56700de4-8e2f-4ede-b163-f152add8e6fa-workload-certs" pod "56700de4-8e2f-4ede-b163-f152add8e6fa" (UID: "56700de4-8e2f-4ede-b163-f152add8e6fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:56.413116 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:56.412698 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a4af0cd-c986-4b31-b123-545551673476-istio-envoy\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:21:56.413116 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:56.412717 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a4af0cd-c986-4b31-b123-545551673476-credential-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:21:56.413116 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:56.412731 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a4af0cd-c986-4b31-b123-545551673476-workload-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:21:56.413116 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:56.412744 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a4af0cd-c986-4b31-b123-545551673476-istio-podinfo\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:21:56.413116 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:56.412759 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56700de4-8e2f-4ede-b163-f152add8e6fa-istio-data\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:21:56.413116 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:56.412773 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a4af0cd-c986-4b31-b123-545551673476-istiod-ca-cert\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:21:56.413116 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:56.412803 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5txdt\" (UniqueName: \"kubernetes.io/projected/5a4af0cd-c986-4b31-b123-545551673476-kube-api-access-5txdt\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:21:56.413116 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:56.412816 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a4af0cd-c986-4b31-b123-545551673476-istio-token\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:21:56.413116 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:56.412830 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a4af0cd-c986-4b31-b123-545551673476-workload-certs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:21:56.413116 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:56.412843 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56700de4-8e2f-4ede-b163-f152add8e6fa-workload-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:21:56.413116 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:56.412855 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a4af0cd-c986-4b31-b123-545551673476-istio-data\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:21:56.413116 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:56.412867 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56700de4-8e2f-4ede-b163-f152add8e6fa-workload-certs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:21:56.413116 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:56.413071 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56700de4-8e2f-4ede-b163-f152add8e6fa-credential-socket" pod "56700de4-8e2f-4ede-b163-f152add8e6fa" (UID: "56700de4-8e2f-4ede-b163-f152add8e6fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:56.414528 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:56.413340 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56700de4-8e2f-4ede-b163-f152add8e6fa-istiod-ca-cert" pod "56700de4-8e2f-4ede-b163-f152add8e6fa" (UID: "56700de4-8e2f-4ede-b163-f152add8e6fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:21:56.416150 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:56.416123 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56700de4-8e2f-4ede-b163-f152add8e6fa-kube-api-access-dktss" pod "56700de4-8e2f-4ede-b163-f152add8e6fa" (UID: "56700de4-8e2f-4ede-b163-f152add8e6fa"). InnerVolumeSpecName "kube-api-access-dktss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:21:56.416777 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:56.416752 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56700de4-8e2f-4ede-b163-f152add8e6fa-istio-envoy" pod "56700de4-8e2f-4ede-b163-f152add8e6fa" (UID: "56700de4-8e2f-4ede-b163-f152add8e6fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:56.417200 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:56.417181 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56700de4-8e2f-4ede-b163-f152add8e6fa-istio-token" pod "56700de4-8e2f-4ede-b163-f152add8e6fa" (UID: "56700de4-8e2f-4ede-b163-f152add8e6fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:21:56.418104 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:56.418053 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/56700de4-8e2f-4ede-b163-f152add8e6fa-istio-podinfo" pod "56700de4-8e2f-4ede-b163-f152add8e6fa" (UID: "56700de4-8e2f-4ede-b163-f152add8e6fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:21:56.514643 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:56.514606 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56700de4-8e2f-4ede-b163-f152add8e6fa-istio-token\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:21:56.514643 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:56.514643 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dktss\" (UniqueName: \"kubernetes.io/projected/56700de4-8e2f-4ede-b163-f152add8e6fa-kube-api-access-dktss\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:21:56.514935 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:56.514657 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56700de4-8e2f-4ede-b163-f152add8e6fa-credential-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:21:56.514935 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:56.514669 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56700de4-8e2f-4ede-b163-f152add8e6fa-istio-envoy\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:21:56.514935 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:56.514680 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56700de4-8e2f-4ede-b163-f152add8e6fa-istiod-ca-cert\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:21:56.514935 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:56.514690 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56700de4-8e2f-4ede-b163-f152add8e6fa-istio-podinfo\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:21:56.558095 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:56.558013 4493 generic.go:356] "Generic (PLEG): container finished" podID="60f8eca9-6103-45f2-809e-cf45f72a7d5b" containerID="60a32d3a3f764df7376ec313753e3141af20e73724baea11cb6a4e6f215178f5" exitCode=0 Oct 30 17:21:56.558489 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:56.558096 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-abda73-d96hh" event={"ID":"60f8eca9-6103-45f2-809e-cf45f72a7d5b","Type":"ContainerDied","Data":"60a32d3a3f764df7376ec313753e3141af20e73724baea11cb6a4e6f215178f5"} Oct 30 17:21:56.559647 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:56.559620 4493 generic.go:356] "Generic (PLEG): container finished" podID="56700de4-8e2f-4ede-b163-f152add8e6fa" containerID="40aea16c219909e45a0fcac30a7f966c94b874ae718a7470de494597b5afbb69" exitCode=0 Oct 30 17:21:56.559763 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:56.559663 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8" Oct 30 17:21:56.559763 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:56.559705 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8" event={"ID":"56700de4-8e2f-4ede-b163-f152add8e6fa","Type":"ContainerDied","Data":"40aea16c219909e45a0fcac30a7f966c94b874ae718a7470de494597b5afbb69"} Oct 30 17:21:56.559763 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:56.559751 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8" event={"ID":"56700de4-8e2f-4ede-b163-f152add8e6fa","Type":"ContainerDied","Data":"e7e56ee69852de55569a30b6efe3328fa8bb32916ac15e7dbdabb5a2608e1ad4"} Oct 30 17:21:56.559925 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:56.559779 4493 scope.go:122] "RemoveContainer" containerID="40aea16c219909e45a0fcac30a7f966c94b874ae718a7470de494597b5afbb69" Oct 30 17:21:56.567129 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:56.567102 4493 generic.go:356] "Generic (PLEG): container finished" podID="5a4af0cd-c986-4b31-b123-545551673476" containerID="aa7da326256c6a80b6c838b35b061c7ac6712e8ff7a185f4408c6392edcaed72" exitCode=0 Oct 30 17:21:56.567238 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:56.567139 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g" Oct 30 17:21:56.567238 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:56.567171 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g" event={"ID":"5a4af0cd-c986-4b31-b123-545551673476","Type":"ContainerDied","Data":"aa7da326256c6a80b6c838b35b061c7ac6712e8ff7a185f4408c6392edcaed72"} Oct 30 17:21:56.567238 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:56.567213 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g" event={"ID":"5a4af0cd-c986-4b31-b123-545551673476","Type":"ContainerDied","Data":"b3b680da4a45bbdef8078b6a59fb01bf9b6652605befa87fc661941b5c5b553f"} Oct 30 17:21:56.578132 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:56.578113 4493 scope.go:122] "RemoveContainer" containerID="40aea16c219909e45a0fcac30a7f966c94b874ae718a7470de494597b5afbb69" Oct 30 17:21:56.578397 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:21:56.578377 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40aea16c219909e45a0fcac30a7f966c94b874ae718a7470de494597b5afbb69\": container with ID starting with 40aea16c219909e45a0fcac30a7f966c94b874ae718a7470de494597b5afbb69 not found: ID does not exist" containerID="40aea16c219909e45a0fcac30a7f966c94b874ae718a7470de494597b5afbb69" Oct 30 17:21:56.578452 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:56.578406 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40aea16c219909e45a0fcac30a7f966c94b874ae718a7470de494597b5afbb69"} err="failed to get container status \"40aea16c219909e45a0fcac30a7f966c94b874ae718a7470de494597b5afbb69\": rpc error: code = NotFound desc = could not find container \"40aea16c219909e45a0fcac30a7f966c94b874ae718a7470de494597b5afbb69\": container with ID starting with 40aea16c219909e45a0fcac30a7f966c94b874ae718a7470de494597b5afbb69 not found: ID does not exist" Oct 30 17:21:56.578452 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:56.578424 4493 scope.go:122] "RemoveContainer" containerID="aa7da326256c6a80b6c838b35b061c7ac6712e8ff7a185f4408c6392edcaed72" Oct 30 17:21:56.590582 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:56.590561 4493 scope.go:122] "RemoveContainer" containerID="aa7da326256c6a80b6c838b35b061c7ac6712e8ff7a185f4408c6392edcaed72" Oct 30 17:21:56.590963 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:21:56.590940 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa7da326256c6a80b6c838b35b061c7ac6712e8ff7a185f4408c6392edcaed72\": container with ID starting with aa7da326256c6a80b6c838b35b061c7ac6712e8ff7a185f4408c6392edcaed72 not found: ID does not exist" containerID="aa7da326256c6a80b6c838b35b061c7ac6712e8ff7a185f4408c6392edcaed72" Oct 30 17:21:56.591086 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:56.590968 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa7da326256c6a80b6c838b35b061c7ac6712e8ff7a185f4408c6392edcaed72"} err="failed to get container status \"aa7da326256c6a80b6c838b35b061c7ac6712e8ff7a185f4408c6392edcaed72\": rpc error: code = NotFound desc = could not find container \"aa7da326256c6a80b6c838b35b061c7ac6712e8ff7a185f4408c6392edcaed72\": container with ID starting with aa7da326256c6a80b6c838b35b061c7ac6712e8ff7a185f4408c6392edcaed72 not found: ID does not exist" Oct 30 17:21:56.636082 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:56.636055 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g"] Oct 30 17:21:56.643026 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:56.642996 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-cwb2g"] Oct 30 17:21:56.675209 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:56.675178 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8"] Oct 30 17:21:56.679037 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:56.679012 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-9s8s8"] Oct 30 17:21:56.938707 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:56.938671 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56700de4-8e2f-4ede-b163-f152add8e6fa" path="/var/lib/kubelet/pods/56700de4-8e2f-4ede-b163-f152add8e6fa/volumes" Oct 30 17:21:56.939184 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:56.939169 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a4af0cd-c986-4b31-b123-545551673476" path="/var/lib/kubelet/pods/5a4af0cd-c986-4b31-b123-545551673476/volumes" Oct 30 17:21:57.774517 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:57.774490 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-abda73-d96hh" Oct 30 17:21:57.844690 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:57.843852 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60f8eca9-6103-45f2-809e-cf45f72a7d5b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60f8eca9-6103-45f2-809e-cf45f72a7d5b-maas-api-xks-ca\") pod \"60f8eca9-6103-45f2-809e-cf45f72a7d5b\" (UID: \"60f8eca9-6103-45f2-809e-cf45f72a7d5b\") " Oct 30 17:21:57.844690 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:57.844081 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60f8eca9-6103-45f2-809e-cf45f72a7d5b-kube-api-access-q4hbk\" (UniqueName: \"kubernetes.io/projected/60f8eca9-6103-45f2-809e-cf45f72a7d5b-kube-api-access-q4hbk\") pod \"60f8eca9-6103-45f2-809e-cf45f72a7d5b\" (UID: \"60f8eca9-6103-45f2-809e-cf45f72a7d5b\") " Oct 30 17:21:57.844690 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:57.844105 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60f8eca9-6103-45f2-809e-cf45f72a7d5b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60f8eca9-6103-45f2-809e-cf45f72a7d5b-maas-api-openshift-ca\") pod \"60f8eca9-6103-45f2-809e-cf45f72a7d5b\" (UID: \"60f8eca9-6103-45f2-809e-cf45f72a7d5b\") " Oct 30 17:21:57.844690 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:57.844589 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60f8eca9-6103-45f2-809e-cf45f72a7d5b-maas-api-openshift-ca" pod "60f8eca9-6103-45f2-809e-cf45f72a7d5b" (UID: "60f8eca9-6103-45f2-809e-cf45f72a7d5b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:21:57.845240 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:57.844749 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60f8eca9-6103-45f2-809e-cf45f72a7d5b-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:21:57.846603 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:57.846535 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60f8eca9-6103-45f2-809e-cf45f72a7d5b-kube-api-access-q4hbk" pod "60f8eca9-6103-45f2-809e-cf45f72a7d5b" (UID: "60f8eca9-6103-45f2-809e-cf45f72a7d5b"). InnerVolumeSpecName "kube-api-access-q4hbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:21:57.848981 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:57.848942 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60f8eca9-6103-45f2-809e-cf45f72a7d5b-maas-api-xks-ca" pod "60f8eca9-6103-45f2-809e-cf45f72a7d5b" (UID: "60f8eca9-6103-45f2-809e-cf45f72a7d5b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:21:57.882155 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:57.882129 4493 scope.go:122] "RemoveContainer" containerID="7da8f2e9769248f89ce9b3553929211b3d55be421968d6dc744c421b35e28df2" Oct 30 17:21:57.893269 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:57.893249 4493 scope.go:122] "RemoveContainer" containerID="587806b7b54ecd6909899c6440a8f8c621517f56101aee3d94afbac812187c23" Oct 30 17:21:57.903031 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:57.903010 4493 scope.go:122] "RemoveContainer" containerID="12d03469e87b8b49b6473a3617919b907b5ea2179a56c923564f45b90a5bf025" Oct 30 17:21:57.946608 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:57.946560 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4hbk\" (UniqueName: \"kubernetes.io/projected/60f8eca9-6103-45f2-809e-cf45f72a7d5b-kube-api-access-q4hbk\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:21:57.946608 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:57.946601 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60f8eca9-6103-45f2-809e-cf45f72a7d5b-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:21:58.598302 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:58.598267 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-abda73-d96hh" Oct 30 17:21:58.598494 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:58.598307 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-abda73-d96hh" event={"ID":"60f8eca9-6103-45f2-809e-cf45f72a7d5b","Type":"ContainerDied","Data":"dfa95f7ff48a91fe968565ed815dbff6ba9535c9f709a06ddfc9eee3406bdba3"} Oct 30 17:21:58.598494 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:21:58.598357 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dfa95f7ff48a91fe968565ed815dbff6ba9535c9f709a06ddfc9eee3406bdba3" Oct 30 17:22:00.349236 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:22:00.349196 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 17:22:27.524496 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:27.524457 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 17:22:30.348312 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:22:30.348271 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 17:22:52.754687 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:52.754647 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95cca0e3-dlzp4"] Oct 30 17:22:52.755655 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:52.755634 4493 cpu_manager.go:414] "containerMap: removing container" podUID="60f8eca9-6103-45f2-809e-cf45f72a7d5b" containerName="revoke-keys" Oct 30 17:22:52.755722 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:52.755659 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60f8eca9-6103-45f2-809e-cf45f72a7d5b" containerName="revoke-keys" Oct 30 17:22:52.755722 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:52.755689 4493 cpu_manager.go:414] "containerMap: removing container" podUID="56700de4-8e2f-4ede-b163-f152add8e6fa" containerName="istio-proxy" Oct 30 17:22:52.755722 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:52.755698 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56700de4-8e2f-4ede-b163-f152add8e6fa" containerName="istio-proxy" Oct 30 17:22:52.755722 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:52.755715 4493 cpu_manager.go:414] "containerMap: removing container" podUID="5a4af0cd-c986-4b31-b123-545551673476" containerName="istio-proxy" Oct 30 17:22:52.755951 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:52.755722 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a4af0cd-c986-4b31-b123-545551673476" containerName="istio-proxy" Oct 30 17:22:52.755951 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:52.755738 4493 cpu_manager.go:414] "containerMap: removing container" podUID="66c82e49-cc58-4f4c-b090-78dbcbdff588" containerName="istio-proxy" Oct 30 17:22:52.755951 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:52.755745 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66c82e49-cc58-4f4c-b090-78dbcbdff588" containerName="istio-proxy" Oct 30 17:22:52.755951 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:52.755887 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="60f8eca9-6103-45f2-809e-cf45f72a7d5b" containerName="revoke-keys" Oct 30 17:22:52.755951 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:52.755900 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="56700de4-8e2f-4ede-b163-f152add8e6fa" containerName="istio-proxy" Oct 30 17:22:52.755951 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:52.755908 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a4af0cd-c986-4b31-b123-545551673476" containerName="istio-proxy" Oct 30 17:22:52.755951 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:52.755914 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="66c82e49-cc58-4f4c-b090-78dbcbdff588" containerName="istio-proxy" Oct 30 17:22:52.759712 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:52.759695 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95cca0e3-dlzp4" Oct 30 17:22:52.771992 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:52.771969 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dabcf42-41ef-4696-94d4-363543a18267-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95cca0e3-dlzp4\" (UID: \"1dabcf42-41ef-4696-94d4-363543a18267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95cca0e3-dlzp4" Oct 30 17:22:52.772126 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:52.772108 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42ssb\" (UniqueName: \"kubernetes.io/projected/1dabcf42-41ef-4696-94d4-363543a18267-kube-api-access-42ssb\") pod \"maas-api-revoke-sub-95cca0e3-dlzp4\" (UID: \"1dabcf42-41ef-4696-94d4-363543a18267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95cca0e3-dlzp4" Oct 30 17:22:52.772190 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:52.772180 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dabcf42-41ef-4696-94d4-363543a18267-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95cca0e3-dlzp4\" (UID: \"1dabcf42-41ef-4696-94d4-363543a18267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95cca0e3-dlzp4" Oct 30 17:22:52.778186 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:52.778160 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95cca0e3-dlzp4"] Oct 30 17:22:52.785308 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:52.785280 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:22:52.785308 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:52.785281 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:22:52.785495 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:52.785341 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:22:52.786050 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:52.786027 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 17:22:52.873674 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:52.873638 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dabcf42-41ef-4696-94d4-363543a18267-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95cca0e3-dlzp4\" (UID: \"1dabcf42-41ef-4696-94d4-363543a18267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95cca0e3-dlzp4" Oct 30 17:22:52.873869 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:52.873744 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42ssb\" (UniqueName: \"kubernetes.io/projected/1dabcf42-41ef-4696-94d4-363543a18267-kube-api-access-42ssb\") pod \"maas-api-revoke-sub-95cca0e3-dlzp4\" (UID: \"1dabcf42-41ef-4696-94d4-363543a18267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95cca0e3-dlzp4" Oct 30 17:22:52.873924 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:52.873904 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dabcf42-41ef-4696-94d4-363543a18267-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95cca0e3-dlzp4\" (UID: \"1dabcf42-41ef-4696-94d4-363543a18267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95cca0e3-dlzp4" Oct 30 17:22:52.874514 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:52.874492 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dabcf42-41ef-4696-94d4-363543a18267-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95cca0e3-dlzp4\" (UID: \"1dabcf42-41ef-4696-94d4-363543a18267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95cca0e3-dlzp4" Oct 30 17:22:52.876309 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:52.876283 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dabcf42-41ef-4696-94d4-363543a18267-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95cca0e3-dlzp4\" (UID: \"1dabcf42-41ef-4696-94d4-363543a18267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95cca0e3-dlzp4" Oct 30 17:22:52.897115 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:52.897085 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42ssb\" (UniqueName: \"kubernetes.io/projected/1dabcf42-41ef-4696-94d4-363543a18267-kube-api-access-42ssb\") pod \"maas-api-revoke-sub-95cca0e3-dlzp4\" (UID: \"1dabcf42-41ef-4696-94d4-363543a18267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95cca0e3-dlzp4" Oct 30 17:22:53.086930 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:53.086859 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 17:22:53.093934 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:53.093909 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95cca0e3-dlzp4" Oct 30 17:22:53.122192 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:53.122152 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6e52c50-9qz44"] Oct 30 17:22:53.127645 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:53.127625 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6e52c50-9qz44" Oct 30 17:22:53.150351 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:53.150325 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6e52c50-9qz44"] Oct 30 17:22:53.283016 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:53.282539 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5285914-ecf4-4ae4-b2bc-da1ae7583379-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6e52c50-9qz44\" (UID: \"b5285914-ecf4-4ae4-b2bc-da1ae7583379\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6e52c50-9qz44" Oct 30 17:22:53.283016 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:53.282636 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkggk\" (UniqueName: \"kubernetes.io/projected/b5285914-ecf4-4ae4-b2bc-da1ae7583379-kube-api-access-pkggk\") pod \"maas-api-revoke-sub-e6e52c50-9qz44\" (UID: \"b5285914-ecf4-4ae4-b2bc-da1ae7583379\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6e52c50-9qz44" Oct 30 17:22:53.283016 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:53.282726 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5285914-ecf4-4ae4-b2bc-da1ae7583379-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6e52c50-9qz44\" (UID: \"b5285914-ecf4-4ae4-b2bc-da1ae7583379\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6e52c50-9qz44" Oct 30 17:22:53.384652 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:53.384576 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5285914-ecf4-4ae4-b2bc-da1ae7583379-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6e52c50-9qz44\" (UID: \"b5285914-ecf4-4ae4-b2bc-da1ae7583379\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6e52c50-9qz44" Oct 30 17:22:53.384652 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:53.384637 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkggk\" (UniqueName: \"kubernetes.io/projected/b5285914-ecf4-4ae4-b2bc-da1ae7583379-kube-api-access-pkggk\") pod \"maas-api-revoke-sub-e6e52c50-9qz44\" (UID: \"b5285914-ecf4-4ae4-b2bc-da1ae7583379\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6e52c50-9qz44" Oct 30 17:22:53.384874 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:53.384694 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5285914-ecf4-4ae4-b2bc-da1ae7583379-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6e52c50-9qz44\" (UID: \"b5285914-ecf4-4ae4-b2bc-da1ae7583379\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6e52c50-9qz44" Oct 30 17:22:53.385374 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:53.385353 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5285914-ecf4-4ae4-b2bc-da1ae7583379-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6e52c50-9qz44\" (UID: \"b5285914-ecf4-4ae4-b2bc-da1ae7583379\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6e52c50-9qz44" Oct 30 17:22:53.387198 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:53.387177 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5285914-ecf4-4ae4-b2bc-da1ae7583379-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6e52c50-9qz44\" (UID: \"b5285914-ecf4-4ae4-b2bc-da1ae7583379\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6e52c50-9qz44" Oct 30 17:22:53.407602 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:53.407569 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkggk\" (UniqueName: \"kubernetes.io/projected/b5285914-ecf4-4ae4-b2bc-da1ae7583379-kube-api-access-pkggk\") pod \"maas-api-revoke-sub-e6e52c50-9qz44\" (UID: \"b5285914-ecf4-4ae4-b2bc-da1ae7583379\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6e52c50-9qz44" Oct 30 17:22:53.419608 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:22:53.419577 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dabcf42_41ef_4696_94d4_363543a18267.slice/crio-c8b18bc8db5f0390e3179d0c9ca9e4c374a0679cd5f10e593c73a5f7b13566e8 WatchSource:0}: Error finding container c8b18bc8db5f0390e3179d0c9ca9e4c374a0679cd5f10e593c73a5f7b13566e8: Status 404 returned error can't find the container with id c8b18bc8db5f0390e3179d0c9ca9e4c374a0679cd5f10e593c73a5f7b13566e8 Oct 30 17:22:53.419718 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:53.419615 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95cca0e3-dlzp4"] Oct 30 17:22:53.452350 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:53.452321 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6e52c50-9qz44" Oct 30 17:22:53.489476 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:53.489442 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95cca0e3-dlzp4" event={"ID":"1dabcf42-41ef-4696-94d4-363543a18267","Type":"ContainerStarted","Data":"c8b18bc8db5f0390e3179d0c9ca9e4c374a0679cd5f10e593c73a5f7b13566e8"} Oct 30 17:22:53.765375 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:22:53.765344 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5285914_ecf4_4ae4_b2bc_da1ae7583379.slice/crio-fb67937f05cc8a65f8edd4ae7f20e3547a7559f672a1aec6b27d21e6d709c8ac WatchSource:0}: Error finding container fb67937f05cc8a65f8edd4ae7f20e3547a7559f672a1aec6b27d21e6d709c8ac: Status 404 returned error can't find the container with id fb67937f05cc8a65f8edd4ae7f20e3547a7559f672a1aec6b27d21e6d709c8ac Oct 30 17:22:53.765711 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:53.765529 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6e52c50-9qz44"] Oct 30 17:22:54.509066 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:54.509023 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95cca0e3-dlzp4" event={"ID":"1dabcf42-41ef-4696-94d4-363543a18267","Type":"ContainerStarted","Data":"98f3de2c58ee2a5126ceae0a0528df90b28d7685c5f99535be8f69f0b302d918"} Oct 30 17:22:54.511168 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:54.511133 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6e52c50-9qz44" event={"ID":"b5285914-ecf4-4ae4-b2bc-da1ae7583379","Type":"ContainerStarted","Data":"5d805ec97f365170da9a04c98da5700edf300b462c207ab6d3a3b1b6403c3192"} Oct 30 17:22:54.511306 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:54.511189 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6e52c50-9qz44" event={"ID":"b5285914-ecf4-4ae4-b2bc-da1ae7583379","Type":"ContainerStarted","Data":"fb67937f05cc8a65f8edd4ae7f20e3547a7559f672a1aec6b27d21e6d709c8ac"} Oct 30 17:22:54.549915 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:54.549851 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95cca0e3-dlzp4" podStartSLOduration=2.549834082 podStartE2EDuration="2.549834082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:22:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:22:54.54867936 +0000 UTC m=+3601.775854462" watchObservedRunningTime="2026-10-30 17:22:54.549834082 +0000 UTC m=+3601.777009183" Oct 30 17:22:54.592227 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:54.592175 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6e52c50-9qz44" podStartSLOduration=1.592159436 podStartE2EDuration="1.592159436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:22:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:22:54.588241524 +0000 UTC m=+3601.815416625" watchObservedRunningTime="2026-10-30 17:22:54.592159436 +0000 UTC m=+3601.819334536" Oct 30 17:22:54.896988 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:54.896954 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb312a11-6d8r8"] Oct 30 17:22:54.901300 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:54.901277 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb312a11-6d8r8" Oct 30 17:22:54.907556 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:54.907522 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9581d15c-a121-4c2a-9d55-a2cf3afbe0b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb312a11-6d8r8\" (UID: \"9581d15c-a121-4c2a-9d55-a2cf3afbe0b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb312a11-6d8r8" Oct 30 17:22:54.907556 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:54.907568 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfbdr\" (UniqueName: \"kubernetes.io/projected/9581d15c-a121-4c2a-9d55-a2cf3afbe0b0-kube-api-access-kfbdr\") pod \"maas-api-revoke-sub-cb312a11-6d8r8\" (UID: \"9581d15c-a121-4c2a-9d55-a2cf3afbe0b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb312a11-6d8r8" Oct 30 17:22:54.907855 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:54.907831 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9581d15c-a121-4c2a-9d55-a2cf3afbe0b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb312a11-6d8r8\" (UID: \"9581d15c-a121-4c2a-9d55-a2cf3afbe0b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb312a11-6d8r8" Oct 30 17:22:54.940029 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:54.939803 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb312a11-6d8r8"] Oct 30 17:22:54.974098 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:54.974070 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-56v9s_d3a1c417-2698-4462-b7ff-1d2e66d7a10f/ovn-acl-logging/0.log" Oct 30 17:22:54.974305 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:54.974278 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-56v9s_d3a1c417-2698-4462-b7ff-1d2e66d7a10f/ovn-acl-logging/0.log" Oct 30 17:22:55.010238 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:55.010206 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9581d15c-a121-4c2a-9d55-a2cf3afbe0b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb312a11-6d8r8\" (UID: \"9581d15c-a121-4c2a-9d55-a2cf3afbe0b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb312a11-6d8r8" Oct 30 17:22:55.010431 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:55.010244 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfbdr\" (UniqueName: \"kubernetes.io/projected/9581d15c-a121-4c2a-9d55-a2cf3afbe0b0-kube-api-access-kfbdr\") pod \"maas-api-revoke-sub-cb312a11-6d8r8\" (UID: \"9581d15c-a121-4c2a-9d55-a2cf3afbe0b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb312a11-6d8r8" Oct 30 17:22:55.010431 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:55.010388 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9581d15c-a121-4c2a-9d55-a2cf3afbe0b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb312a11-6d8r8\" (UID: \"9581d15c-a121-4c2a-9d55-a2cf3afbe0b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb312a11-6d8r8" Oct 30 17:22:55.010946 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:55.010916 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9581d15c-a121-4c2a-9d55-a2cf3afbe0b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb312a11-6d8r8\" (UID: \"9581d15c-a121-4c2a-9d55-a2cf3afbe0b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb312a11-6d8r8" Oct 30 17:22:55.012970 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:55.012943 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9581d15c-a121-4c2a-9d55-a2cf3afbe0b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb312a11-6d8r8\" (UID: \"9581d15c-a121-4c2a-9d55-a2cf3afbe0b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb312a11-6d8r8" Oct 30 17:22:55.033768 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:55.033736 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfbdr\" (UniqueName: \"kubernetes.io/projected/9581d15c-a121-4c2a-9d55-a2cf3afbe0b0-kube-api-access-kfbdr\") pod \"maas-api-revoke-sub-cb312a11-6d8r8\" (UID: \"9581d15c-a121-4c2a-9d55-a2cf3afbe0b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb312a11-6d8r8" Oct 30 17:22:55.218313 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:55.218279 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb312a11-6d8r8" Oct 30 17:22:55.521145 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:55.521058 4493 generic.go:356] "Generic (PLEG): container finished" podID="b5285914-ecf4-4ae4-b2bc-da1ae7583379" containerID="5d805ec97f365170da9a04c98da5700edf300b462c207ab6d3a3b1b6403c3192" exitCode=0 Oct 30 17:22:55.521311 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:55.521143 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6e52c50-9qz44" event={"ID":"b5285914-ecf4-4ae4-b2bc-da1ae7583379","Type":"ContainerDied","Data":"5d805ec97f365170da9a04c98da5700edf300b462c207ab6d3a3b1b6403c3192"} Oct 30 17:22:55.522885 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:55.522860 4493 generic.go:356] "Generic (PLEG): container finished" podID="1dabcf42-41ef-4696-94d4-363543a18267" containerID="98f3de2c58ee2a5126ceae0a0528df90b28d7685c5f99535be8f69f0b302d918" exitCode=0 Oct 30 17:22:55.523007 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:55.522940 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95cca0e3-dlzp4" event={"ID":"1dabcf42-41ef-4696-94d4-363543a18267","Type":"ContainerDied","Data":"98f3de2c58ee2a5126ceae0a0528df90b28d7685c5f99535be8f69f0b302d918"} Oct 30 17:22:55.532403 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:55.532379 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb312a11-6d8r8"] Oct 30 17:22:55.532516 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:22:55.532491 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9581d15c_a121_4c2a_9d55_a2cf3afbe0b0.slice/crio-c2220a126c2a0340649b74a68bb3355f97b68edc96eb9ee0b8c5eff3d79fc72c WatchSource:0}: Error finding container c2220a126c2a0340649b74a68bb3355f97b68edc96eb9ee0b8c5eff3d79fc72c: Status 404 returned error can't find the container with id c2220a126c2a0340649b74a68bb3355f97b68edc96eb9ee0b8c5eff3d79fc72c Oct 30 17:22:56.552822 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:56.543927 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-abda73-d96hh"] Oct 30 17:22:56.770810 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:56.763258 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb312a11-6d8r8" event={"ID":"9581d15c-a121-4c2a-9d55-a2cf3afbe0b0","Type":"ContainerStarted","Data":"cb92fa4084d6961370fb8be9f926ec76ecc988a2fe152db6a1aa6bfe10a75551"} Oct 30 17:22:56.770810 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:56.763319 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb312a11-6d8r8" event={"ID":"9581d15c-a121-4c2a-9d55-a2cf3afbe0b0","Type":"ContainerStarted","Data":"c2220a126c2a0340649b74a68bb3355f97b68edc96eb9ee0b8c5eff3d79fc72c"} Oct 30 17:22:56.770810 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:56.767915 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-abda73-d96hh"] Oct 30 17:22:56.841074 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:56.837718 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb312a11-6d8r8" podStartSLOduration=2.837696351 podStartE2EDuration="2.837696351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:22:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:22:56.82458444 +0000 UTC m=+3604.051759542" watchObservedRunningTime="2026-10-30 17:22:56.837696351 +0000 UTC m=+3604.064871452" Oct 30 17:22:57.005814 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:57.005128 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60f8eca9-6103-45f2-809e-cf45f72a7d5b" path="/var/lib/kubelet/pods/60f8eca9-6103-45f2-809e-cf45f72a7d5b/volumes" Oct 30 17:22:57.216593 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:57.216561 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6e52c50-9qz44" Oct 30 17:22:57.270158 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:57.268453 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95cca0e3-dlzp4" Oct 30 17:22:57.355319 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:57.355283 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1dabcf42-41ef-4696-94d4-363543a18267-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dabcf42-41ef-4696-94d4-363543a18267-maas-api-openshift-ca\") pod \"1dabcf42-41ef-4696-94d4-363543a18267\" (UID: \"1dabcf42-41ef-4696-94d4-363543a18267\") " Oct 30 17:22:57.355520 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:57.355429 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5285914-ecf4-4ae4-b2bc-da1ae7583379-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5285914-ecf4-4ae4-b2bc-da1ae7583379-maas-api-openshift-ca\") pod \"b5285914-ecf4-4ae4-b2bc-da1ae7583379\" (UID: \"b5285914-ecf4-4ae4-b2bc-da1ae7583379\") " Oct 30 17:22:57.355605 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:57.355555 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5285914-ecf4-4ae4-b2bc-da1ae7583379-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5285914-ecf4-4ae4-b2bc-da1ae7583379-maas-api-xks-ca\") pod \"b5285914-ecf4-4ae4-b2bc-da1ae7583379\" (UID: \"b5285914-ecf4-4ae4-b2bc-da1ae7583379\") " Oct 30 17:22:57.355684 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:57.355613 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dabcf42-41ef-4696-94d4-363543a18267-kube-api-access-42ssb\" (UniqueName: \"kubernetes.io/projected/1dabcf42-41ef-4696-94d4-363543a18267-kube-api-access-42ssb\") pod \"1dabcf42-41ef-4696-94d4-363543a18267\" (UID: \"1dabcf42-41ef-4696-94d4-363543a18267\") " Oct 30 17:22:57.355684 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:57.355646 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5285914-ecf4-4ae4-b2bc-da1ae7583379-kube-api-access-pkggk\" (UniqueName: \"kubernetes.io/projected/b5285914-ecf4-4ae4-b2bc-da1ae7583379-kube-api-access-pkggk\") pod \"b5285914-ecf4-4ae4-b2bc-da1ae7583379\" (UID: \"b5285914-ecf4-4ae4-b2bc-da1ae7583379\") " Oct 30 17:22:57.355875 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:57.355752 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1dabcf42-41ef-4696-94d4-363543a18267-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dabcf42-41ef-4696-94d4-363543a18267-maas-api-xks-ca\") pod \"1dabcf42-41ef-4696-94d4-363543a18267\" (UID: \"1dabcf42-41ef-4696-94d4-363543a18267\") " Oct 30 17:22:57.360128 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:57.360097 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dabcf42-41ef-4696-94d4-363543a18267-maas-api-openshift-ca" pod "1dabcf42-41ef-4696-94d4-363543a18267" (UID: "1dabcf42-41ef-4696-94d4-363543a18267"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:22:57.360711 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:57.360684 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5285914-ecf4-4ae4-b2bc-da1ae7583379-maas-api-openshift-ca" pod "b5285914-ecf4-4ae4-b2bc-da1ae7583379" (UID: "b5285914-ecf4-4ae4-b2bc-da1ae7583379"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:22:57.366116 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:57.366090 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5285914-ecf4-4ae4-b2bc-da1ae7583379-kube-api-access-pkggk" pod "b5285914-ecf4-4ae4-b2bc-da1ae7583379" (UID: "b5285914-ecf4-4ae4-b2bc-da1ae7583379"). InnerVolumeSpecName "kube-api-access-pkggk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:22:57.366225 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:57.366170 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dabcf42-41ef-4696-94d4-363543a18267-kube-api-access-42ssb" pod "1dabcf42-41ef-4696-94d4-363543a18267" (UID: "1dabcf42-41ef-4696-94d4-363543a18267"). InnerVolumeSpecName "kube-api-access-42ssb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:22:57.366908 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:57.366882 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5285914-ecf4-4ae4-b2bc-da1ae7583379-maas-api-xks-ca" pod "b5285914-ecf4-4ae4-b2bc-da1ae7583379" (UID: "b5285914-ecf4-4ae4-b2bc-da1ae7583379"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:22:57.367331 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:57.367303 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1dabcf42-41ef-4696-94d4-363543a18267-maas-api-xks-ca" pod "1dabcf42-41ef-4696-94d4-363543a18267" (UID: "1dabcf42-41ef-4696-94d4-363543a18267"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:22:57.458354 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:57.458266 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42ssb\" (UniqueName: \"kubernetes.io/projected/1dabcf42-41ef-4696-94d4-363543a18267-kube-api-access-42ssb\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:22:57.458354 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:57.458296 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkggk\" (UniqueName: \"kubernetes.io/projected/b5285914-ecf4-4ae4-b2bc-da1ae7583379-kube-api-access-pkggk\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:22:57.458354 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:57.458306 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dabcf42-41ef-4696-94d4-363543a18267-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:22:57.458354 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:57.458315 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dabcf42-41ef-4696-94d4-363543a18267-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:22:57.458354 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:57.458325 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5285914-ecf4-4ae4-b2bc-da1ae7583379-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:22:57.458354 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:57.458333 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5285914-ecf4-4ae4-b2bc-da1ae7583379-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:22:57.771579 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:57.771488 4493 generic.go:356] "Generic (PLEG): container finished" podID="9581d15c-a121-4c2a-9d55-a2cf3afbe0b0" containerID="cb92fa4084d6961370fb8be9f926ec76ecc988a2fe152db6a1aa6bfe10a75551" exitCode=0 Oct 30 17:22:57.772033 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:57.771577 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb312a11-6d8r8" event={"ID":"9581d15c-a121-4c2a-9d55-a2cf3afbe0b0","Type":"ContainerDied","Data":"cb92fa4084d6961370fb8be9f926ec76ecc988a2fe152db6a1aa6bfe10a75551"} Oct 30 17:22:57.773174 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:57.773154 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6e52c50-9qz44" Oct 30 17:22:57.773296 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:57.773194 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6e52c50-9qz44" event={"ID":"b5285914-ecf4-4ae4-b2bc-da1ae7583379","Type":"ContainerDied","Data":"fb67937f05cc8a65f8edd4ae7f20e3547a7559f672a1aec6b27d21e6d709c8ac"} Oct 30 17:22:57.773296 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:57.773231 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb67937f05cc8a65f8edd4ae7f20e3547a7559f672a1aec6b27d21e6d709c8ac" Oct 30 17:22:57.774676 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:57.774658 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95cca0e3-dlzp4" Oct 30 17:22:57.774676 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:57.774663 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95cca0e3-dlzp4" event={"ID":"1dabcf42-41ef-4696-94d4-363543a18267","Type":"ContainerDied","Data":"c8b18bc8db5f0390e3179d0c9ca9e4c374a0679cd5f10e593c73a5f7b13566e8"} Oct 30 17:22:57.774855 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:57.774702 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8b18bc8db5f0390e3179d0c9ca9e4c374a0679cd5f10e593c73a5f7b13566e8" Oct 30 17:22:58.954147 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:58.954102 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb312a11-6d8r8" Oct 30 17:22:59.083886 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:59.083809 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9581d15c-a121-4c2a-9d55-a2cf3afbe0b0-kube-api-access-kfbdr\" (UniqueName: \"kubernetes.io/projected/9581d15c-a121-4c2a-9d55-a2cf3afbe0b0-kube-api-access-kfbdr\") pod \"9581d15c-a121-4c2a-9d55-a2cf3afbe0b0\" (UID: \"9581d15c-a121-4c2a-9d55-a2cf3afbe0b0\") " Oct 30 17:22:59.084058 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:59.083918 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9581d15c-a121-4c2a-9d55-a2cf3afbe0b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9581d15c-a121-4c2a-9d55-a2cf3afbe0b0-maas-api-xks-ca\") pod \"9581d15c-a121-4c2a-9d55-a2cf3afbe0b0\" (UID: \"9581d15c-a121-4c2a-9d55-a2cf3afbe0b0\") " Oct 30 17:22:59.084138 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:59.084088 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9581d15c-a121-4c2a-9d55-a2cf3afbe0b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9581d15c-a121-4c2a-9d55-a2cf3afbe0b0-maas-api-openshift-ca\") pod \"9581d15c-a121-4c2a-9d55-a2cf3afbe0b0\" (UID: \"9581d15c-a121-4c2a-9d55-a2cf3afbe0b0\") " Oct 30 17:22:59.084470 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:59.084427 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9581d15c-a121-4c2a-9d55-a2cf3afbe0b0-maas-api-openshift-ca" pod "9581d15c-a121-4c2a-9d55-a2cf3afbe0b0" (UID: "9581d15c-a121-4c2a-9d55-a2cf3afbe0b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:22:59.085130 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:59.085107 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9581d15c-a121-4c2a-9d55-a2cf3afbe0b0-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:22:59.086176 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:59.086147 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9581d15c-a121-4c2a-9d55-a2cf3afbe0b0-maas-api-xks-ca" pod "9581d15c-a121-4c2a-9d55-a2cf3afbe0b0" (UID: "9581d15c-a121-4c2a-9d55-a2cf3afbe0b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:22:59.086292 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:59.086216 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9581d15c-a121-4c2a-9d55-a2cf3afbe0b0-kube-api-access-kfbdr" pod "9581d15c-a121-4c2a-9d55-a2cf3afbe0b0" (UID: "9581d15c-a121-4c2a-9d55-a2cf3afbe0b0"). InnerVolumeSpecName "kube-api-access-kfbdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:22:59.187349 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:59.187318 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfbdr\" (UniqueName: \"kubernetes.io/projected/9581d15c-a121-4c2a-9d55-a2cf3afbe0b0-kube-api-access-kfbdr\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:22:59.187349 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:59.187345 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9581d15c-a121-4c2a-9d55-a2cf3afbe0b0-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:22:59.791756 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:59.791725 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb312a11-6d8r8" Oct 30 17:22:59.791970 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:59.791763 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb312a11-6d8r8" event={"ID":"9581d15c-a121-4c2a-9d55-a2cf3afbe0b0","Type":"ContainerDied","Data":"c2220a126c2a0340649b74a68bb3355f97b68edc96eb9ee0b8c5eff3d79fc72c"} Oct 30 17:22:59.791970 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:22:59.791853 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2220a126c2a0340649b74a68bb3355f97b68edc96eb9ee0b8c5eff3d79fc72c" Oct 30 17:23:00.399903 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:23:00.399865 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 17:23:03.729450 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:03.729403 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-6e08bd/test-model-6e08bd-kserve-79f874df7-qk2h6"] Oct 30 17:23:04.276630 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:04.276597 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f775cacd-j5hhq"] Oct 30 17:23:04.277373 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:04.277343 4493 cpu_manager.go:414] "containerMap: removing container" podUID="9581d15c-a121-4c2a-9d55-a2cf3afbe0b0" containerName="revoke-keys" Oct 30 17:23:04.277373 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:04.277365 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9581d15c-a121-4c2a-9d55-a2cf3afbe0b0" containerName="revoke-keys" Oct 30 17:23:04.277600 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:04.277394 4493 cpu_manager.go:414] "containerMap: removing container" podUID="1dabcf42-41ef-4696-94d4-363543a18267" containerName="revoke-keys" Oct 30 17:23:04.277600 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:04.277399 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dabcf42-41ef-4696-94d4-363543a18267" containerName="revoke-keys" Oct 30 17:23:04.277600 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:04.277415 4493 cpu_manager.go:414] "containerMap: removing container" podUID="b5285914-ecf4-4ae4-b2bc-da1ae7583379" containerName="revoke-keys" Oct 30 17:23:04.277600 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:04.277420 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5285914-ecf4-4ae4-b2bc-da1ae7583379" containerName="revoke-keys" Oct 30 17:23:04.277600 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:04.277542 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5285914-ecf4-4ae4-b2bc-da1ae7583379" containerName="revoke-keys" Oct 30 17:23:04.277600 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:04.277558 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="9581d15c-a121-4c2a-9d55-a2cf3afbe0b0" containerName="revoke-keys" Oct 30 17:23:04.277600 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:04.277568 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dabcf42-41ef-4696-94d4-363543a18267" containerName="revoke-keys" Oct 30 17:23:04.282099 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:04.282076 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f775cacd-j5hhq" Oct 30 17:23:04.297013 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:04.296991 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 17:23:04.297555 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:04.297535 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:23:04.302172 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:04.302150 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f775cacd-j5hhq"] Oct 30 17:23:04.388129 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:04.388098 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:23:04.388460 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:04.388443 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:23:04.436342 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:04.436303 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca45995c-3de8-46d3-9035-19ee8c3408a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f775cacd-j5hhq\" (UID: \"ca45995c-3de8-46d3-9035-19ee8c3408a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f775cacd-j5hhq" Oct 30 17:23:04.436701 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:04.436677 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r9hw\" (UniqueName: \"kubernetes.io/projected/ca45995c-3de8-46d3-9035-19ee8c3408a0-kube-api-access-6r9hw\") pod \"maas-api-revoke-sub-f775cacd-j5hhq\" (UID: \"ca45995c-3de8-46d3-9035-19ee8c3408a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f775cacd-j5hhq" Oct 30 17:23:04.436949 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:04.436931 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca45995c-3de8-46d3-9035-19ee8c3408a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f775cacd-j5hhq\" (UID: \"ca45995c-3de8-46d3-9035-19ee8c3408a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f775cacd-j5hhq" Oct 30 17:23:04.588240 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:04.588145 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca45995c-3de8-46d3-9035-19ee8c3408a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f775cacd-j5hhq\" (UID: \"ca45995c-3de8-46d3-9035-19ee8c3408a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f775cacd-j5hhq" Oct 30 17:23:04.588430 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:04.588269 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r9hw\" (UniqueName: \"kubernetes.io/projected/ca45995c-3de8-46d3-9035-19ee8c3408a0-kube-api-access-6r9hw\") pod \"maas-api-revoke-sub-f775cacd-j5hhq\" (UID: \"ca45995c-3de8-46d3-9035-19ee8c3408a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f775cacd-j5hhq" Oct 30 17:23:04.588430 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:04.588366 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca45995c-3de8-46d3-9035-19ee8c3408a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f775cacd-j5hhq\" (UID: \"ca45995c-3de8-46d3-9035-19ee8c3408a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f775cacd-j5hhq" Oct 30 17:23:04.608650 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:04.589761 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca45995c-3de8-46d3-9035-19ee8c3408a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f775cacd-j5hhq\" (UID: \"ca45995c-3de8-46d3-9035-19ee8c3408a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f775cacd-j5hhq" Oct 30 17:23:04.619835 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:04.618732 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca45995c-3de8-46d3-9035-19ee8c3408a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f775cacd-j5hhq\" (UID: \"ca45995c-3de8-46d3-9035-19ee8c3408a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f775cacd-j5hhq" Oct 30 17:23:04.623720 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:04.622980 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r9hw\" (UniqueName: \"kubernetes.io/projected/ca45995c-3de8-46d3-9035-19ee8c3408a0-kube-api-access-6r9hw\") pod \"maas-api-revoke-sub-f775cacd-j5hhq\" (UID: \"ca45995c-3de8-46d3-9035-19ee8c3408a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f775cacd-j5hhq" Oct 30 17:23:04.760861 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:04.760807 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn"] Oct 30 17:23:04.761352 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:04.761063 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn" podUID="43ebbaf3-58f1-4f88-99d2-171e4ca824c6" containerName="istio-proxy" containerID="cri-o://7e75f8583e4bd652ef609889cf82d0f2bbe06e4767db2b1226ad15bdedecee62" gracePeriod=30 Oct 30 17:23:04.810448 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:04.810406 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10ec30ee-5s8tt"] Oct 30 17:23:04.815390 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:04.815366 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ec30ee-5s8tt" Oct 30 17:23:04.838885 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:04.838730 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10ec30ee-5s8tt"] Oct 30 17:23:04.939646 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:04.939257 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f775cacd-j5hhq" Oct 30 17:23:05.045100 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:05.002647 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld9qk\" (UniqueName: \"kubernetes.io/projected/89047d9c-0062-4d18-a8d1-3e119f1c01f8-kube-api-access-ld9qk\") pod \"maas-api-revoke-sub-10ec30ee-5s8tt\" (UID: \"89047d9c-0062-4d18-a8d1-3e119f1c01f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ec30ee-5s8tt" Oct 30 17:23:05.045100 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:05.002848 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89047d9c-0062-4d18-a8d1-3e119f1c01f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10ec30ee-5s8tt\" (UID: \"89047d9c-0062-4d18-a8d1-3e119f1c01f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ec30ee-5s8tt" Oct 30 17:23:05.045100 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:05.002877 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89047d9c-0062-4d18-a8d1-3e119f1c01f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10ec30ee-5s8tt\" (UID: \"89047d9c-0062-4d18-a8d1-3e119f1c01f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ec30ee-5s8tt" Oct 30 17:23:05.128524 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:05.107840 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89047d9c-0062-4d18-a8d1-3e119f1c01f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10ec30ee-5s8tt\" (UID: \"89047d9c-0062-4d18-a8d1-3e119f1c01f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ec30ee-5s8tt" Oct 30 17:23:05.132214 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:05.132186 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89047d9c-0062-4d18-a8d1-3e119f1c01f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10ec30ee-5s8tt\" (UID: \"89047d9c-0062-4d18-a8d1-3e119f1c01f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ec30ee-5s8tt" Oct 30 17:23:05.133190 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:05.133170 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld9qk\" (UniqueName: \"kubernetes.io/projected/89047d9c-0062-4d18-a8d1-3e119f1c01f8-kube-api-access-ld9qk\") pod \"maas-api-revoke-sub-10ec30ee-5s8tt\" (UID: \"89047d9c-0062-4d18-a8d1-3e119f1c01f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ec30ee-5s8tt" Oct 30 17:23:05.133839 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:05.111314 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89047d9c-0062-4d18-a8d1-3e119f1c01f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10ec30ee-5s8tt\" (UID: \"89047d9c-0062-4d18-a8d1-3e119f1c01f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ec30ee-5s8tt" Oct 30 17:23:05.151845 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:05.141127 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89047d9c-0062-4d18-a8d1-3e119f1c01f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10ec30ee-5s8tt\" (UID: \"89047d9c-0062-4d18-a8d1-3e119f1c01f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ec30ee-5s8tt" Oct 30 17:23:05.174474 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:05.163276 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld9qk\" (UniqueName: \"kubernetes.io/projected/89047d9c-0062-4d18-a8d1-3e119f1c01f8-kube-api-access-ld9qk\") pod \"maas-api-revoke-sub-10ec30ee-5s8tt\" (UID: \"89047d9c-0062-4d18-a8d1-3e119f1c01f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ec30ee-5s8tt" Oct 30 17:23:05.350069 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:05.350035 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-dd715e-8jndg"] Oct 30 17:23:05.355978 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:05.355954 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-dd715e-8jndg" Oct 30 17:23:05.376558 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:05.376530 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-dd715e-8jndg"] Oct 30 17:23:05.437661 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:05.437575 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ec30ee-5s8tt" Oct 30 17:23:05.438134 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:05.438105 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3e59134-3d8f-4193-8499-33db2dc834f6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-dd715e-8jndg\" (UID: \"f3e59134-3d8f-4193-8499-33db2dc834f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-dd715e-8jndg" Oct 30 17:23:05.438481 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:05.438455 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdzvd\" (UniqueName: \"kubernetes.io/projected/f3e59134-3d8f-4193-8499-33db2dc834f6-kube-api-access-zdzvd\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-dd715e-8jndg\" (UID: \"f3e59134-3d8f-4193-8499-33db2dc834f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-dd715e-8jndg" Oct 30 17:23:05.438626 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:05.438602 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3e59134-3d8f-4193-8499-33db2dc834f6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-dd715e-8jndg\" (UID: \"f3e59134-3d8f-4193-8499-33db2dc834f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-dd715e-8jndg" Oct 30 17:23:05.464671 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:05.464648 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f775cacd-j5hhq"] Oct 30 17:23:05.465307 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:23:05.465281 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca45995c_3de8_46d3_9035_19ee8c3408a0.slice/crio-26c95cded0923033562e96cccc1b820b0555631c1dd2a6cb92420de1a5003100 WatchSource:0}: Error finding container 26c95cded0923033562e96cccc1b820b0555631c1dd2a6cb92420de1a5003100: Status 404 returned error can't find the container with id 26c95cded0923033562e96cccc1b820b0555631c1dd2a6cb92420de1a5003100 Oct 30 17:23:05.540438 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:05.540404 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdzvd\" (UniqueName: \"kubernetes.io/projected/f3e59134-3d8f-4193-8499-33db2dc834f6-kube-api-access-zdzvd\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-dd715e-8jndg\" (UID: \"f3e59134-3d8f-4193-8499-33db2dc834f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-dd715e-8jndg" Oct 30 17:23:05.540614 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:05.540490 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3e59134-3d8f-4193-8499-33db2dc834f6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-dd715e-8jndg\" (UID: \"f3e59134-3d8f-4193-8499-33db2dc834f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-dd715e-8jndg" Oct 30 17:23:05.540614 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:05.540547 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3e59134-3d8f-4193-8499-33db2dc834f6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-dd715e-8jndg\" (UID: \"f3e59134-3d8f-4193-8499-33db2dc834f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-dd715e-8jndg" Oct 30 17:23:05.541428 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:05.541382 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3e59134-3d8f-4193-8499-33db2dc834f6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-dd715e-8jndg\" (UID: \"f3e59134-3d8f-4193-8499-33db2dc834f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-dd715e-8jndg" Oct 30 17:23:05.604763 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:05.604736 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3e59134-3d8f-4193-8499-33db2dc834f6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-dd715e-8jndg\" (UID: \"f3e59134-3d8f-4193-8499-33db2dc834f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-dd715e-8jndg" Oct 30 17:23:05.605125 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:05.605103 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdzvd\" (UniqueName: \"kubernetes.io/projected/f3e59134-3d8f-4193-8499-33db2dc834f6-kube-api-access-zdzvd\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-dd715e-8jndg\" (UID: \"f3e59134-3d8f-4193-8499-33db2dc834f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-dd715e-8jndg" Oct 30 17:23:05.672003 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:05.671975 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-dd715e-8jndg" Oct 30 17:23:05.760289 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:05.760258 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10ec30ee-5s8tt"] Oct 30 17:23:05.989754 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:05.989730 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-dd715e-8jndg"] Oct 30 17:23:05.991091 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:23:05.991056 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3e59134_3d8f_4193_8499_33db2dc834f6.slice/crio-1489167b0d9517c569feceea8c83b9ea5f62d9cb19104dce9ee43690e47f3a7d WatchSource:0}: Error finding container 1489167b0d9517c569feceea8c83b9ea5f62d9cb19104dce9ee43690e47f3a7d: Status 404 returned error can't find the container with id 1489167b0d9517c569feceea8c83b9ea5f62d9cb19104dce9ee43690e47f3a7d Oct 30 17:23:06.070481 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:06.070446 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ec30ee-5s8tt" event={"ID":"89047d9c-0062-4d18-a8d1-3e119f1c01f8","Type":"ContainerStarted","Data":"ace56a9ac37f72bdf551ecb065c1e8a01bc7e2af4fc9cfaf6b81fb7a7f28e3c7"} Oct 30 17:23:06.072182 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:06.072156 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f775cacd-j5hhq" event={"ID":"ca45995c-3de8-46d3-9035-19ee8c3408a0","Type":"ContainerStarted","Data":"8f7a1a75b24ef405aa92fdb7c4bdff4fcff29b9e9fa73be7048933f99a6ccd3e"} Oct 30 17:23:06.072310 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:06.072197 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f775cacd-j5hhq" event={"ID":"ca45995c-3de8-46d3-9035-19ee8c3408a0","Type":"ContainerStarted","Data":"26c95cded0923033562e96cccc1b820b0555631c1dd2a6cb92420de1a5003100"} Oct 30 17:23:06.073464 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:06.073441 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-dd715e-8jndg" event={"ID":"f3e59134-3d8f-4193-8499-33db2dc834f6","Type":"ContainerStarted","Data":"1489167b0d9517c569feceea8c83b9ea5f62d9cb19104dce9ee43690e47f3a7d"} Oct 30 17:23:06.120416 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:06.120349 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f775cacd-j5hhq" podStartSLOduration=2.120330854 podStartE2EDuration="2.120330854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:23:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:23:06.11816781 +0000 UTC m=+3613.345342912" watchObservedRunningTime="2026-10-30 17:23:06.120330854 +0000 UTC m=+3613.347505961" Oct 30 17:23:07.093331 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:07.093288 4493 generic.go:356] "Generic (PLEG): container finished" podID="89047d9c-0062-4d18-a8d1-3e119f1c01f8" containerID="ed2bcc5c09009b68efad7aeb042081349233d760e8a8ddb9fbafd614807fbc77" exitCode=0 Oct 30 17:23:07.093853 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:07.093406 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ec30ee-5s8tt" event={"ID":"89047d9c-0062-4d18-a8d1-3e119f1c01f8","Type":"ContainerDied","Data":"ed2bcc5c09009b68efad7aeb042081349233d760e8a8ddb9fbafd614807fbc77"} Oct 30 17:23:07.095275 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:07.095245 4493 generic.go:356] "Generic (PLEG): container finished" podID="ca45995c-3de8-46d3-9035-19ee8c3408a0" containerID="8f7a1a75b24ef405aa92fdb7c4bdff4fcff29b9e9fa73be7048933f99a6ccd3e" exitCode=0 Oct 30 17:23:07.095513 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:07.095490 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f775cacd-j5hhq" event={"ID":"ca45995c-3de8-46d3-9035-19ee8c3408a0","Type":"ContainerDied","Data":"8f7a1a75b24ef405aa92fdb7c4bdff4fcff29b9e9fa73be7048933f99a6ccd3e"} Oct 30 17:23:07.097378 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:07.097348 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-dd715e-8jndg" event={"ID":"f3e59134-3d8f-4193-8499-33db2dc834f6","Type":"ContainerStarted","Data":"ff2288bb24c1f2814b517da918364af404918eaaba93cc6aff96d912c64e8940"} Oct 30 17:23:07.172452 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:07.172384 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-dd715e-8jndg" podStartSLOduration=2.172364935 podStartE2EDuration="2.172364935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:23:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:23:07.171944189 +0000 UTC m=+3614.399119290" watchObservedRunningTime="2026-10-30 17:23:07.172364935 +0000 UTC m=+3614.399540036" Oct 30 17:23:07.948024 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:07.947979 4493 patch_prober.go:29] interesting pod/e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:23:07.948242 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:07.948058 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn" podUID="43ebbaf3-58f1-4f88-99d2-171e4ca824c6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:23:08.108298 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:08.108261 4493 generic.go:356] "Generic (PLEG): container finished" podID="f3e59134-3d8f-4193-8499-33db2dc834f6" containerID="ff2288bb24c1f2814b517da918364af404918eaaba93cc6aff96d912c64e8940" exitCode=0 Oct 30 17:23:08.109681 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:08.109127 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-dd715e-8jndg" event={"ID":"f3e59134-3d8f-4193-8499-33db2dc834f6","Type":"ContainerDied","Data":"ff2288bb24c1f2814b517da918364af404918eaaba93cc6aff96d912c64e8940"} Oct 30 17:23:08.378097 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:08.373039 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f775cacd-j5hhq" Oct 30 17:23:08.436085 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:08.436055 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ec30ee-5s8tt" Oct 30 17:23:08.521486 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:08.521396 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89047d9c-0062-4d18-a8d1-3e119f1c01f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89047d9c-0062-4d18-a8d1-3e119f1c01f8-maas-api-openshift-ca\") pod \"89047d9c-0062-4d18-a8d1-3e119f1c01f8\" (UID: \"89047d9c-0062-4d18-a8d1-3e119f1c01f8\") " Oct 30 17:23:08.521486 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:08.521452 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca45995c-3de8-46d3-9035-19ee8c3408a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca45995c-3de8-46d3-9035-19ee8c3408a0-maas-api-openshift-ca\") pod \"ca45995c-3de8-46d3-9035-19ee8c3408a0\" (UID: \"ca45995c-3de8-46d3-9035-19ee8c3408a0\") " Oct 30 17:23:08.521714 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:08.521538 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca45995c-3de8-46d3-9035-19ee8c3408a0-kube-api-access-6r9hw\" (UniqueName: \"kubernetes.io/projected/ca45995c-3de8-46d3-9035-19ee8c3408a0-kube-api-access-6r9hw\") pod \"ca45995c-3de8-46d3-9035-19ee8c3408a0\" (UID: \"ca45995c-3de8-46d3-9035-19ee8c3408a0\") " Oct 30 17:23:08.521714 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:08.521597 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca45995c-3de8-46d3-9035-19ee8c3408a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca45995c-3de8-46d3-9035-19ee8c3408a0-maas-api-xks-ca\") pod \"ca45995c-3de8-46d3-9035-19ee8c3408a0\" (UID: \"ca45995c-3de8-46d3-9035-19ee8c3408a0\") " Oct 30 17:23:08.521714 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:08.521636 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89047d9c-0062-4d18-a8d1-3e119f1c01f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89047d9c-0062-4d18-a8d1-3e119f1c01f8-maas-api-xks-ca\") pod \"89047d9c-0062-4d18-a8d1-3e119f1c01f8\" (UID: \"89047d9c-0062-4d18-a8d1-3e119f1c01f8\") " Oct 30 17:23:08.521714 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:08.521675 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89047d9c-0062-4d18-a8d1-3e119f1c01f8-kube-api-access-ld9qk\" (UniqueName: \"kubernetes.io/projected/89047d9c-0062-4d18-a8d1-3e119f1c01f8-kube-api-access-ld9qk\") pod \"89047d9c-0062-4d18-a8d1-3e119f1c01f8\" (UID: \"89047d9c-0062-4d18-a8d1-3e119f1c01f8\") " Oct 30 17:23:08.523348 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:08.523320 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89047d9c-0062-4d18-a8d1-3e119f1c01f8-maas-api-openshift-ca" pod "89047d9c-0062-4d18-a8d1-3e119f1c01f8" (UID: "89047d9c-0062-4d18-a8d1-3e119f1c01f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:23:08.524262 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:08.524235 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca45995c-3de8-46d3-9035-19ee8c3408a0-maas-api-openshift-ca" pod "ca45995c-3de8-46d3-9035-19ee8c3408a0" (UID: "ca45995c-3de8-46d3-9035-19ee8c3408a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:23:08.525695 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:08.525662 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89047d9c-0062-4d18-a8d1-3e119f1c01f8-kube-api-access-ld9qk" pod "89047d9c-0062-4d18-a8d1-3e119f1c01f8" (UID: "89047d9c-0062-4d18-a8d1-3e119f1c01f8"). InnerVolumeSpecName "kube-api-access-ld9qk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:23:08.525985 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:08.525951 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca45995c-3de8-46d3-9035-19ee8c3408a0-kube-api-access-6r9hw" pod "ca45995c-3de8-46d3-9035-19ee8c3408a0" (UID: "ca45995c-3de8-46d3-9035-19ee8c3408a0"). InnerVolumeSpecName "kube-api-access-6r9hw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:23:08.526823 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:08.526527 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca45995c-3de8-46d3-9035-19ee8c3408a0-maas-api-xks-ca" pod "ca45995c-3de8-46d3-9035-19ee8c3408a0" (UID: "ca45995c-3de8-46d3-9035-19ee8c3408a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:23:08.527961 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:08.527931 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89047d9c-0062-4d18-a8d1-3e119f1c01f8-maas-api-xks-ca" pod "89047d9c-0062-4d18-a8d1-3e119f1c01f8" (UID: "89047d9c-0062-4d18-a8d1-3e119f1c01f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:23:08.624614 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:08.624578 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89047d9c-0062-4d18-a8d1-3e119f1c01f8-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:23:08.624614 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:08.624615 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ld9qk\" (UniqueName: \"kubernetes.io/projected/89047d9c-0062-4d18-a8d1-3e119f1c01f8-kube-api-access-ld9qk\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:23:08.624902 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:08.624630 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89047d9c-0062-4d18-a8d1-3e119f1c01f8-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:23:08.624902 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:08.624646 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca45995c-3de8-46d3-9035-19ee8c3408a0-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:23:08.624902 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:08.624662 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6r9hw\" (UniqueName: \"kubernetes.io/projected/ca45995c-3de8-46d3-9035-19ee8c3408a0-kube-api-access-6r9hw\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:23:08.624902 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:08.624676 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca45995c-3de8-46d3-9035-19ee8c3408a0-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:23:09.120222 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:09.120193 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ec30ee-5s8tt" Oct 30 17:23:09.120764 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:09.120217 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10ec30ee-5s8tt" event={"ID":"89047d9c-0062-4d18-a8d1-3e119f1c01f8","Type":"ContainerDied","Data":"ace56a9ac37f72bdf551ecb065c1e8a01bc7e2af4fc9cfaf6b81fb7a7f28e3c7"} Oct 30 17:23:09.120764 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:09.120284 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ace56a9ac37f72bdf551ecb065c1e8a01bc7e2af4fc9cfaf6b81fb7a7f28e3c7" Oct 30 17:23:09.121988 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:09.121968 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f775cacd-j5hhq" Oct 30 17:23:09.122225 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:09.122004 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f775cacd-j5hhq" event={"ID":"ca45995c-3de8-46d3-9035-19ee8c3408a0","Type":"ContainerDied","Data":"26c95cded0923033562e96cccc1b820b0555631c1dd2a6cb92420de1a5003100"} Oct 30 17:23:09.122337 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:09.122246 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26c95cded0923033562e96cccc1b820b0555631c1dd2a6cb92420de1a5003100" Oct 30 17:23:09.286294 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:09.286270 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-dd715e-8jndg" Oct 30 17:23:09.438903 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:09.438874 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3e59134-3d8f-4193-8499-33db2dc834f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3e59134-3d8f-4193-8499-33db2dc834f6-maas-api-openshift-ca\") pod \"f3e59134-3d8f-4193-8499-33db2dc834f6\" (UID: \"f3e59134-3d8f-4193-8499-33db2dc834f6\") " Oct 30 17:23:09.439127 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:09.439114 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3e59134-3d8f-4193-8499-33db2dc834f6-kube-api-access-zdzvd\" (UniqueName: \"kubernetes.io/projected/f3e59134-3d8f-4193-8499-33db2dc834f6-kube-api-access-zdzvd\") pod \"f3e59134-3d8f-4193-8499-33db2dc834f6\" (UID: \"f3e59134-3d8f-4193-8499-33db2dc834f6\") " Oct 30 17:23:09.439251 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:09.439238 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3e59134-3d8f-4193-8499-33db2dc834f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3e59134-3d8f-4193-8499-33db2dc834f6-maas-api-xks-ca\") pod \"f3e59134-3d8f-4193-8499-33db2dc834f6\" (UID: \"f3e59134-3d8f-4193-8499-33db2dc834f6\") " Oct 30 17:23:09.439373 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:09.439338 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3e59134-3d8f-4193-8499-33db2dc834f6-maas-api-openshift-ca" pod "f3e59134-3d8f-4193-8499-33db2dc834f6" (UID: "f3e59134-3d8f-4193-8499-33db2dc834f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:23:09.439940 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:09.439900 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3e59134-3d8f-4193-8499-33db2dc834f6-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:23:09.441867 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:09.441840 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3e59134-3d8f-4193-8499-33db2dc834f6-kube-api-access-zdzvd" pod "f3e59134-3d8f-4193-8499-33db2dc834f6" (UID: "f3e59134-3d8f-4193-8499-33db2dc834f6"). InnerVolumeSpecName "kube-api-access-zdzvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:23:09.441991 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:09.441969 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3e59134-3d8f-4193-8499-33db2dc834f6-maas-api-xks-ca" pod "f3e59134-3d8f-4193-8499-33db2dc834f6" (UID: "f3e59134-3d8f-4193-8499-33db2dc834f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:23:09.543870 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:09.542524 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdzvd\" (UniqueName: \"kubernetes.io/projected/f3e59134-3d8f-4193-8499-33db2dc834f6-kube-api-access-zdzvd\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:23:09.543870 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:09.542562 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3e59134-3d8f-4193-8499-33db2dc834f6-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:23:10.050566 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:10.050498 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn" Oct 30 17:23:10.145806 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:10.137489 4493 generic.go:356] "Generic (PLEG): container finished" podID="43ebbaf3-58f1-4f88-99d2-171e4ca824c6" containerID="7e75f8583e4bd652ef609889cf82d0f2bbe06e4767db2b1226ad15bdedecee62" exitCode=0 Oct 30 17:23:10.145806 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:10.137577 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn" Oct 30 17:23:10.145806 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:10.137707 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn" event={"ID":"43ebbaf3-58f1-4f88-99d2-171e4ca824c6","Type":"ContainerDied","Data":"7e75f8583e4bd652ef609889cf82d0f2bbe06e4767db2b1226ad15bdedecee62"} Oct 30 17:23:10.145806 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:10.137755 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn" event={"ID":"43ebbaf3-58f1-4f88-99d2-171e4ca824c6","Type":"ContainerDied","Data":"742910dc22cadde3d6ae915d69499747946cef283825f44c8edb4cff5af71d88"} Oct 30 17:23:10.145806 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:10.137836 4493 scope.go:122] "RemoveContainer" containerID="7e75f8583e4bd652ef609889cf82d0f2bbe06e4767db2b1226ad15bdedecee62" Oct 30 17:23:10.145806 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:10.140409 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-dd715e-8jndg" event={"ID":"f3e59134-3d8f-4193-8499-33db2dc834f6","Type":"ContainerDied","Data":"1489167b0d9517c569feceea8c83b9ea5f62d9cb19104dce9ee43690e47f3a7d"} Oct 30 17:23:10.145806 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:10.140483 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1489167b0d9517c569feceea8c83b9ea5f62d9cb19104dce9ee43690e47f3a7d" Oct 30 17:23:10.145806 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:10.140425 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-dd715e-8jndg" Oct 30 17:23:10.152312 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:10.152293 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43ebbaf3-58f1-4f88-99d2-171e4ca824c6-istio-token\" (UniqueName: \"kubernetes.io/projected/43ebbaf3-58f1-4f88-99d2-171e4ca824c6-istio-token\") pod \"43ebbaf3-58f1-4f88-99d2-171e4ca824c6\" (UID: \"43ebbaf3-58f1-4f88-99d2-171e4ca824c6\") " Oct 30 17:23:10.152406 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:10.152394 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43ebbaf3-58f1-4f88-99d2-171e4ca824c6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43ebbaf3-58f1-4f88-99d2-171e4ca824c6-workload-socket\") pod \"43ebbaf3-58f1-4f88-99d2-171e4ca824c6\" (UID: \"43ebbaf3-58f1-4f88-99d2-171e4ca824c6\") " Oct 30 17:23:10.152506 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:10.152480 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43ebbaf3-58f1-4f88-99d2-171e4ca824c6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43ebbaf3-58f1-4f88-99d2-171e4ca824c6-workload-certs\") pod \"43ebbaf3-58f1-4f88-99d2-171e4ca824c6\" (UID: \"43ebbaf3-58f1-4f88-99d2-171e4ca824c6\") " Oct 30 17:23:10.152939 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:10.152907 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43ebbaf3-58f1-4f88-99d2-171e4ca824c6-workload-certs" pod "43ebbaf3-58f1-4f88-99d2-171e4ca824c6" (UID: "43ebbaf3-58f1-4f88-99d2-171e4ca824c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:23:10.153098 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:10.153057 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43ebbaf3-58f1-4f88-99d2-171e4ca824c6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43ebbaf3-58f1-4f88-99d2-171e4ca824c6-credential-socket\") pod \"43ebbaf3-58f1-4f88-99d2-171e4ca824c6\" (UID: \"43ebbaf3-58f1-4f88-99d2-171e4ca824c6\") " Oct 30 17:23:10.153098 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:10.152922 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43ebbaf3-58f1-4f88-99d2-171e4ca824c6-workload-socket" pod "43ebbaf3-58f1-4f88-99d2-171e4ca824c6" (UID: "43ebbaf3-58f1-4f88-99d2-171e4ca824c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:23:10.153256 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:10.153109 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43ebbaf3-58f1-4f88-99d2-171e4ca824c6-kube-api-access-dgffk\" (UniqueName: \"kubernetes.io/projected/43ebbaf3-58f1-4f88-99d2-171e4ca824c6-kube-api-access-dgffk\") pod \"43ebbaf3-58f1-4f88-99d2-171e4ca824c6\" (UID: \"43ebbaf3-58f1-4f88-99d2-171e4ca824c6\") " Oct 30 17:23:10.153256 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:10.153203 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43ebbaf3-58f1-4f88-99d2-171e4ca824c6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43ebbaf3-58f1-4f88-99d2-171e4ca824c6-istio-envoy\") pod \"43ebbaf3-58f1-4f88-99d2-171e4ca824c6\" (UID: \"43ebbaf3-58f1-4f88-99d2-171e4ca824c6\") " Oct 30 17:23:10.153256 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:10.153238 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43ebbaf3-58f1-4f88-99d2-171e4ca824c6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43ebbaf3-58f1-4f88-99d2-171e4ca824c6-istiod-ca-cert\") pod \"43ebbaf3-58f1-4f88-99d2-171e4ca824c6\" (UID: \"43ebbaf3-58f1-4f88-99d2-171e4ca824c6\") " Oct 30 17:23:10.153427 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:10.153288 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43ebbaf3-58f1-4f88-99d2-171e4ca824c6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43ebbaf3-58f1-4f88-99d2-171e4ca824c6-istio-data\") pod \"43ebbaf3-58f1-4f88-99d2-171e4ca824c6\" (UID: \"43ebbaf3-58f1-4f88-99d2-171e4ca824c6\") " Oct 30 17:23:10.153427 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:10.153336 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/43ebbaf3-58f1-4f88-99d2-171e4ca824c6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43ebbaf3-58f1-4f88-99d2-171e4ca824c6-istio-podinfo\") pod \"43ebbaf3-58f1-4f88-99d2-171e4ca824c6\" (UID: \"43ebbaf3-58f1-4f88-99d2-171e4ca824c6\") " Oct 30 17:23:10.153427 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:10.153338 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43ebbaf3-58f1-4f88-99d2-171e4ca824c6-credential-socket" pod "43ebbaf3-58f1-4f88-99d2-171e4ca824c6" (UID: "43ebbaf3-58f1-4f88-99d2-171e4ca824c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:23:10.154396 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:10.153991 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43ebbaf3-58f1-4f88-99d2-171e4ca824c6-istiod-ca-cert" pod "43ebbaf3-58f1-4f88-99d2-171e4ca824c6" (UID: "43ebbaf3-58f1-4f88-99d2-171e4ca824c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:23:10.154396 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:10.154082 4493 scope.go:122] "RemoveContainer" containerID="fdb775980abbbda51861f2242ffb1f4e8423cd2cc5e89b2aceeadc035555db07" Oct 30 17:23:10.155271 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:10.154596 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43ebbaf3-58f1-4f88-99d2-171e4ca824c6-istio-data" pod "43ebbaf3-58f1-4f88-99d2-171e4ca824c6" (UID: "43ebbaf3-58f1-4f88-99d2-171e4ca824c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:23:10.156305 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:10.156279 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43ebbaf3-58f1-4f88-99d2-171e4ca824c6-workload-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:23:10.156414 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:10.156309 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43ebbaf3-58f1-4f88-99d2-171e4ca824c6-workload-certs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:23:10.156414 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:10.156325 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43ebbaf3-58f1-4f88-99d2-171e4ca824c6-credential-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:23:10.156414 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:10.156349 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43ebbaf3-58f1-4f88-99d2-171e4ca824c6-istiod-ca-cert\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:23:10.156414 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:10.156363 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43ebbaf3-58f1-4f88-99d2-171e4ca824c6-istio-data\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:23:10.156826 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:10.156754 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43ebbaf3-58f1-4f88-99d2-171e4ca824c6-istio-token" pod "43ebbaf3-58f1-4f88-99d2-171e4ca824c6" (UID: "43ebbaf3-58f1-4f88-99d2-171e4ca824c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:23:10.157619 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:10.157598 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43ebbaf3-58f1-4f88-99d2-171e4ca824c6-kube-api-access-dgffk" pod "43ebbaf3-58f1-4f88-99d2-171e4ca824c6" (UID: "43ebbaf3-58f1-4f88-99d2-171e4ca824c6"). InnerVolumeSpecName "kube-api-access-dgffk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:23:10.157711 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:10.157677 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/43ebbaf3-58f1-4f88-99d2-171e4ca824c6-istio-podinfo" pod "43ebbaf3-58f1-4f88-99d2-171e4ca824c6" (UID: "43ebbaf3-58f1-4f88-99d2-171e4ca824c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:23:10.158003 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:10.157982 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43ebbaf3-58f1-4f88-99d2-171e4ca824c6-istio-envoy" pod "43ebbaf3-58f1-4f88-99d2-171e4ca824c6" (UID: "43ebbaf3-58f1-4f88-99d2-171e4ca824c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:23:10.169401 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:10.169379 4493 scope.go:122] "RemoveContainer" containerID="7e75f8583e4bd652ef609889cf82d0f2bbe06e4767db2b1226ad15bdedecee62" Oct 30 17:23:10.169717 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:23:10.169696 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e75f8583e4bd652ef609889cf82d0f2bbe06e4767db2b1226ad15bdedecee62\": container with ID starting with 7e75f8583e4bd652ef609889cf82d0f2bbe06e4767db2b1226ad15bdedecee62 not found: ID does not exist" containerID="7e75f8583e4bd652ef609889cf82d0f2bbe06e4767db2b1226ad15bdedecee62" Oct 30 17:23:10.169864 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:10.169722 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e75f8583e4bd652ef609889cf82d0f2bbe06e4767db2b1226ad15bdedecee62"} err="failed to get container status \"7e75f8583e4bd652ef609889cf82d0f2bbe06e4767db2b1226ad15bdedecee62\": rpc error: code = NotFound desc = could not find container \"7e75f8583e4bd652ef609889cf82d0f2bbe06e4767db2b1226ad15bdedecee62\": container with ID starting with 7e75f8583e4bd652ef609889cf82d0f2bbe06e4767db2b1226ad15bdedecee62 not found: ID does not exist" Oct 30 17:23:10.169864 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:10.169740 4493 scope.go:122] "RemoveContainer" containerID="fdb775980abbbda51861f2242ffb1f4e8423cd2cc5e89b2aceeadc035555db07" Oct 30 17:23:10.170038 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:23:10.170019 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdb775980abbbda51861f2242ffb1f4e8423cd2cc5e89b2aceeadc035555db07\": container with ID starting with fdb775980abbbda51861f2242ffb1f4e8423cd2cc5e89b2aceeadc035555db07 not found: ID does not exist" containerID="fdb775980abbbda51861f2242ffb1f4e8423cd2cc5e89b2aceeadc035555db07" Oct 30 17:23:10.170105 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:10.170045 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdb775980abbbda51861f2242ffb1f4e8423cd2cc5e89b2aceeadc035555db07"} err="failed to get container status \"fdb775980abbbda51861f2242ffb1f4e8423cd2cc5e89b2aceeadc035555db07\": rpc error: code = NotFound desc = could not find container \"fdb775980abbbda51861f2242ffb1f4e8423cd2cc5e89b2aceeadc035555db07\": container with ID starting with fdb775980abbbda51861f2242ffb1f4e8423cd2cc5e89b2aceeadc035555db07 not found: ID does not exist" Oct 30 17:23:10.258392 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:10.258342 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgffk\" (UniqueName: \"kubernetes.io/projected/43ebbaf3-58f1-4f88-99d2-171e4ca824c6-kube-api-access-dgffk\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:23:10.258392 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:10.258379 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43ebbaf3-58f1-4f88-99d2-171e4ca824c6-istio-envoy\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:23:10.258392 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:10.258391 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43ebbaf3-58f1-4f88-99d2-171e4ca824c6-istio-podinfo\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:23:10.258392 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:10.258399 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43ebbaf3-58f1-4f88-99d2-171e4ca824c6-istio-token\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:23:10.487413 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:10.487381 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn"] Oct 30 17:23:10.491908 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:10.491881 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-abda73-openshift-default-6698487c65-tppcn"] Oct 30 17:23:10.939002 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:10.938971 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43ebbaf3-58f1-4f88-99d2-171e4ca824c6" path="/var/lib/kubelet/pods/43ebbaf3-58f1-4f88-99d2-171e4ca824c6/volumes" Oct 30 17:23:18.439880 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:18.434680 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-85eb87/test-model-85eb87-kserve-686468c445-ssq2z"] Oct 30 17:23:18.775179 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:18.775099 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e15d1f-9pk85"] Oct 30 17:23:18.775743 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:18.775730 4493 cpu_manager.go:414] "containerMap: removing container" podUID="f3e59134-3d8f-4193-8499-33db2dc834f6" containerName="revoke-keys" Oct 30 17:23:18.775803 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:18.775745 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3e59134-3d8f-4193-8499-33db2dc834f6" containerName="revoke-keys" Oct 30 17:23:18.775803 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:18.775763 4493 cpu_manager.go:414] "containerMap: removing container" podUID="43ebbaf3-58f1-4f88-99d2-171e4ca824c6" containerName="istio-proxy" Oct 30 17:23:18.775803 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:18.775768 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43ebbaf3-58f1-4f88-99d2-171e4ca824c6" containerName="istio-proxy" Oct 30 17:23:18.775803 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:18.775794 4493 cpu_manager.go:414] "containerMap: removing container" podUID="ca45995c-3de8-46d3-9035-19ee8c3408a0" containerName="revoke-keys" Oct 30 17:23:18.775803 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:18.775800 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca45995c-3de8-46d3-9035-19ee8c3408a0" containerName="revoke-keys" Oct 30 17:23:18.775977 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:18.775814 4493 cpu_manager.go:414] "containerMap: removing container" podUID="43ebbaf3-58f1-4f88-99d2-171e4ca824c6" containerName="istio-proxy" Oct 30 17:23:18.775977 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:18.775819 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43ebbaf3-58f1-4f88-99d2-171e4ca824c6" containerName="istio-proxy" Oct 30 17:23:18.775977 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:18.775844 4493 cpu_manager.go:414] "containerMap: removing container" podUID="89047d9c-0062-4d18-a8d1-3e119f1c01f8" containerName="revoke-keys" Oct 30 17:23:18.775977 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:18.775848 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89047d9c-0062-4d18-a8d1-3e119f1c01f8" containerName="revoke-keys" Oct 30 17:23:18.775977 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:18.775952 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="43ebbaf3-58f1-4f88-99d2-171e4ca824c6" containerName="istio-proxy" Oct 30 17:23:18.775977 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:18.775964 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="89047d9c-0062-4d18-a8d1-3e119f1c01f8" containerName="revoke-keys" Oct 30 17:23:18.775977 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:18.775976 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3e59134-3d8f-4193-8499-33db2dc834f6" containerName="revoke-keys" Oct 30 17:23:18.776212 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:18.775984 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca45995c-3de8-46d3-9035-19ee8c3408a0" containerName="revoke-keys" Oct 30 17:23:18.780998 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:18.780971 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e15d1f-9pk85" Oct 30 17:23:18.797033 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:18.797005 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:23:18.797351 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:18.797322 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:23:18.797733 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:18.797714 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:23:18.802899 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:18.802874 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e15d1f-9pk85"] Oct 30 17:23:18.808904 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:18.808884 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 17:23:18.855291 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:18.855248 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpftf\" (UniqueName: \"kubernetes.io/projected/9d783148-4efc-4e09-b3d5-449262bd5647-kube-api-access-qpftf\") pod \"maas-api-revoke-keys-e2e-inf-a-e15d1f-9pk85\" (UID: \"9d783148-4efc-4e09-b3d5-449262bd5647\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e15d1f-9pk85" Oct 30 17:23:18.855474 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:18.855315 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d783148-4efc-4e09-b3d5-449262bd5647-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e15d1f-9pk85\" (UID: \"9d783148-4efc-4e09-b3d5-449262bd5647\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e15d1f-9pk85" Oct 30 17:23:18.855474 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:18.855363 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d783148-4efc-4e09-b3d5-449262bd5647-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e15d1f-9pk85\" (UID: \"9d783148-4efc-4e09-b3d5-449262bd5647\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e15d1f-9pk85" Oct 30 17:23:18.956392 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:18.956357 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d783148-4efc-4e09-b3d5-449262bd5647-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e15d1f-9pk85\" (UID: \"9d783148-4efc-4e09-b3d5-449262bd5647\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e15d1f-9pk85" Oct 30 17:23:18.956613 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:18.956423 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d783148-4efc-4e09-b3d5-449262bd5647-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e15d1f-9pk85\" (UID: \"9d783148-4efc-4e09-b3d5-449262bd5647\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e15d1f-9pk85" Oct 30 17:23:18.956738 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:18.956715 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpftf\" (UniqueName: \"kubernetes.io/projected/9d783148-4efc-4e09-b3d5-449262bd5647-kube-api-access-qpftf\") pod \"maas-api-revoke-keys-e2e-inf-a-e15d1f-9pk85\" (UID: \"9d783148-4efc-4e09-b3d5-449262bd5647\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e15d1f-9pk85" Oct 30 17:23:18.957324 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:18.957237 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d783148-4efc-4e09-b3d5-449262bd5647-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e15d1f-9pk85\" (UID: \"9d783148-4efc-4e09-b3d5-449262bd5647\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e15d1f-9pk85" Oct 30 17:23:18.960182 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:18.960150 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d783148-4efc-4e09-b3d5-449262bd5647-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e15d1f-9pk85\" (UID: \"9d783148-4efc-4e09-b3d5-449262bd5647\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e15d1f-9pk85" Oct 30 17:23:18.980178 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:18.980150 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpftf\" (UniqueName: \"kubernetes.io/projected/9d783148-4efc-4e09-b3d5-449262bd5647-kube-api-access-qpftf\") pod \"maas-api-revoke-keys-e2e-inf-a-e15d1f-9pk85\" (UID: \"9d783148-4efc-4e09-b3d5-449262bd5647\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e15d1f-9pk85" Oct 30 17:23:19.031693 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:19.031621 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-6e08bd/test-model-6e08bd-kserve-79f874df7-qk2h6" podUID="c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877" containerName="main" containerID="cri-o://5881d7dafc5e1770cf00f1105b50e3e1cfb66e30a344cbc19bd6ba3d0d8f561e" gracePeriod=45 Oct 30 17:23:19.096741 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:19.096713 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e15d1f-9pk85" Oct 30 17:23:19.230444 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:19.229955 4493 generic.go:356] "Generic (PLEG): container finished" podID="c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877" containerID="5881d7dafc5e1770cf00f1105b50e3e1cfb66e30a344cbc19bd6ba3d0d8f561e" exitCode=0 Oct 30 17:23:19.230444 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:19.230189 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6e08bd/test-model-6e08bd-kserve-79f874df7-qk2h6" event={"ID":"c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877","Type":"ContainerDied","Data":"5881d7dafc5e1770cf00f1105b50e3e1cfb66e30a344cbc19bd6ba3d0d8f561e"} Oct 30 17:23:19.381691 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:19.381668 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6e08bd/test-model-6e08bd-kserve-79f874df7-qk2h6" Oct 30 17:23:19.419503 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:19.419474 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e15d1f-9pk85"] Oct 30 17:23:19.419650 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:23:19.419593 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d783148_4efc_4e09_b3d5_449262bd5647.slice/crio-dc043ca467066becd11f880ef2b035735e1bdca2efe9df8ae9ed44eeb2d91a6a WatchSource:0}: Error finding container dc043ca467066becd11f880ef2b035735e1bdca2efe9df8ae9ed44eeb2d91a6a: Status 404 returned error can't find the container with id dc043ca467066becd11f880ef2b035735e1bdca2efe9df8ae9ed44eeb2d91a6a Oct 30 17:23:19.475190 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:19.475158 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877-tmp-dir\") pod \"c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877\" (UID: \"c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877\") " Oct 30 17:23:19.475573 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:19.475263 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877-dshm\") pod \"c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877\" (UID: \"c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877\") " Oct 30 17:23:19.475573 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:19.475308 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877-home\" (UniqueName: \"kubernetes.io/empty-dir/c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877-home\") pod \"c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877\" (UID: \"c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877\") " Oct 30 17:23:19.475573 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:19.475344 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877-model-cache\") pod \"c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877\" (UID: \"c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877\") " Oct 30 17:23:19.475573 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:19.475420 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877-tls-certs\" (UniqueName: \"kubernetes.io/secret/c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877-tls-certs\") pod \"c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877\" (UID: \"c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877\") " Oct 30 17:23:19.475573 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:19.475447 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877-tmp-dir" pod "c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877" (UID: "c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:23:19.475573 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:19.475461 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877-kube-api-access-mwjnb\" (UniqueName: \"kubernetes.io/projected/c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877-kube-api-access-mwjnb\") pod \"c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877\" (UID: \"c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877\") " Oct 30 17:23:19.475988 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:19.475668 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877-model-cache" pod "c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877" (UID: "c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:23:19.476179 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:19.476155 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877-home" pod "c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877" (UID: "c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:23:19.476447 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:19.476422 4493 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877-tmp-dir\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:23:19.476670 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:19.476450 4493 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877-home\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:23:19.476670 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:19.476466 4493 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877-model-cache\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:23:19.477841 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:19.477814 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877-dshm" pod "c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877" (UID: "c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:23:19.478778 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:19.478756 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877-tls-certs" pod "c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877" (UID: "c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:23:19.478778 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:19.478772 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877-kube-api-access-mwjnb" pod "c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877" (UID: "c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877"). InnerVolumeSpecName "kube-api-access-mwjnb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:23:19.577645 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:19.577605 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwjnb\" (UniqueName: \"kubernetes.io/projected/c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877-kube-api-access-mwjnb\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:23:19.577645 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:19.577633 4493 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877-dshm\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:23:19.577645 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:19.577643 4493 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877-tls-certs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:23:20.406082 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:20.406039 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6e08bd/test-model-6e08bd-kserve-79f874df7-qk2h6" event={"ID":"c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877","Type":"ContainerDied","Data":"0b514b9cc78b12e56346b8fa3e2cfecbd7f4f03fb4eca42b3ccbe332e44c0750"} Oct 30 17:23:20.426708 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:20.426673 4493 scope.go:122] "RemoveContainer" containerID="5881d7dafc5e1770cf00f1105b50e3e1cfb66e30a344cbc19bd6ba3d0d8f561e" Oct 30 17:23:20.427211 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:20.427187 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6e08bd/test-model-6e08bd-kserve-79f874df7-qk2h6" Oct 30 17:23:20.484588 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:20.482470 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e15d1f-9pk85" event={"ID":"9d783148-4efc-4e09-b3d5-449262bd5647","Type":"ContainerStarted","Data":"9dcff2368ec2b4ef5da5c5ce7a6d4deab48379b07e8a270d8dbb8da5e3cb4fd8"} Oct 30 17:23:20.484588 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:20.482537 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e15d1f-9pk85" event={"ID":"9d783148-4efc-4e09-b3d5-449262bd5647","Type":"ContainerStarted","Data":"dc043ca467066becd11f880ef2b035735e1bdca2efe9df8ae9ed44eeb2d91a6a"} Oct 30 17:23:20.568015 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:20.567981 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-6e08bd/test-model-6e08bd-kserve-79f874df7-qk2h6"] Oct 30 17:23:20.703811 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:20.680377 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-6e08bd/test-model-6e08bd-kserve-79f874df7-qk2h6"] Oct 30 17:23:20.720877 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:20.714620 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e15d1f-9pk85" podStartSLOduration=2.714598656 podStartE2EDuration="2.714598656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:23:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:23:20.594543597 +0000 UTC m=+3627.821718698" watchObservedRunningTime="2026-10-30 17:23:20.714598656 +0000 UTC m=+3627.941773756" Oct 30 17:23:20.858363 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:20.858330 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-dd715e-8jndg"] Oct 30 17:23:20.862231 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:20.862209 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-dd715e-8jndg"] Oct 30 17:23:20.974371 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:20.973934 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877" path="/var/lib/kubelet/pods/c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877/volumes" Oct 30 17:23:20.974672 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:20.974649 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3e59134-3d8f-4193-8499-33db2dc834f6" path="/var/lib/kubelet/pods/f3e59134-3d8f-4193-8499-33db2dc834f6/volumes" Oct 30 17:23:21.494603 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:21.494571 4493 generic.go:356] "Generic (PLEG): container finished" podID="9d783148-4efc-4e09-b3d5-449262bd5647" containerID="9dcff2368ec2b4ef5da5c5ce7a6d4deab48379b07e8a270d8dbb8da5e3cb4fd8" exitCode=0 Oct 30 17:23:21.495026 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:21.494619 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e15d1f-9pk85" event={"ID":"9d783148-4efc-4e09-b3d5-449262bd5647","Type":"ContainerDied","Data":"9dcff2368ec2b4ef5da5c5ce7a6d4deab48379b07e8a270d8dbb8da5e3cb4fd8"} Oct 30 17:23:22.680670 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:22.680638 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e15d1f-9pk85" Oct 30 17:23:22.776373 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:22.776279 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d783148-4efc-4e09-b3d5-449262bd5647-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d783148-4efc-4e09-b3d5-449262bd5647-maas-api-xks-ca\") pod \"9d783148-4efc-4e09-b3d5-449262bd5647\" (UID: \"9d783148-4efc-4e09-b3d5-449262bd5647\") " Oct 30 17:23:22.776565 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:22.776377 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d783148-4efc-4e09-b3d5-449262bd5647-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d783148-4efc-4e09-b3d5-449262bd5647-maas-api-openshift-ca\") pod \"9d783148-4efc-4e09-b3d5-449262bd5647\" (UID: \"9d783148-4efc-4e09-b3d5-449262bd5647\") " Oct 30 17:23:22.776565 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:22.776464 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d783148-4efc-4e09-b3d5-449262bd5647-kube-api-access-qpftf\" (UniqueName: \"kubernetes.io/projected/9d783148-4efc-4e09-b3d5-449262bd5647-kube-api-access-qpftf\") pod \"9d783148-4efc-4e09-b3d5-449262bd5647\" (UID: \"9d783148-4efc-4e09-b3d5-449262bd5647\") " Oct 30 17:23:22.776727 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:22.776700 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d783148-4efc-4e09-b3d5-449262bd5647-maas-api-openshift-ca" pod "9d783148-4efc-4e09-b3d5-449262bd5647" (UID: "9d783148-4efc-4e09-b3d5-449262bd5647"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:23:22.777085 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:22.777062 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d783148-4efc-4e09-b3d5-449262bd5647-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:23:22.779075 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:22.779048 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d783148-4efc-4e09-b3d5-449262bd5647-kube-api-access-qpftf" pod "9d783148-4efc-4e09-b3d5-449262bd5647" (UID: "9d783148-4efc-4e09-b3d5-449262bd5647"). InnerVolumeSpecName "kube-api-access-qpftf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:23:22.779219 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:22.779195 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d783148-4efc-4e09-b3d5-449262bd5647-maas-api-xks-ca" pod "9d783148-4efc-4e09-b3d5-449262bd5647" (UID: "9d783148-4efc-4e09-b3d5-449262bd5647"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:23:22.878213 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:22.878181 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d783148-4efc-4e09-b3d5-449262bd5647-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:23:22.878213 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:22.878211 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpftf\" (UniqueName: \"kubernetes.io/projected/9d783148-4efc-4e09-b3d5-449262bd5647-kube-api-access-qpftf\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:23:23.513139 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:23.513102 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e15d1f-9pk85" event={"ID":"9d783148-4efc-4e09-b3d5-449262bd5647","Type":"ContainerDied","Data":"dc043ca467066becd11f880ef2b035735e1bdca2efe9df8ae9ed44eeb2d91a6a"} Oct 30 17:23:23.513310 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:23.513155 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e15d1f-9pk85" Oct 30 17:23:23.513310 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:23.513164 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc043ca467066becd11f880ef2b035735e1bdca2efe9df8ae9ed44eeb2d91a6a" Oct 30 17:23:23.864222 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:23.864133 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-9afbb2/test-model-9afbb2-kserve-6c5d6c968c-sfbmg"] Oct 30 17:23:24.446316 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:24.446284 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6df86982-g995k"] Oct 30 17:23:24.447260 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:24.447238 4493 cpu_manager.go:414] "containerMap: removing container" podUID="c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877" containerName="main" Oct 30 17:23:24.447340 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:24.447263 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877" containerName="main" Oct 30 17:23:24.447340 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:24.447284 4493 cpu_manager.go:414] "containerMap: removing container" podUID="9d783148-4efc-4e09-b3d5-449262bd5647" containerName="revoke-keys" Oct 30 17:23:24.447340 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:24.447292 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d783148-4efc-4e09-b3d5-449262bd5647" containerName="revoke-keys" Oct 30 17:23:24.447444 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:24.447432 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="43ebbaf3-58f1-4f88-99d2-171e4ca824c6" containerName="istio-proxy" Oct 30 17:23:24.447478 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:24.447447 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5ac6f4d-1b9c-4c0f-a438-8647e6fd4877" containerName="main" Oct 30 17:23:24.447478 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:24.447461 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d783148-4efc-4e09-b3d5-449262bd5647" containerName="revoke-keys" Oct 30 17:23:24.451691 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:24.451672 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6df86982-g995k" Oct 30 17:23:24.468083 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:24.468058 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:23:24.468206 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:24.468140 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 17:23:24.485427 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:24.485395 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6df86982-g995k"] Oct 30 17:23:24.520136 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:24.520101 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:23:24.520302 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:24.520165 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:23:24.524134 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:24.524105 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqsgv\" (UniqueName: \"kubernetes.io/projected/3104ca8b-193a-4371-9dd4-119e7b69887a-kube-api-access-rqsgv\") pod \"maas-api-revoke-sub-6df86982-g995k\" (UID: \"3104ca8b-193a-4371-9dd4-119e7b69887a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6df86982-g995k" Oct 30 17:23:24.524344 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:24.524325 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3104ca8b-193a-4371-9dd4-119e7b69887a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6df86982-g995k\" (UID: \"3104ca8b-193a-4371-9dd4-119e7b69887a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6df86982-g995k" Oct 30 17:23:24.524461 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:24.524442 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3104ca8b-193a-4371-9dd4-119e7b69887a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6df86982-g995k\" (UID: \"3104ca8b-193a-4371-9dd4-119e7b69887a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6df86982-g995k" Oct 30 17:23:24.646022 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:24.645963 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3104ca8b-193a-4371-9dd4-119e7b69887a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6df86982-g995k\" (UID: \"3104ca8b-193a-4371-9dd4-119e7b69887a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6df86982-g995k" Oct 30 17:23:24.646303 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:24.646251 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqsgv\" (UniqueName: \"kubernetes.io/projected/3104ca8b-193a-4371-9dd4-119e7b69887a-kube-api-access-rqsgv\") pod \"maas-api-revoke-sub-6df86982-g995k\" (UID: \"3104ca8b-193a-4371-9dd4-119e7b69887a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6df86982-g995k" Oct 30 17:23:24.653864 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:24.653814 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3104ca8b-193a-4371-9dd4-119e7b69887a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6df86982-g995k\" (UID: \"3104ca8b-193a-4371-9dd4-119e7b69887a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6df86982-g995k" Oct 30 17:23:24.655654 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:24.655614 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3104ca8b-193a-4371-9dd4-119e7b69887a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6df86982-g995k\" (UID: \"3104ca8b-193a-4371-9dd4-119e7b69887a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6df86982-g995k" Oct 30 17:23:24.680045 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:24.679938 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqsgv\" (UniqueName: \"kubernetes.io/projected/3104ca8b-193a-4371-9dd4-119e7b69887a-kube-api-access-rqsgv\") pod \"maas-api-revoke-sub-6df86982-g995k\" (UID: \"3104ca8b-193a-4371-9dd4-119e7b69887a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6df86982-g995k" Oct 30 17:23:24.682778 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:24.682744 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3104ca8b-193a-4371-9dd4-119e7b69887a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6df86982-g995k\" (UID: \"3104ca8b-193a-4371-9dd4-119e7b69887a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6df86982-g995k" Oct 30 17:23:24.766403 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:24.766325 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6df86982-g995k" Oct 30 17:23:25.085910 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:25.085873 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6df86982-g995k"] Oct 30 17:23:25.086918 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:23:25.086875 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3104ca8b_193a_4371_9dd4_119e7b69887a.slice/crio-1951aca034aae4602e5251daa98fc064cb16d73574847beb810ab38d20ab4b57 WatchSource:0}: Error finding container 1951aca034aae4602e5251daa98fc064cb16d73574847beb810ab38d20ab4b57: Status 404 returned error can't find the container with id 1951aca034aae4602e5251daa98fc064cb16d73574847beb810ab38d20ab4b57 Oct 30 17:23:25.535307 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:25.535269 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6df86982-g995k" event={"ID":"3104ca8b-193a-4371-9dd4-119e7b69887a","Type":"ContainerStarted","Data":"1ba3f30f3509e86b2bfb0fade6a8a317e3831547786d509c0c6eb9d599c12a23"} Oct 30 17:23:25.535447 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:25.535372 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6df86982-g995k" event={"ID":"3104ca8b-193a-4371-9dd4-119e7b69887a","Type":"ContainerStarted","Data":"1951aca034aae4602e5251daa98fc064cb16d73574847beb810ab38d20ab4b57"} Oct 30 17:23:25.576368 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:25.576317 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6df86982-g995k" podStartSLOduration=1.57630376 podStartE2EDuration="1.57630376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:23:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:23:25.574537782 +0000 UTC m=+3632.801712882" watchObservedRunningTime="2026-10-30 17:23:25.57630376 +0000 UTC m=+3632.803478861" Oct 30 17:23:25.679809 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:25.679570 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-bbc36f/test-model-bbc36f-kserve-6c7bbb6b47-dsbs2"] Oct 30 17:23:25.952486 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:25.952453 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6e08bd-cg4q6"] Oct 30 17:23:25.958594 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:25.958558 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6e08bd-cg4q6" Oct 30 17:23:25.979704 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:25.979674 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6e08bd-cg4q6"] Oct 30 17:23:26.011513 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:26.010674 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b56c14bc-2ce9-4e40-9a24-f86e3e6f79d3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6e08bd-cg4q6\" (UID: \"b56c14bc-2ce9-4e40-9a24-f86e3e6f79d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6e08bd-cg4q6" Oct 30 17:23:26.011513 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:26.010953 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b56c14bc-2ce9-4e40-9a24-f86e3e6f79d3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6e08bd-cg4q6\" (UID: \"b56c14bc-2ce9-4e40-9a24-f86e3e6f79d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6e08bd-cg4q6" Oct 30 17:23:26.011513 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:26.011018 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz7lz\" (UniqueName: \"kubernetes.io/projected/b56c14bc-2ce9-4e40-9a24-f86e3e6f79d3-kube-api-access-xz7lz\") pod \"maas-api-revoke-keys-e2e-inf-a-6e08bd-cg4q6\" (UID: \"b56c14bc-2ce9-4e40-9a24-f86e3e6f79d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6e08bd-cg4q6" Oct 30 17:23:26.112425 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:26.112390 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b56c14bc-2ce9-4e40-9a24-f86e3e6f79d3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6e08bd-cg4q6\" (UID: \"b56c14bc-2ce9-4e40-9a24-f86e3e6f79d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6e08bd-cg4q6" Oct 30 17:23:26.113069 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:26.112470 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz7lz\" (UniqueName: \"kubernetes.io/projected/b56c14bc-2ce9-4e40-9a24-f86e3e6f79d3-kube-api-access-xz7lz\") pod \"maas-api-revoke-keys-e2e-inf-a-6e08bd-cg4q6\" (UID: \"b56c14bc-2ce9-4e40-9a24-f86e3e6f79d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6e08bd-cg4q6" Oct 30 17:23:26.113069 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:26.112602 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b56c14bc-2ce9-4e40-9a24-f86e3e6f79d3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6e08bd-cg4q6\" (UID: \"b56c14bc-2ce9-4e40-9a24-f86e3e6f79d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6e08bd-cg4q6" Oct 30 17:23:26.113468 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:26.113442 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b56c14bc-2ce9-4e40-9a24-f86e3e6f79d3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6e08bd-cg4q6\" (UID: \"b56c14bc-2ce9-4e40-9a24-f86e3e6f79d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6e08bd-cg4q6" Oct 30 17:23:26.115355 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:26.115325 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b56c14bc-2ce9-4e40-9a24-f86e3e6f79d3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6e08bd-cg4q6\" (UID: \"b56c14bc-2ce9-4e40-9a24-f86e3e6f79d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6e08bd-cg4q6" Oct 30 17:23:26.135682 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:26.135660 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz7lz\" (UniqueName: \"kubernetes.io/projected/b56c14bc-2ce9-4e40-9a24-f86e3e6f79d3-kube-api-access-xz7lz\") pod \"maas-api-revoke-keys-e2e-inf-a-6e08bd-cg4q6\" (UID: \"b56c14bc-2ce9-4e40-9a24-f86e3e6f79d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6e08bd-cg4q6" Oct 30 17:23:26.278455 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:26.278354 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6e08bd-cg4q6" Oct 30 17:23:26.605258 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:26.605159 4493 generic.go:356] "Generic (PLEG): container finished" podID="3104ca8b-193a-4371-9dd4-119e7b69887a" containerID="1ba3f30f3509e86b2bfb0fade6a8a317e3831547786d509c0c6eb9d599c12a23" exitCode=0 Oct 30 17:23:26.605609 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:26.605466 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6df86982-g995k" event={"ID":"3104ca8b-193a-4371-9dd4-119e7b69887a","Type":"ContainerDied","Data":"1ba3f30f3509e86b2bfb0fade6a8a317e3831547786d509c0c6eb9d599c12a23"} Oct 30 17:23:26.632080 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:26.632047 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6e08bd-cg4q6"] Oct 30 17:23:26.632322 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:23:26.632301 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb56c14bc_2ce9_4e40_9a24_f86e3e6f79d3.slice/crio-b7c1e3320969cd0858399b32aee82114241e8785c9a11c0bd79b4e23ab098ffe WatchSource:0}: Error finding container b7c1e3320969cd0858399b32aee82114241e8785c9a11c0bd79b4e23ab098ffe: Status 404 returned error can't find the container with id b7c1e3320969cd0858399b32aee82114241e8785c9a11c0bd79b4e23ab098ffe Oct 30 17:23:27.620185 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:27.620148 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6e08bd-cg4q6" event={"ID":"b56c14bc-2ce9-4e40-9a24-f86e3e6f79d3","Type":"ContainerStarted","Data":"451967d264159ac7d1a60996b8f2d4a97013a806c3115c7e76e180ca0cfb57df"} Oct 30 17:23:27.620649 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:27.620208 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6e08bd-cg4q6" event={"ID":"b56c14bc-2ce9-4e40-9a24-f86e3e6f79d3","Type":"ContainerStarted","Data":"b7c1e3320969cd0858399b32aee82114241e8785c9a11c0bd79b4e23ab098ffe"} Oct 30 17:23:27.661970 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:27.661917 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6e08bd-cg4q6" podStartSLOduration=2.661901679 podStartE2EDuration="2.661901679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:23:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:23:27.65921641 +0000 UTC m=+3634.886391511" watchObservedRunningTime="2026-10-30 17:23:27.661901679 +0000 UTC m=+3634.889076780" Oct 30 17:23:27.787385 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:27.787355 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6df86982-g995k" Oct 30 17:23:27.840305 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:27.840266 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3104ca8b-193a-4371-9dd4-119e7b69887a-kube-api-access-rqsgv\" (UniqueName: \"kubernetes.io/projected/3104ca8b-193a-4371-9dd4-119e7b69887a-kube-api-access-rqsgv\") pod \"3104ca8b-193a-4371-9dd4-119e7b69887a\" (UID: \"3104ca8b-193a-4371-9dd4-119e7b69887a\") " Oct 30 17:23:27.840462 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:27.840397 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3104ca8b-193a-4371-9dd4-119e7b69887a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3104ca8b-193a-4371-9dd4-119e7b69887a-maas-api-xks-ca\") pod \"3104ca8b-193a-4371-9dd4-119e7b69887a\" (UID: \"3104ca8b-193a-4371-9dd4-119e7b69887a\") " Oct 30 17:23:27.840902 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:27.840870 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3104ca8b-193a-4371-9dd4-119e7b69887a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3104ca8b-193a-4371-9dd4-119e7b69887a-maas-api-openshift-ca\") pod \"3104ca8b-193a-4371-9dd4-119e7b69887a\" (UID: \"3104ca8b-193a-4371-9dd4-119e7b69887a\") " Oct 30 17:23:27.841355 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:27.841330 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3104ca8b-193a-4371-9dd4-119e7b69887a-maas-api-openshift-ca" pod "3104ca8b-193a-4371-9dd4-119e7b69887a" (UID: "3104ca8b-193a-4371-9dd4-119e7b69887a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:23:27.841654 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:27.841606 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3104ca8b-193a-4371-9dd4-119e7b69887a-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:23:27.843228 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:27.843200 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3104ca8b-193a-4371-9dd4-119e7b69887a-kube-api-access-rqsgv" pod "3104ca8b-193a-4371-9dd4-119e7b69887a" (UID: "3104ca8b-193a-4371-9dd4-119e7b69887a"). InnerVolumeSpecName "kube-api-access-rqsgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:23:27.843399 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:27.843376 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3104ca8b-193a-4371-9dd4-119e7b69887a-maas-api-xks-ca" pod "3104ca8b-193a-4371-9dd4-119e7b69887a" (UID: "3104ca8b-193a-4371-9dd4-119e7b69887a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:23:27.943474 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:27.943442 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqsgv\" (UniqueName: \"kubernetes.io/projected/3104ca8b-193a-4371-9dd4-119e7b69887a-kube-api-access-rqsgv\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:23:27.943474 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:27.943474 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3104ca8b-193a-4371-9dd4-119e7b69887a-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:23:28.235161 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:28.235066 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn"] Oct 30 17:23:28.235354 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:28.235302 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn" podUID="94c03808-f5c9-4486-9cb9-3251d0de671d" containerName="istio-proxy" containerID="cri-o://d173fbcc27f17ac896559d111bf61a822dc3a786e1737baa35bb6a277b8db548" gracePeriod=30 Oct 30 17:23:28.641285 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:28.641248 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6df86982-g995k" event={"ID":"3104ca8b-193a-4371-9dd4-119e7b69887a","Type":"ContainerDied","Data":"1951aca034aae4602e5251daa98fc064cb16d73574847beb810ab38d20ab4b57"} Oct 30 17:23:28.641762 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:28.641313 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1951aca034aae4602e5251daa98fc064cb16d73574847beb810ab38d20ab4b57" Oct 30 17:23:28.641762 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:28.641385 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6df86982-g995k" Oct 30 17:23:28.644361 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:28.643900 4493 generic.go:356] "Generic (PLEG): container finished" podID="b56c14bc-2ce9-4e40-9a24-f86e3e6f79d3" containerID="451967d264159ac7d1a60996b8f2d4a97013a806c3115c7e76e180ca0cfb57df" exitCode=0 Oct 30 17:23:28.644361 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:28.643973 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6e08bd-cg4q6" event={"ID":"b56c14bc-2ce9-4e40-9a24-f86e3e6f79d3","Type":"ContainerDied","Data":"451967d264159ac7d1a60996b8f2d4a97013a806c3115c7e76e180ca0cfb57df"} Oct 30 17:23:28.819822 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:28.819771 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8bff6e-zps26"] Oct 30 17:23:28.820777 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:28.820756 4493 cpu_manager.go:414] "containerMap: removing container" podUID="3104ca8b-193a-4371-9dd4-119e7b69887a" containerName="revoke-keys" Oct 30 17:23:28.820777 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:28.820798 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3104ca8b-193a-4371-9dd4-119e7b69887a" containerName="revoke-keys" Oct 30 17:23:28.821050 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:28.821038 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="3104ca8b-193a-4371-9dd4-119e7b69887a" containerName="revoke-keys" Oct 30 17:23:28.825262 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:28.825240 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8bff6e-zps26" Oct 30 17:23:28.843856 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:28.843832 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8bff6e-zps26"] Oct 30 17:23:29.001590 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:29.001500 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tslj\" (UniqueName: \"kubernetes.io/projected/7521ace7-b975-4772-a31d-75b30cf36283-kube-api-access-5tslj\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-8bff6e-zps26\" (UID: \"7521ace7-b975-4772-a31d-75b30cf36283\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8bff6e-zps26" Oct 30 17:23:29.001767 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:29.001672 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7521ace7-b975-4772-a31d-75b30cf36283-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-8bff6e-zps26\" (UID: \"7521ace7-b975-4772-a31d-75b30cf36283\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8bff6e-zps26" Oct 30 17:23:29.001767 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:29.001739 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7521ace7-b975-4772-a31d-75b30cf36283-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-8bff6e-zps26\" (UID: \"7521ace7-b975-4772-a31d-75b30cf36283\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8bff6e-zps26" Oct 30 17:23:29.103495 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:29.103462 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7521ace7-b975-4772-a31d-75b30cf36283-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-8bff6e-zps26\" (UID: \"7521ace7-b975-4772-a31d-75b30cf36283\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8bff6e-zps26" Oct 30 17:23:29.104662 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:29.103710 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tslj\" (UniqueName: \"kubernetes.io/projected/7521ace7-b975-4772-a31d-75b30cf36283-kube-api-access-5tslj\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-8bff6e-zps26\" (UID: \"7521ace7-b975-4772-a31d-75b30cf36283\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8bff6e-zps26" Oct 30 17:23:29.104662 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:29.103990 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7521ace7-b975-4772-a31d-75b30cf36283-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-8bff6e-zps26\" (UID: \"7521ace7-b975-4772-a31d-75b30cf36283\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8bff6e-zps26" Oct 30 17:23:29.104662 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:29.104268 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7521ace7-b975-4772-a31d-75b30cf36283-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-8bff6e-zps26\" (UID: \"7521ace7-b975-4772-a31d-75b30cf36283\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8bff6e-zps26" Oct 30 17:23:29.106875 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:29.106846 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7521ace7-b975-4772-a31d-75b30cf36283-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-8bff6e-zps26\" (UID: \"7521ace7-b975-4772-a31d-75b30cf36283\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8bff6e-zps26" Oct 30 17:23:29.130360 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:29.130330 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tslj\" (UniqueName: \"kubernetes.io/projected/7521ace7-b975-4772-a31d-75b30cf36283-kube-api-access-5tslj\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-8bff6e-zps26\" (UID: \"7521ace7-b975-4772-a31d-75b30cf36283\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8bff6e-zps26" Oct 30 17:23:29.140420 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:29.140394 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8bff6e-zps26" Oct 30 17:23:29.474151 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:29.474116 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8bff6e-zps26"] Oct 30 17:23:29.474831 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:23:29.474771 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7521ace7_b975_4772_a31d_75b30cf36283.slice/crio-322d7e4c654205ed5e8cc2a2c46d6a8e18afd416d59c8cb289a40d714fbc090e WatchSource:0}: Error finding container 322d7e4c654205ed5e8cc2a2c46d6a8e18afd416d59c8cb289a40d714fbc090e: Status 404 returned error can't find the container with id 322d7e4c654205ed5e8cc2a2c46d6a8e18afd416d59c8cb289a40d714fbc090e Oct 30 17:23:29.656868 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:29.656825 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8bff6e-zps26" event={"ID":"7521ace7-b975-4772-a31d-75b30cf36283","Type":"ContainerStarted","Data":"322d7e4c654205ed5e8cc2a2c46d6a8e18afd416d59c8cb289a40d714fbc090e"} Oct 30 17:23:30.040094 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:30.040064 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6e08bd-cg4q6" Oct 30 17:23:30.156375 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:30.156347 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b56c14bc-2ce9-4e40-9a24-f86e3e6f79d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b56c14bc-2ce9-4e40-9a24-f86e3e6f79d3-maas-api-openshift-ca\") pod \"b56c14bc-2ce9-4e40-9a24-f86e3e6f79d3\" (UID: \"b56c14bc-2ce9-4e40-9a24-f86e3e6f79d3\") " Oct 30 17:23:30.156545 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:30.156408 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b56c14bc-2ce9-4e40-9a24-f86e3e6f79d3-kube-api-access-xz7lz\" (UniqueName: \"kubernetes.io/projected/b56c14bc-2ce9-4e40-9a24-f86e3e6f79d3-kube-api-access-xz7lz\") pod \"b56c14bc-2ce9-4e40-9a24-f86e3e6f79d3\" (UID: \"b56c14bc-2ce9-4e40-9a24-f86e3e6f79d3\") " Oct 30 17:23:30.156545 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:30.156502 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b56c14bc-2ce9-4e40-9a24-f86e3e6f79d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b56c14bc-2ce9-4e40-9a24-f86e3e6f79d3-maas-api-xks-ca\") pod \"b56c14bc-2ce9-4e40-9a24-f86e3e6f79d3\" (UID: \"b56c14bc-2ce9-4e40-9a24-f86e3e6f79d3\") " Oct 30 17:23:30.156704 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:30.156683 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b56c14bc-2ce9-4e40-9a24-f86e3e6f79d3-maas-api-openshift-ca" pod "b56c14bc-2ce9-4e40-9a24-f86e3e6f79d3" (UID: "b56c14bc-2ce9-4e40-9a24-f86e3e6f79d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:23:30.157136 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:30.157099 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b56c14bc-2ce9-4e40-9a24-f86e3e6f79d3-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:23:30.158889 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:30.158865 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b56c14bc-2ce9-4e40-9a24-f86e3e6f79d3-maas-api-xks-ca" pod "b56c14bc-2ce9-4e40-9a24-f86e3e6f79d3" (UID: "b56c14bc-2ce9-4e40-9a24-f86e3e6f79d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:23:30.158969 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:30.158897 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b56c14bc-2ce9-4e40-9a24-f86e3e6f79d3-kube-api-access-xz7lz" pod "b56c14bc-2ce9-4e40-9a24-f86e3e6f79d3" (UID: "b56c14bc-2ce9-4e40-9a24-f86e3e6f79d3"). InnerVolumeSpecName "kube-api-access-xz7lz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:23:30.258663 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:30.258630 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xz7lz\" (UniqueName: \"kubernetes.io/projected/b56c14bc-2ce9-4e40-9a24-f86e3e6f79d3-kube-api-access-xz7lz\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:23:30.258663 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:30.258662 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b56c14bc-2ce9-4e40-9a24-f86e3e6f79d3-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:23:30.344572 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:23:30.344484 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 17:23:30.682058 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:30.682027 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6e08bd-cg4q6" Oct 30 17:23:30.682058 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:30.682046 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6e08bd-cg4q6" event={"ID":"b56c14bc-2ce9-4e40-9a24-f86e3e6f79d3","Type":"ContainerDied","Data":"b7c1e3320969cd0858399b32aee82114241e8785c9a11c0bd79b4e23ab098ffe"} Oct 30 17:23:30.682607 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:30.682097 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7c1e3320969cd0858399b32aee82114241e8785c9a11c0bd79b4e23ab098ffe" Oct 30 17:23:30.683906 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:30.683880 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8bff6e-zps26" event={"ID":"7521ace7-b975-4772-a31d-75b30cf36283","Type":"ContainerStarted","Data":"ce5ad454abc286bcc0fa9d88e4739e8790bee04f3d45897d3faba7a41740a377"} Oct 30 17:23:30.728278 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:30.728236 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8bff6e-zps26" podStartSLOduration=2.72822234 podStartE2EDuration="2.72822234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:23:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:23:30.724418527 +0000 UTC m=+3637.951593629" watchObservedRunningTime="2026-10-30 17:23:30.72822234 +0000 UTC m=+3637.955397440" Oct 30 17:23:31.693998 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:31.693957 4493 generic.go:356] "Generic (PLEG): container finished" podID="7521ace7-b975-4772-a31d-75b30cf36283" containerID="ce5ad454abc286bcc0fa9d88e4739e8790bee04f3d45897d3faba7a41740a377" exitCode=0 Oct 30 17:23:31.694388 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:31.694009 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8bff6e-zps26" event={"ID":"7521ace7-b975-4772-a31d-75b30cf36283","Type":"ContainerDied","Data":"ce5ad454abc286bcc0fa9d88e4739e8790bee04f3d45897d3faba7a41740a377"} Oct 30 17:23:32.874981 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:32.874957 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8bff6e-zps26" Oct 30 17:23:33.004757 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:33.004662 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7521ace7-b975-4772-a31d-75b30cf36283-kube-api-access-5tslj\" (UniqueName: \"kubernetes.io/projected/7521ace7-b975-4772-a31d-75b30cf36283-kube-api-access-5tslj\") pod \"7521ace7-b975-4772-a31d-75b30cf36283\" (UID: \"7521ace7-b975-4772-a31d-75b30cf36283\") " Oct 30 17:23:33.004757 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:33.004730 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7521ace7-b975-4772-a31d-75b30cf36283-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7521ace7-b975-4772-a31d-75b30cf36283-maas-api-openshift-ca\") pod \"7521ace7-b975-4772-a31d-75b30cf36283\" (UID: \"7521ace7-b975-4772-a31d-75b30cf36283\") " Oct 30 17:23:33.005007 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:33.004900 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7521ace7-b975-4772-a31d-75b30cf36283-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7521ace7-b975-4772-a31d-75b30cf36283-maas-api-xks-ca\") pod \"7521ace7-b975-4772-a31d-75b30cf36283\" (UID: \"7521ace7-b975-4772-a31d-75b30cf36283\") " Oct 30 17:23:33.005449 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:33.005419 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7521ace7-b975-4772-a31d-75b30cf36283-maas-api-openshift-ca" pod "7521ace7-b975-4772-a31d-75b30cf36283" (UID: "7521ace7-b975-4772-a31d-75b30cf36283"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:23:33.005797 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:33.005759 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7521ace7-b975-4772-a31d-75b30cf36283-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:23:33.007324 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:33.007298 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7521ace7-b975-4772-a31d-75b30cf36283-kube-api-access-5tslj" pod "7521ace7-b975-4772-a31d-75b30cf36283" (UID: "7521ace7-b975-4772-a31d-75b30cf36283"). InnerVolumeSpecName "kube-api-access-5tslj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:23:33.007724 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:33.007705 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7521ace7-b975-4772-a31d-75b30cf36283-maas-api-xks-ca" pod "7521ace7-b975-4772-a31d-75b30cf36283" (UID: "7521ace7-b975-4772-a31d-75b30cf36283"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:23:33.107336 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:33.107299 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tslj\" (UniqueName: \"kubernetes.io/projected/7521ace7-b975-4772-a31d-75b30cf36283-kube-api-access-5tslj\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:23:33.107336 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:33.107334 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7521ace7-b975-4772-a31d-75b30cf36283-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:23:33.270694 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:33.270615 4493 patch_prober.go:29] interesting pod/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:23:33.270694 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:33.270678 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn" podUID="94c03808-f5c9-4486-9cb9-3251d0de671d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:23:33.469811 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:33.469735 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-85eb87/test-model-85eb87-kserve-686468c445-ssq2z" podUID="0d5eb370-685d-4d9f-b450-47d754945375" containerName="main" containerID="cri-o://28f01c79c71f88d779da209dc4badbe457a0bc7779efcd09cebffddefe51e70f" gracePeriod=45 Oct 30 17:23:33.720771 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:33.720720 4493 generic.go:356] "Generic (PLEG): container finished" podID="0d5eb370-685d-4d9f-b450-47d754945375" containerID="28f01c79c71f88d779da209dc4badbe457a0bc7779efcd09cebffddefe51e70f" exitCode=0 Oct 30 17:23:33.720979 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:33.720922 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-85eb87/test-model-85eb87-kserve-686468c445-ssq2z" event={"ID":"0d5eb370-685d-4d9f-b450-47d754945375","Type":"ContainerDied","Data":"28f01c79c71f88d779da209dc4badbe457a0bc7779efcd09cebffddefe51e70f"} Oct 30 17:23:33.722768 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:33.722735 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8bff6e-zps26" event={"ID":"7521ace7-b975-4772-a31d-75b30cf36283","Type":"ContainerDied","Data":"322d7e4c654205ed5e8cc2a2c46d6a8e18afd416d59c8cb289a40d714fbc090e"} Oct 30 17:23:33.722951 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:33.722812 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="322d7e4c654205ed5e8cc2a2c46d6a8e18afd416d59c8cb289a40d714fbc090e" Oct 30 17:23:33.722951 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:33.722886 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8bff6e-zps26" Oct 30 17:23:33.731134 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:33.729344 4493 generic.go:356] "Generic (PLEG): container finished" podID="94c03808-f5c9-4486-9cb9-3251d0de671d" containerID="d173fbcc27f17ac896559d111bf61a822dc3a786e1737baa35bb6a277b8db548" exitCode=0 Oct 30 17:23:33.731134 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:33.729463 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn" event={"ID":"94c03808-f5c9-4486-9cb9-3251d0de671d","Type":"ContainerDied","Data":"d173fbcc27f17ac896559d111bf61a822dc3a786e1737baa35bb6a277b8db548"} Oct 30 17:23:33.773017 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:33.772988 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-85eb87/test-model-85eb87-kserve-686468c445-ssq2z" Oct 30 17:23:33.818521 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:33.818483 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d5eb370-685d-4d9f-b450-47d754945375-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d5eb370-685d-4d9f-b450-47d754945375-tmp-dir\") pod \"0d5eb370-685d-4d9f-b450-47d754945375\" (UID: \"0d5eb370-685d-4d9f-b450-47d754945375\") " Oct 30 17:23:33.818770 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:33.818533 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d5eb370-685d-4d9f-b450-47d754945375-tls-certs\" (UniqueName: \"kubernetes.io/secret/0d5eb370-685d-4d9f-b450-47d754945375-tls-certs\") pod \"0d5eb370-685d-4d9f-b450-47d754945375\" (UID: \"0d5eb370-685d-4d9f-b450-47d754945375\") " Oct 30 17:23:33.818770 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:33.818584 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d5eb370-685d-4d9f-b450-47d754945375-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d5eb370-685d-4d9f-b450-47d754945375-dshm\") pod \"0d5eb370-685d-4d9f-b450-47d754945375\" (UID: \"0d5eb370-685d-4d9f-b450-47d754945375\") " Oct 30 17:23:33.818770 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:33.818617 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d5eb370-685d-4d9f-b450-47d754945375-home\" (UniqueName: \"kubernetes.io/empty-dir/0d5eb370-685d-4d9f-b450-47d754945375-home\") pod \"0d5eb370-685d-4d9f-b450-47d754945375\" (UID: \"0d5eb370-685d-4d9f-b450-47d754945375\") " Oct 30 17:23:33.819123 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:33.818836 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d5eb370-685d-4d9f-b450-47d754945375-tmp-dir" pod "0d5eb370-685d-4d9f-b450-47d754945375" (UID: "0d5eb370-685d-4d9f-b450-47d754945375"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:23:33.819123 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:33.818842 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d5eb370-685d-4d9f-b450-47d754945375-kube-api-access-s6fxg\" (UniqueName: \"kubernetes.io/projected/0d5eb370-685d-4d9f-b450-47d754945375-kube-api-access-s6fxg\") pod \"0d5eb370-685d-4d9f-b450-47d754945375\" (UID: \"0d5eb370-685d-4d9f-b450-47d754945375\") " Oct 30 17:23:33.819123 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:33.818984 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d5eb370-685d-4d9f-b450-47d754945375-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d5eb370-685d-4d9f-b450-47d754945375-model-cache\") pod \"0d5eb370-685d-4d9f-b450-47d754945375\" (UID: \"0d5eb370-685d-4d9f-b450-47d754945375\") " Oct 30 17:23:33.819748 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:33.819722 4493 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d5eb370-685d-4d9f-b450-47d754945375-tmp-dir\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:23:33.820015 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:33.819996 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d5eb370-685d-4d9f-b450-47d754945375-model-cache" pod "0d5eb370-685d-4d9f-b450-47d754945375" (UID: "0d5eb370-685d-4d9f-b450-47d754945375"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:23:33.820470 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:33.820437 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d5eb370-685d-4d9f-b450-47d754945375-home" pod "0d5eb370-685d-4d9f-b450-47d754945375" (UID: "0d5eb370-685d-4d9f-b450-47d754945375"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:23:33.822197 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:33.822174 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d5eb370-685d-4d9f-b450-47d754945375-kube-api-access-s6fxg" pod "0d5eb370-685d-4d9f-b450-47d754945375" (UID: "0d5eb370-685d-4d9f-b450-47d754945375"). InnerVolumeSpecName "kube-api-access-s6fxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:23:33.823076 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:33.823056 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d5eb370-685d-4d9f-b450-47d754945375-dshm" pod "0d5eb370-685d-4d9f-b450-47d754945375" (UID: "0d5eb370-685d-4d9f-b450-47d754945375"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:23:33.823618 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:33.823593 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d5eb370-685d-4d9f-b450-47d754945375-tls-certs" pod "0d5eb370-685d-4d9f-b450-47d754945375" (UID: "0d5eb370-685d-4d9f-b450-47d754945375"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:23:33.847243 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:33.847213 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn" Oct 30 17:23:33.922292 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:33.921244 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94c03808-f5c9-4486-9cb9-3251d0de671d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94c03808-f5c9-4486-9cb9-3251d0de671d-credential-socket\") pod \"94c03808-f5c9-4486-9cb9-3251d0de671d\" (UID: \"94c03808-f5c9-4486-9cb9-3251d0de671d\") " Oct 30 17:23:33.922292 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:33.921287 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94c03808-f5c9-4486-9cb9-3251d0de671d-istio-token\" (UniqueName: \"kubernetes.io/projected/94c03808-f5c9-4486-9cb9-3251d0de671d-istio-token\") pod \"94c03808-f5c9-4486-9cb9-3251d0de671d\" (UID: \"94c03808-f5c9-4486-9cb9-3251d0de671d\") " Oct 30 17:23:33.922292 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:33.921383 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94c03808-f5c9-4486-9cb9-3251d0de671d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94c03808-f5c9-4486-9cb9-3251d0de671d-istio-envoy\") pod \"94c03808-f5c9-4486-9cb9-3251d0de671d\" (UID: \"94c03808-f5c9-4486-9cb9-3251d0de671d\") " Oct 30 17:23:33.922292 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:33.921423 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94c03808-f5c9-4486-9cb9-3251d0de671d-kube-api-access-9gtnk\" (UniqueName: \"kubernetes.io/projected/94c03808-f5c9-4486-9cb9-3251d0de671d-kube-api-access-9gtnk\") pod \"94c03808-f5c9-4486-9cb9-3251d0de671d\" (UID: \"94c03808-f5c9-4486-9cb9-3251d0de671d\") " Oct 30 17:23:33.922292 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:33.921507 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94c03808-f5c9-4486-9cb9-3251d0de671d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94c03808-f5c9-4486-9cb9-3251d0de671d-workload-certs\") pod \"94c03808-f5c9-4486-9cb9-3251d0de671d\" (UID: \"94c03808-f5c9-4486-9cb9-3251d0de671d\") " Oct 30 17:23:33.922292 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:33.921560 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94c03808-f5c9-4486-9cb9-3251d0de671d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94c03808-f5c9-4486-9cb9-3251d0de671d-istio-data\") pod \"94c03808-f5c9-4486-9cb9-3251d0de671d\" (UID: \"94c03808-f5c9-4486-9cb9-3251d0de671d\") " Oct 30 17:23:33.922292 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:33.921598 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94c03808-f5c9-4486-9cb9-3251d0de671d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94c03808-f5c9-4486-9cb9-3251d0de671d-istiod-ca-cert\") pod \"94c03808-f5c9-4486-9cb9-3251d0de671d\" (UID: \"94c03808-f5c9-4486-9cb9-3251d0de671d\") " Oct 30 17:23:33.922292 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:33.921685 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/94c03808-f5c9-4486-9cb9-3251d0de671d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94c03808-f5c9-4486-9cb9-3251d0de671d-istio-podinfo\") pod \"94c03808-f5c9-4486-9cb9-3251d0de671d\" (UID: \"94c03808-f5c9-4486-9cb9-3251d0de671d\") " Oct 30 17:23:33.922292 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:33.921722 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94c03808-f5c9-4486-9cb9-3251d0de671d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94c03808-f5c9-4486-9cb9-3251d0de671d-workload-socket\") pod \"94c03808-f5c9-4486-9cb9-3251d0de671d\" (UID: \"94c03808-f5c9-4486-9cb9-3251d0de671d\") " Oct 30 17:23:33.927131 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:33.927103 4493 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d5eb370-685d-4d9f-b450-47d754945375-tls-certs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:23:33.927131 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:33.927129 4493 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d5eb370-685d-4d9f-b450-47d754945375-dshm\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:23:33.927131 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:33.927137 4493 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d5eb370-685d-4d9f-b450-47d754945375-home\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:23:33.927393 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:33.927145 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6fxg\" (UniqueName: \"kubernetes.io/projected/0d5eb370-685d-4d9f-b450-47d754945375-kube-api-access-s6fxg\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:23:33.927393 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:33.927154 4493 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d5eb370-685d-4d9f-b450-47d754945375-model-cache\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:23:33.927512 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:33.927402 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94c03808-f5c9-4486-9cb9-3251d0de671d-workload-certs" pod "94c03808-f5c9-4486-9cb9-3251d0de671d" (UID: "94c03808-f5c9-4486-9cb9-3251d0de671d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:23:33.927573 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:33.927527 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94c03808-f5c9-4486-9cb9-3251d0de671d-istio-data" pod "94c03808-f5c9-4486-9cb9-3251d0de671d" (UID: "94c03808-f5c9-4486-9cb9-3251d0de671d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:23:33.927738 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:33.927721 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94c03808-f5c9-4486-9cb9-3251d0de671d-istiod-ca-cert" pod "94c03808-f5c9-4486-9cb9-3251d0de671d" (UID: "94c03808-f5c9-4486-9cb9-3251d0de671d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:23:33.928727 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:33.928689 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94c03808-f5c9-4486-9cb9-3251d0de671d-credential-socket" pod "94c03808-f5c9-4486-9cb9-3251d0de671d" (UID: "94c03808-f5c9-4486-9cb9-3251d0de671d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:23:33.929381 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:33.929358 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94c03808-f5c9-4486-9cb9-3251d0de671d-workload-socket" pod "94c03808-f5c9-4486-9cb9-3251d0de671d" (UID: "94c03808-f5c9-4486-9cb9-3251d0de671d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:23:33.930842 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:33.930810 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/94c03808-f5c9-4486-9cb9-3251d0de671d-istio-podinfo" pod "94c03808-f5c9-4486-9cb9-3251d0de671d" (UID: "94c03808-f5c9-4486-9cb9-3251d0de671d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:23:33.930967 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:33.930947 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94c03808-f5c9-4486-9cb9-3251d0de671d-istio-token" pod "94c03808-f5c9-4486-9cb9-3251d0de671d" (UID: "94c03808-f5c9-4486-9cb9-3251d0de671d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:23:33.932310 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:33.932280 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94c03808-f5c9-4486-9cb9-3251d0de671d-kube-api-access-9gtnk" pod "94c03808-f5c9-4486-9cb9-3251d0de671d" (UID: "94c03808-f5c9-4486-9cb9-3251d0de671d"). InnerVolumeSpecName "kube-api-access-9gtnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:23:33.932700 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:33.932680 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94c03808-f5c9-4486-9cb9-3251d0de671d-istio-envoy" pod "94c03808-f5c9-4486-9cb9-3251d0de671d" (UID: "94c03808-f5c9-4486-9cb9-3251d0de671d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:23:34.028328 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:34.028292 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94c03808-f5c9-4486-9cb9-3251d0de671d-credential-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:23:34.028328 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:34.028326 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94c03808-f5c9-4486-9cb9-3251d0de671d-istio-token\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:23:34.028529 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:34.028341 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94c03808-f5c9-4486-9cb9-3251d0de671d-istio-envoy\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:23:34.028529 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:34.028350 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gtnk\" (UniqueName: \"kubernetes.io/projected/94c03808-f5c9-4486-9cb9-3251d0de671d-kube-api-access-9gtnk\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:23:34.028529 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:34.028359 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94c03808-f5c9-4486-9cb9-3251d0de671d-workload-certs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:23:34.028529 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:34.028367 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94c03808-f5c9-4486-9cb9-3251d0de671d-istio-data\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:23:34.028529 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:34.028376 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94c03808-f5c9-4486-9cb9-3251d0de671d-istiod-ca-cert\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:23:34.028529 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:34.028384 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94c03808-f5c9-4486-9cb9-3251d0de671d-istio-podinfo\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:23:34.028529 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:34.028392 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94c03808-f5c9-4486-9cb9-3251d0de671d-workload-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:23:34.741193 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:34.741162 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn" Oct 30 17:23:34.741193 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:34.741183 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn" event={"ID":"94c03808-f5c9-4486-9cb9-3251d0de671d","Type":"ContainerDied","Data":"fa4508f60247fd40ec2e4923d7c935a253d5f04f7b30e1e69a980ba530186953"} Oct 30 17:23:34.741473 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:34.741246 4493 scope.go:122] "RemoveContainer" containerID="d173fbcc27f17ac896559d111bf61a822dc3a786e1737baa35bb6a277b8db548" Oct 30 17:23:34.743067 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:34.742902 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-85eb87/test-model-85eb87-kserve-686468c445-ssq2z" event={"ID":"0d5eb370-685d-4d9f-b450-47d754945375","Type":"ContainerDied","Data":"07e8fe4d2ef21f9a19ff4d92ea4f565b8833c889c6509a42b384a4734eecbdef"} Oct 30 17:23:34.743067 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:34.742985 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-85eb87/test-model-85eb87-kserve-686468c445-ssq2z" Oct 30 17:23:34.756381 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:34.756361 4493 scope.go:122] "RemoveContainer" containerID="28f01c79c71f88d779da209dc4badbe457a0bc7779efcd09cebffddefe51e70f" Oct 30 17:23:34.811452 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:34.811418 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn"] Oct 30 17:23:34.823847 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:34.823816 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-dd715e-openshift-default-85dc99f79c-scqnn"] Oct 30 17:23:34.848368 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:34.848338 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-85eb87/test-model-85eb87-kserve-686468c445-ssq2z"] Oct 30 17:23:34.852929 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:34.852904 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-85eb87/test-model-85eb87-kserve-686468c445-ssq2z"] Oct 30 17:23:34.934941 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:34.934907 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d5eb370-685d-4d9f-b450-47d754945375" path="/var/lib/kubelet/pods/0d5eb370-685d-4d9f-b450-47d754945375/volumes" Oct 30 17:23:34.935467 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:34.935449 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94c03808-f5c9-4486-9cb9-3251d0de671d" path="/var/lib/kubelet/pods/94c03808-f5c9-4486-9cb9-3251d0de671d/volumes" Oct 30 17:23:37.505809 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:37.497045 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9afbb2-6njdf"] Oct 30 17:23:37.505809 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:37.498087 4493 cpu_manager.go:414] "containerMap: removing container" podUID="7521ace7-b975-4772-a31d-75b30cf36283" containerName="revoke-keys" Oct 30 17:23:37.505809 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:37.498110 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7521ace7-b975-4772-a31d-75b30cf36283" containerName="revoke-keys" Oct 30 17:23:37.505809 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:37.498173 4493 cpu_manager.go:414] "containerMap: removing container" podUID="b56c14bc-2ce9-4e40-9a24-f86e3e6f79d3" containerName="revoke-keys" Oct 30 17:23:37.505809 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:37.498181 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b56c14bc-2ce9-4e40-9a24-f86e3e6f79d3" containerName="revoke-keys" Oct 30 17:23:37.505809 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:37.498246 4493 cpu_manager.go:414] "containerMap: removing container" podUID="0d5eb370-685d-4d9f-b450-47d754945375" containerName="main" Oct 30 17:23:37.505809 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:37.498254 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d5eb370-685d-4d9f-b450-47d754945375" containerName="main" Oct 30 17:23:37.505809 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:37.498277 4493 cpu_manager.go:414] "containerMap: removing container" podUID="94c03808-f5c9-4486-9cb9-3251d0de671d" containerName="istio-proxy" Oct 30 17:23:37.505809 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:37.498285 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94c03808-f5c9-4486-9cb9-3251d0de671d" containerName="istio-proxy" Oct 30 17:23:37.505809 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:37.498466 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="b56c14bc-2ce9-4e40-9a24-f86e3e6f79d3" containerName="revoke-keys" Oct 30 17:23:37.505809 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:37.498482 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d5eb370-685d-4d9f-b450-47d754945375" containerName="main" Oct 30 17:23:37.505809 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:37.498496 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="94c03808-f5c9-4486-9cb9-3251d0de671d" containerName="istio-proxy" Oct 30 17:23:37.505809 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:37.498509 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="7521ace7-b975-4772-a31d-75b30cf36283" containerName="revoke-keys" Oct 30 17:23:37.556906 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:37.549124 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9afbb2-6njdf" Oct 30 17:23:37.571805 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:37.566172 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9afbb2-6njdf"] Oct 30 17:23:37.597910 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:37.576600 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:23:37.597910 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:37.593810 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:23:37.597910 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:37.593920 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 17:23:37.597910 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:37.593988 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:23:37.599767 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:37.599731 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h626r\" (UniqueName: \"kubernetes.io/projected/ac8c430b-0861-4ccd-b4ab-ac0a02716019-kube-api-access-h626r\") pod \"maas-api-revoke-keys-e2e-inf-a-9afbb2-6njdf\" (UID: \"ac8c430b-0861-4ccd-b4ab-ac0a02716019\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9afbb2-6njdf" Oct 30 17:23:37.601007 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:37.600934 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac8c430b-0861-4ccd-b4ab-ac0a02716019-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9afbb2-6njdf\" (UID: \"ac8c430b-0861-4ccd-b4ab-ac0a02716019\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9afbb2-6njdf" Oct 30 17:23:37.601338 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:37.601315 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac8c430b-0861-4ccd-b4ab-ac0a02716019-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9afbb2-6njdf\" (UID: \"ac8c430b-0861-4ccd-b4ab-ac0a02716019\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9afbb2-6njdf" Oct 30 17:23:37.702979 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:37.702936 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac8c430b-0861-4ccd-b4ab-ac0a02716019-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9afbb2-6njdf\" (UID: \"ac8c430b-0861-4ccd-b4ab-ac0a02716019\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9afbb2-6njdf" Oct 30 17:23:37.703150 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:37.703049 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h626r\" (UniqueName: \"kubernetes.io/projected/ac8c430b-0861-4ccd-b4ab-ac0a02716019-kube-api-access-h626r\") pod \"maas-api-revoke-keys-e2e-inf-a-9afbb2-6njdf\" (UID: \"ac8c430b-0861-4ccd-b4ab-ac0a02716019\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9afbb2-6njdf" Oct 30 17:23:37.703150 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:37.703087 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac8c430b-0861-4ccd-b4ab-ac0a02716019-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9afbb2-6njdf\" (UID: \"ac8c430b-0861-4ccd-b4ab-ac0a02716019\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9afbb2-6njdf" Oct 30 17:23:37.703879 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:37.703729 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac8c430b-0861-4ccd-b4ab-ac0a02716019-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9afbb2-6njdf\" (UID: \"ac8c430b-0861-4ccd-b4ab-ac0a02716019\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9afbb2-6njdf" Oct 30 17:23:37.705718 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:37.705691 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac8c430b-0861-4ccd-b4ab-ac0a02716019-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9afbb2-6njdf\" (UID: \"ac8c430b-0861-4ccd-b4ab-ac0a02716019\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9afbb2-6njdf" Oct 30 17:23:37.728132 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:37.728104 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h626r\" (UniqueName: \"kubernetes.io/projected/ac8c430b-0861-4ccd-b4ab-ac0a02716019-kube-api-access-h626r\") pod \"maas-api-revoke-keys-e2e-inf-a-9afbb2-6njdf\" (UID: \"ac8c430b-0861-4ccd-b4ab-ac0a02716019\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9afbb2-6njdf" Oct 30 17:23:37.899022 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:37.898998 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9afbb2-6njdf" Oct 30 17:23:38.216719 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:38.216688 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9afbb2-6njdf"] Oct 30 17:23:38.218738 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:23:38.218712 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac8c430b_0861_4ccd_b4ab_ac0a02716019.slice/crio-61a6736e9d662a104a591d6fd4e53bd197b7ceca30746f7a604317cb30596308 WatchSource:0}: Error finding container 61a6736e9d662a104a591d6fd4e53bd197b7ceca30746f7a604317cb30596308: Status 404 returned error can't find the container with id 61a6736e9d662a104a591d6fd4e53bd197b7ceca30746f7a604317cb30596308 Oct 30 17:23:38.797892 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:38.797845 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9afbb2-6njdf" event={"ID":"ac8c430b-0861-4ccd-b4ab-ac0a02716019","Type":"ContainerStarted","Data":"77e1e14a1ea94032a5956f942e525ccf03c86f1b977595f5a38ad84c7c9d035c"} Oct 30 17:23:38.798257 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:38.797917 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9afbb2-6njdf" event={"ID":"ac8c430b-0861-4ccd-b4ab-ac0a02716019","Type":"ContainerStarted","Data":"61a6736e9d662a104a591d6fd4e53bd197b7ceca30746f7a604317cb30596308"} Oct 30 17:23:38.893676 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:38.893640 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-9afbb2/test-model-9afbb2-kserve-6c5d6c968c-sfbmg" podUID="a94190b0-d6ef-4227-9b06-d10cbf89c567" containerName="main" containerID="cri-o://4ec528c3b973232a1ff94187310213b0246c32f3d46d5f8d4241f8ba9205b680" gracePeriod=45 Oct 30 17:23:38.969197 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:38.969119 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-9afbb2/test-model-9afbb2-kserve-6c5d6c968c-sfbmg" podUID="a94190b0-d6ef-4227-9b06-d10cbf89c567" containerName="main" probeResult="failure" output="Get \"https://10.134.1.51:8000/ready\": dial tcp 10.134.1.51:8000: connect: connection refused" Oct 30 17:23:39.189141 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:39.189107 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9afbb2/test-model-9afbb2-kserve-6c5d6c968c-sfbmg" Oct 30 17:23:39.225672 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:39.225589 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a94190b0-d6ef-4227-9b06-d10cbf89c567-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a94190b0-d6ef-4227-9b06-d10cbf89c567-model-cache\") pod \"a94190b0-d6ef-4227-9b06-d10cbf89c567\" (UID: \"a94190b0-d6ef-4227-9b06-d10cbf89c567\") " Oct 30 17:23:39.225855 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:39.225674 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a94190b0-d6ef-4227-9b06-d10cbf89c567-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a94190b0-d6ef-4227-9b06-d10cbf89c567-dshm\") pod \"a94190b0-d6ef-4227-9b06-d10cbf89c567\" (UID: \"a94190b0-d6ef-4227-9b06-d10cbf89c567\") " Oct 30 17:23:39.225855 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:39.225729 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a94190b0-d6ef-4227-9b06-d10cbf89c567-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a94190b0-d6ef-4227-9b06-d10cbf89c567-tmp-dir\") pod \"a94190b0-d6ef-4227-9b06-d10cbf89c567\" (UID: \"a94190b0-d6ef-4227-9b06-d10cbf89c567\") " Oct 30 17:23:39.225998 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:39.225921 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a94190b0-d6ef-4227-9b06-d10cbf89c567-model-cache" pod "a94190b0-d6ef-4227-9b06-d10cbf89c567" (UID: "a94190b0-d6ef-4227-9b06-d10cbf89c567"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:23:39.225998 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:39.225941 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a94190b0-d6ef-4227-9b06-d10cbf89c567-kube-api-access-w66qv\" (UniqueName: \"kubernetes.io/projected/a94190b0-d6ef-4227-9b06-d10cbf89c567-kube-api-access-w66qv\") pod \"a94190b0-d6ef-4227-9b06-d10cbf89c567\" (UID: \"a94190b0-d6ef-4227-9b06-d10cbf89c567\") " Oct 30 17:23:39.226119 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:39.226043 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a94190b0-d6ef-4227-9b06-d10cbf89c567-tmp-dir" pod "a94190b0-d6ef-4227-9b06-d10cbf89c567" (UID: "a94190b0-d6ef-4227-9b06-d10cbf89c567"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:23:39.226119 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:39.226057 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a94190b0-d6ef-4227-9b06-d10cbf89c567-tls-certs\" (UniqueName: \"kubernetes.io/secret/a94190b0-d6ef-4227-9b06-d10cbf89c567-tls-certs\") pod \"a94190b0-d6ef-4227-9b06-d10cbf89c567\" (UID: \"a94190b0-d6ef-4227-9b06-d10cbf89c567\") " Oct 30 17:23:39.226119 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:39.226099 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a94190b0-d6ef-4227-9b06-d10cbf89c567-home\" (UniqueName: \"kubernetes.io/empty-dir/a94190b0-d6ef-4227-9b06-d10cbf89c567-home\") pod \"a94190b0-d6ef-4227-9b06-d10cbf89c567\" (UID: \"a94190b0-d6ef-4227-9b06-d10cbf89c567\") " Oct 30 17:23:39.226596 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:39.226535 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a94190b0-d6ef-4227-9b06-d10cbf89c567-home" pod "a94190b0-d6ef-4227-9b06-d10cbf89c567" (UID: "a94190b0-d6ef-4227-9b06-d10cbf89c567"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:23:39.226735 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:39.226704 4493 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a94190b0-d6ef-4227-9b06-d10cbf89c567-home\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:23:39.226735 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:39.226722 4493 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a94190b0-d6ef-4227-9b06-d10cbf89c567-model-cache\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:23:39.226875 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:39.226737 4493 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a94190b0-d6ef-4227-9b06-d10cbf89c567-tmp-dir\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:23:39.228577 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:39.228548 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a94190b0-d6ef-4227-9b06-d10cbf89c567-tls-certs" pod "a94190b0-d6ef-4227-9b06-d10cbf89c567" (UID: "a94190b0-d6ef-4227-9b06-d10cbf89c567"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:23:39.228695 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:39.228646 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a94190b0-d6ef-4227-9b06-d10cbf89c567-dshm" pod "a94190b0-d6ef-4227-9b06-d10cbf89c567" (UID: "a94190b0-d6ef-4227-9b06-d10cbf89c567"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:23:39.228744 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:39.228700 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a94190b0-d6ef-4227-9b06-d10cbf89c567-kube-api-access-w66qv" pod "a94190b0-d6ef-4227-9b06-d10cbf89c567" (UID: "a94190b0-d6ef-4227-9b06-d10cbf89c567"). InnerVolumeSpecName "kube-api-access-w66qv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:23:39.236635 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:39.236584 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9afbb2-6njdf" podStartSLOduration=2.236567622 podStartE2EDuration="2.236567622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:23:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:23:38.837008245 +0000 UTC m=+3646.064183346" watchObservedRunningTime="2026-10-30 17:23:39.236567622 +0000 UTC m=+3646.463742724" Oct 30 17:23:39.328647 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:39.328614 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w66qv\" (UniqueName: \"kubernetes.io/projected/a94190b0-d6ef-4227-9b06-d10cbf89c567-kube-api-access-w66qv\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:23:39.328765 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:39.328652 4493 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a94190b0-d6ef-4227-9b06-d10cbf89c567-tls-certs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:23:39.328765 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:39.328666 4493 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a94190b0-d6ef-4227-9b06-d10cbf89c567-dshm\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:23:39.807531 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:39.807494 4493 generic.go:356] "Generic (PLEG): container finished" podID="a94190b0-d6ef-4227-9b06-d10cbf89c567" containerID="4ec528c3b973232a1ff94187310213b0246c32f3d46d5f8d4241f8ba9205b680" exitCode=0 Oct 30 17:23:39.808034 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:39.807539 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9afbb2/test-model-9afbb2-kserve-6c5d6c968c-sfbmg" Oct 30 17:23:39.808034 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:39.807585 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9afbb2/test-model-9afbb2-kserve-6c5d6c968c-sfbmg" event={"ID":"a94190b0-d6ef-4227-9b06-d10cbf89c567","Type":"ContainerDied","Data":"4ec528c3b973232a1ff94187310213b0246c32f3d46d5f8d4241f8ba9205b680"} Oct 30 17:23:39.808034 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:39.807634 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9afbb2/test-model-9afbb2-kserve-6c5d6c968c-sfbmg" event={"ID":"a94190b0-d6ef-4227-9b06-d10cbf89c567","Type":"ContainerDied","Data":"f282ec52e11aeecc05c24173bbe6f0661f9fe7acbbc213a659803d2bb5950389"} Oct 30 17:23:39.808034 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:39.807658 4493 scope.go:122] "RemoveContainer" containerID="4ec528c3b973232a1ff94187310213b0246c32f3d46d5f8d4241f8ba9205b680" Oct 30 17:23:39.809964 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:39.809938 4493 generic.go:356] "Generic (PLEG): container finished" podID="ac8c430b-0861-4ccd-b4ab-ac0a02716019" containerID="77e1e14a1ea94032a5956f942e525ccf03c86f1b977595f5a38ad84c7c9d035c" exitCode=0 Oct 30 17:23:39.810183 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:39.810068 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9afbb2-6njdf" event={"ID":"ac8c430b-0861-4ccd-b4ab-ac0a02716019","Type":"ContainerDied","Data":"77e1e14a1ea94032a5956f942e525ccf03c86f1b977595f5a38ad84c7c9d035c"} Oct 30 17:23:39.820154 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:39.820135 4493 scope.go:122] "RemoveContainer" containerID="4ec528c3b973232a1ff94187310213b0246c32f3d46d5f8d4241f8ba9205b680" Oct 30 17:23:39.820435 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:23:39.820404 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ec528c3b973232a1ff94187310213b0246c32f3d46d5f8d4241f8ba9205b680\": container with ID starting with 4ec528c3b973232a1ff94187310213b0246c32f3d46d5f8d4241f8ba9205b680 not found: ID does not exist" containerID="4ec528c3b973232a1ff94187310213b0246c32f3d46d5f8d4241f8ba9205b680" Oct 30 17:23:39.820538 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:39.820438 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ec528c3b973232a1ff94187310213b0246c32f3d46d5f8d4241f8ba9205b680"} err="failed to get container status \"4ec528c3b973232a1ff94187310213b0246c32f3d46d5f8d4241f8ba9205b680\": rpc error: code = NotFound desc = could not find container \"4ec528c3b973232a1ff94187310213b0246c32f3d46d5f8d4241f8ba9205b680\": container with ID starting with 4ec528c3b973232a1ff94187310213b0246c32f3d46d5f8d4241f8ba9205b680 not found: ID does not exist" Oct 30 17:23:39.860188 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:39.860163 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-9afbb2/test-model-9afbb2-kserve-6c5d6c968c-sfbmg"] Oct 30 17:23:39.935196 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:39.935165 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-9afbb2/test-model-9afbb2-kserve-6c5d6c968c-sfbmg"] Oct 30 17:23:40.718665 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:40.718599 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-bbc36f/test-model-bbc36f-kserve-6c7bbb6b47-dsbs2" podUID="86707cad-e7dd-4a77-b04b-3bf99540cc18" containerName="main" containerID="cri-o://e2229f8e473381fdbfabf3b52afb310e14211ab21a16abce81087bff8f720494" gracePeriod=45 Oct 30 17:23:40.948989 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:40.948877 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a94190b0-d6ef-4227-9b06-d10cbf89c567" path="/var/lib/kubelet/pods/a94190b0-d6ef-4227-9b06-d10cbf89c567/volumes" Oct 30 17:23:41.022162 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:41.022107 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9afbb2-6njdf" Oct 30 17:23:41.094303 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:41.094267 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-bbc36f/test-model-bbc36f-kserve-6c7bbb6b47-dsbs2" Oct 30 17:23:41.205516 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:41.205477 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86707cad-e7dd-4a77-b04b-3bf99540cc18-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86707cad-e7dd-4a77-b04b-3bf99540cc18-model-cache\") pod \"86707cad-e7dd-4a77-b04b-3bf99540cc18\" (UID: \"86707cad-e7dd-4a77-b04b-3bf99540cc18\") " Oct 30 17:23:41.205918 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:41.205896 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac8c430b-0861-4ccd-b4ab-ac0a02716019-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac8c430b-0861-4ccd-b4ab-ac0a02716019-maas-api-openshift-ca\") pod \"ac8c430b-0861-4ccd-b4ab-ac0a02716019\" (UID: \"ac8c430b-0861-4ccd-b4ab-ac0a02716019\") " Oct 30 17:23:41.206092 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:41.206076 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac8c430b-0861-4ccd-b4ab-ac0a02716019-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac8c430b-0861-4ccd-b4ab-ac0a02716019-maas-api-xks-ca\") pod \"ac8c430b-0861-4ccd-b4ab-ac0a02716019\" (UID: \"ac8c430b-0861-4ccd-b4ab-ac0a02716019\") " Oct 30 17:23:41.206251 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:41.206230 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86707cad-e7dd-4a77-b04b-3bf99540cc18-kube-api-access-qhnkd\" (UniqueName: \"kubernetes.io/projected/86707cad-e7dd-4a77-b04b-3bf99540cc18-kube-api-access-qhnkd\") pod \"86707cad-e7dd-4a77-b04b-3bf99540cc18\" (UID: \"86707cad-e7dd-4a77-b04b-3bf99540cc18\") " Oct 30 17:23:41.206429 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:41.206413 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86707cad-e7dd-4a77-b04b-3bf99540cc18-dshm\" (UniqueName: \"kubernetes.io/empty-dir/86707cad-e7dd-4a77-b04b-3bf99540cc18-dshm\") pod \"86707cad-e7dd-4a77-b04b-3bf99540cc18\" (UID: \"86707cad-e7dd-4a77-b04b-3bf99540cc18\") " Oct 30 17:23:41.206649 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:41.206636 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86707cad-e7dd-4a77-b04b-3bf99540cc18-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86707cad-e7dd-4a77-b04b-3bf99540cc18-tmp-dir\") pod \"86707cad-e7dd-4a77-b04b-3bf99540cc18\" (UID: \"86707cad-e7dd-4a77-b04b-3bf99540cc18\") " Oct 30 17:23:41.206823 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:41.206809 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac8c430b-0861-4ccd-b4ab-ac0a02716019-kube-api-access-h626r\" (UniqueName: \"kubernetes.io/projected/ac8c430b-0861-4ccd-b4ab-ac0a02716019-kube-api-access-h626r\") pod \"ac8c430b-0861-4ccd-b4ab-ac0a02716019\" (UID: \"ac8c430b-0861-4ccd-b4ab-ac0a02716019\") " Oct 30 17:23:41.206979 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:41.206966 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86707cad-e7dd-4a77-b04b-3bf99540cc18-home\" (UniqueName: \"kubernetes.io/empty-dir/86707cad-e7dd-4a77-b04b-3bf99540cc18-home\") pod \"86707cad-e7dd-4a77-b04b-3bf99540cc18\" (UID: \"86707cad-e7dd-4a77-b04b-3bf99540cc18\") " Oct 30 17:23:41.207133 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:41.207114 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86707cad-e7dd-4a77-b04b-3bf99540cc18-tls-certs\" (UniqueName: \"kubernetes.io/secret/86707cad-e7dd-4a77-b04b-3bf99540cc18-tls-certs\") pod \"86707cad-e7dd-4a77-b04b-3bf99540cc18\" (UID: \"86707cad-e7dd-4a77-b04b-3bf99540cc18\") " Oct 30 17:23:41.208633 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:41.208606 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86707cad-e7dd-4a77-b04b-3bf99540cc18-tmp-dir" pod "86707cad-e7dd-4a77-b04b-3bf99540cc18" (UID: "86707cad-e7dd-4a77-b04b-3bf99540cc18"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:23:41.229929 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:41.212561 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86707cad-e7dd-4a77-b04b-3bf99540cc18-home" pod "86707cad-e7dd-4a77-b04b-3bf99540cc18" (UID: "86707cad-e7dd-4a77-b04b-3bf99540cc18"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:23:41.229929 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:41.212750 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86707cad-e7dd-4a77-b04b-3bf99540cc18-model-cache" pod "86707cad-e7dd-4a77-b04b-3bf99540cc18" (UID: "86707cad-e7dd-4a77-b04b-3bf99540cc18"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:23:41.229929 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:41.217461 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86707cad-e7dd-4a77-b04b-3bf99540cc18-dshm" pod "86707cad-e7dd-4a77-b04b-3bf99540cc18" (UID: "86707cad-e7dd-4a77-b04b-3bf99540cc18"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:23:41.229929 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:41.217535 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86707cad-e7dd-4a77-b04b-3bf99540cc18-tls-certs" pod "86707cad-e7dd-4a77-b04b-3bf99540cc18" (UID: "86707cad-e7dd-4a77-b04b-3bf99540cc18"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:23:41.229929 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:41.217670 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac8c430b-0861-4ccd-b4ab-ac0a02716019-kube-api-access-h626r" pod "ac8c430b-0861-4ccd-b4ab-ac0a02716019" (UID: "ac8c430b-0861-4ccd-b4ab-ac0a02716019"). InnerVolumeSpecName "kube-api-access-h626r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:23:41.229929 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:41.219963 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac8c430b-0861-4ccd-b4ab-ac0a02716019-maas-api-openshift-ca" pod "ac8c430b-0861-4ccd-b4ab-ac0a02716019" (UID: "ac8c430b-0861-4ccd-b4ab-ac0a02716019"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:23:41.240992 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:41.240928 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac8c430b-0861-4ccd-b4ab-ac0a02716019-maas-api-xks-ca" pod "ac8c430b-0861-4ccd-b4ab-ac0a02716019" (UID: "ac8c430b-0861-4ccd-b4ab-ac0a02716019"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:23:41.241411 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:41.241389 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86707cad-e7dd-4a77-b04b-3bf99540cc18-kube-api-access-qhnkd" pod "86707cad-e7dd-4a77-b04b-3bf99540cc18" (UID: "86707cad-e7dd-4a77-b04b-3bf99540cc18"). InnerVolumeSpecName "kube-api-access-qhnkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:23:41.309848 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:41.309812 4493 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86707cad-e7dd-4a77-b04b-3bf99540cc18-tmp-dir\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:23:41.309848 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:41.309850 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h626r\" (UniqueName: \"kubernetes.io/projected/ac8c430b-0861-4ccd-b4ab-ac0a02716019-kube-api-access-h626r\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:23:41.310043 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:41.309863 4493 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86707cad-e7dd-4a77-b04b-3bf99540cc18-home\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:23:41.310043 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:41.309877 4493 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86707cad-e7dd-4a77-b04b-3bf99540cc18-tls-certs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:23:41.310043 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:41.309897 4493 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86707cad-e7dd-4a77-b04b-3bf99540cc18-model-cache\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:23:41.310043 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:41.309910 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac8c430b-0861-4ccd-b4ab-ac0a02716019-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:23:41.310043 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:41.309922 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac8c430b-0861-4ccd-b4ab-ac0a02716019-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:23:41.310043 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:41.309936 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhnkd\" (UniqueName: \"kubernetes.io/projected/86707cad-e7dd-4a77-b04b-3bf99540cc18-kube-api-access-qhnkd\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:23:41.310043 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:41.309948 4493 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86707cad-e7dd-4a77-b04b-3bf99540cc18-dshm\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:23:41.834805 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:41.834752 4493 generic.go:356] "Generic (PLEG): container finished" podID="86707cad-e7dd-4a77-b04b-3bf99540cc18" containerID="e2229f8e473381fdbfabf3b52afb310e14211ab21a16abce81087bff8f720494" exitCode=0 Oct 30 17:23:41.834998 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:41.834919 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-bbc36f/test-model-bbc36f-kserve-6c7bbb6b47-dsbs2" Oct 30 17:23:41.836388 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:41.836359 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-bbc36f/test-model-bbc36f-kserve-6c7bbb6b47-dsbs2" event={"ID":"86707cad-e7dd-4a77-b04b-3bf99540cc18","Type":"ContainerDied","Data":"e2229f8e473381fdbfabf3b52afb310e14211ab21a16abce81087bff8f720494"} Oct 30 17:23:41.836600 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:41.836574 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-bbc36f/test-model-bbc36f-kserve-6c7bbb6b47-dsbs2" event={"ID":"86707cad-e7dd-4a77-b04b-3bf99540cc18","Type":"ContainerDied","Data":"0990298a7a001339d62571b2687ecaeae2bab785728c6ce36dd3943978e2fb0e"} Oct 30 17:23:41.836723 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:41.836609 4493 scope.go:122] "RemoveContainer" containerID="e2229f8e473381fdbfabf3b52afb310e14211ab21a16abce81087bff8f720494" Oct 30 17:23:41.838511 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:41.838483 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9afbb2-6njdf" event={"ID":"ac8c430b-0861-4ccd-b4ab-ac0a02716019","Type":"ContainerDied","Data":"61a6736e9d662a104a591d6fd4e53bd197b7ceca30746f7a604317cb30596308"} Oct 30 17:23:41.838616 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:41.838538 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61a6736e9d662a104a591d6fd4e53bd197b7ceca30746f7a604317cb30596308" Oct 30 17:23:41.838616 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:41.838598 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9afbb2-6njdf" Oct 30 17:23:41.856673 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:41.856648 4493 scope.go:122] "RemoveContainer" containerID="e2229f8e473381fdbfabf3b52afb310e14211ab21a16abce81087bff8f720494" Oct 30 17:23:41.857036 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:23:41.857017 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2229f8e473381fdbfabf3b52afb310e14211ab21a16abce81087bff8f720494\": container with ID starting with e2229f8e473381fdbfabf3b52afb310e14211ab21a16abce81087bff8f720494 not found: ID does not exist" containerID="e2229f8e473381fdbfabf3b52afb310e14211ab21a16abce81087bff8f720494" Oct 30 17:23:41.857135 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:41.857043 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2229f8e473381fdbfabf3b52afb310e14211ab21a16abce81087bff8f720494"} err="failed to get container status \"e2229f8e473381fdbfabf3b52afb310e14211ab21a16abce81087bff8f720494\": rpc error: code = NotFound desc = could not find container \"e2229f8e473381fdbfabf3b52afb310e14211ab21a16abce81087bff8f720494\": container with ID starting with e2229f8e473381fdbfabf3b52afb310e14211ab21a16abce81087bff8f720494 not found: ID does not exist" Oct 30 17:23:41.906994 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:41.906960 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-bbc36f/test-model-bbc36f-kserve-6c7bbb6b47-dsbs2"] Oct 30 17:23:41.912988 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:41.912958 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-bbc36f/test-model-bbc36f-kserve-6c7bbb6b47-dsbs2"] Oct 30 17:23:41.951759 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:41.951720 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-bbc36f/test-model-bbc36f-kserve-6c7bbb6b47-dsbs2" podUID="86707cad-e7dd-4a77-b04b-3bf99540cc18" containerName="main" probeResult="failure" output="Get \"https://10.134.1.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:23:42.937563 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:42.937523 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86707cad-e7dd-4a77-b04b-3bf99540cc18" path="/var/lib/kubelet/pods/86707cad-e7dd-4a77-b04b-3bf99540cc18/volumes" Oct 30 17:23:43.023263 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:43.023228 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 17:23:44.066809 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:44.065864 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f63f380-kvm2k"] Oct 30 17:23:44.096813 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:44.091009 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f63f380-kvm2k"] Oct 30 17:23:44.934100 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:44.934056 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6033dc55-f3f8-45e2-bfe3-0d968ff023a1" path="/var/lib/kubelet/pods/6033dc55-f3f8-45e2-bfe3-0d968ff023a1/volumes" Oct 30 17:23:51.087039 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:51.087007 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br"] Oct 30 17:23:51.087498 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:51.087193 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br" podUID="b30c2fa0-a3d3-4942-9596-bb6b245a238f" containerName="istio-proxy" containerID="cri-o://75c694a4b9d0335d33b9544ceba1690e4f05a135e5cb26ce021dd775702cdaa9" gracePeriod=30 Oct 30 17:23:53.502328 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:53.502286 4493 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:23:53.502719 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:53.502343 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br" podUID="b30c2fa0-a3d3-4942-9596-bb6b245a238f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:23:56.384976 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:56.384947 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br" Oct 30 17:23:56.439748 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:56.439699 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b30c2fa0-a3d3-4942-9596-bb6b245a238f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b30c2fa0-a3d3-4942-9596-bb6b245a238f-workload-socket\") pod \"b30c2fa0-a3d3-4942-9596-bb6b245a238f\" (UID: \"b30c2fa0-a3d3-4942-9596-bb6b245a238f\") " Oct 30 17:23:56.439748 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:56.439746 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b30c2fa0-a3d3-4942-9596-bb6b245a238f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b30c2fa0-a3d3-4942-9596-bb6b245a238f-credential-socket\") pod \"b30c2fa0-a3d3-4942-9596-bb6b245a238f\" (UID: \"b30c2fa0-a3d3-4942-9596-bb6b245a238f\") " Oct 30 17:23:56.440039 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:56.439815 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b30c2fa0-a3d3-4942-9596-bb6b245a238f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b30c2fa0-a3d3-4942-9596-bb6b245a238f-istio-data\") pod \"b30c2fa0-a3d3-4942-9596-bb6b245a238f\" (UID: \"b30c2fa0-a3d3-4942-9596-bb6b245a238f\") " Oct 30 17:23:56.440039 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:56.439927 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b30c2fa0-a3d3-4942-9596-bb6b245a238f-kube-api-access-8t5n5\" (UniqueName: \"kubernetes.io/projected/b30c2fa0-a3d3-4942-9596-bb6b245a238f-kube-api-access-8t5n5\") pod \"b30c2fa0-a3d3-4942-9596-bb6b245a238f\" (UID: \"b30c2fa0-a3d3-4942-9596-bb6b245a238f\") " Oct 30 17:23:56.440167 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:56.440086 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b30c2fa0-a3d3-4942-9596-bb6b245a238f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b30c2fa0-a3d3-4942-9596-bb6b245a238f-istio-envoy\") pod \"b30c2fa0-a3d3-4942-9596-bb6b245a238f\" (UID: \"b30c2fa0-a3d3-4942-9596-bb6b245a238f\") " Oct 30 17:23:56.440167 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:56.440120 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b30c2fa0-a3d3-4942-9596-bb6b245a238f-workload-socket" pod "b30c2fa0-a3d3-4942-9596-bb6b245a238f" (UID: "b30c2fa0-a3d3-4942-9596-bb6b245a238f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:23:56.440288 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:56.440181 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b30c2fa0-a3d3-4942-9596-bb6b245a238f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b30c2fa0-a3d3-4942-9596-bb6b245a238f-istiod-ca-cert\") pod \"b30c2fa0-a3d3-4942-9596-bb6b245a238f\" (UID: \"b30c2fa0-a3d3-4942-9596-bb6b245a238f\") " Oct 30 17:23:56.440288 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:56.440226 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b30c2fa0-a3d3-4942-9596-bb6b245a238f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b30c2fa0-a3d3-4942-9596-bb6b245a238f-istio-podinfo\") pod \"b30c2fa0-a3d3-4942-9596-bb6b245a238f\" (UID: \"b30c2fa0-a3d3-4942-9596-bb6b245a238f\") " Oct 30 17:23:56.440288 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:56.440245 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b30c2fa0-a3d3-4942-9596-bb6b245a238f-istio-data" pod "b30c2fa0-a3d3-4942-9596-bb6b245a238f" (UID: "b30c2fa0-a3d3-4942-9596-bb6b245a238f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:23:56.440476 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:56.440301 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b30c2fa0-a3d3-4942-9596-bb6b245a238f-istio-token\" (UniqueName: \"kubernetes.io/projected/b30c2fa0-a3d3-4942-9596-bb6b245a238f-istio-token\") pod \"b30c2fa0-a3d3-4942-9596-bb6b245a238f\" (UID: \"b30c2fa0-a3d3-4942-9596-bb6b245a238f\") " Oct 30 17:23:56.440476 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:56.440336 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b30c2fa0-a3d3-4942-9596-bb6b245a238f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b30c2fa0-a3d3-4942-9596-bb6b245a238f-workload-certs\") pod \"b30c2fa0-a3d3-4942-9596-bb6b245a238f\" (UID: \"b30c2fa0-a3d3-4942-9596-bb6b245a238f\") " Oct 30 17:23:56.440476 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:56.440336 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b30c2fa0-a3d3-4942-9596-bb6b245a238f-credential-socket" pod "b30c2fa0-a3d3-4942-9596-bb6b245a238f" (UID: "b30c2fa0-a3d3-4942-9596-bb6b245a238f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:23:56.440634 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:56.440538 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b30c2fa0-a3d3-4942-9596-bb6b245a238f-istiod-ca-cert" pod "b30c2fa0-a3d3-4942-9596-bb6b245a238f" (UID: "b30c2fa0-a3d3-4942-9596-bb6b245a238f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:23:56.440846 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:56.440823 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b30c2fa0-a3d3-4942-9596-bb6b245a238f-workload-certs" pod "b30c2fa0-a3d3-4942-9596-bb6b245a238f" (UID: "b30c2fa0-a3d3-4942-9596-bb6b245a238f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:23:56.441121 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:56.441085 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b30c2fa0-a3d3-4942-9596-bb6b245a238f-istio-data\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:23:56.441121 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:56.441112 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b30c2fa0-a3d3-4942-9596-bb6b245a238f-istiod-ca-cert\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:23:56.441323 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:56.441126 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b30c2fa0-a3d3-4942-9596-bb6b245a238f-workload-certs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:23:56.441323 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:56.441138 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b30c2fa0-a3d3-4942-9596-bb6b245a238f-workload-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:23:56.441323 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:56.441150 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b30c2fa0-a3d3-4942-9596-bb6b245a238f-credential-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:23:56.449369 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:56.445858 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b30c2fa0-a3d3-4942-9596-bb6b245a238f-kube-api-access-8t5n5" pod "b30c2fa0-a3d3-4942-9596-bb6b245a238f" (UID: "b30c2fa0-a3d3-4942-9596-bb6b245a238f"). InnerVolumeSpecName "kube-api-access-8t5n5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:23:56.449369 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:56.446081 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b30c2fa0-a3d3-4942-9596-bb6b245a238f-istio-envoy" pod "b30c2fa0-a3d3-4942-9596-bb6b245a238f" (UID: "b30c2fa0-a3d3-4942-9596-bb6b245a238f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:23:56.449369 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:56.446092 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b30c2fa0-a3d3-4942-9596-bb6b245a238f-istio-token" pod "b30c2fa0-a3d3-4942-9596-bb6b245a238f" (UID: "b30c2fa0-a3d3-4942-9596-bb6b245a238f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:23:56.449369 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:56.446849 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b30c2fa0-a3d3-4942-9596-bb6b245a238f-istio-podinfo" pod "b30c2fa0-a3d3-4942-9596-bb6b245a238f" (UID: "b30c2fa0-a3d3-4942-9596-bb6b245a238f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:23:56.543195 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:56.543158 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b30c2fa0-a3d3-4942-9596-bb6b245a238f-istio-envoy\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:23:56.543195 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:56.543196 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b30c2fa0-a3d3-4942-9596-bb6b245a238f-istio-podinfo\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:23:56.543391 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:56.543209 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b30c2fa0-a3d3-4942-9596-bb6b245a238f-istio-token\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:23:56.543391 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:56.543224 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8t5n5\" (UniqueName: \"kubernetes.io/projected/b30c2fa0-a3d3-4942-9596-bb6b245a238f-kube-api-access-8t5n5\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:23:57.098963 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:57.098932 4493 generic.go:356] "Generic (PLEG): container finished" podID="b30c2fa0-a3d3-4942-9596-bb6b245a238f" containerID="75c694a4b9d0335d33b9544ceba1690e4f05a135e5cb26ce021dd775702cdaa9" exitCode=0 Oct 30 17:23:57.099164 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:57.098981 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br" Oct 30 17:23:57.099164 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:57.099023 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br" event={"ID":"b30c2fa0-a3d3-4942-9596-bb6b245a238f","Type":"ContainerDied","Data":"75c694a4b9d0335d33b9544ceba1690e4f05a135e5cb26ce021dd775702cdaa9"} Oct 30 17:23:57.099164 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:57.099082 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br" event={"ID":"b30c2fa0-a3d3-4942-9596-bb6b245a238f","Type":"ContainerDied","Data":"c74b4dcbb76dc9a177937c013c4f606af39c6fde1e6db5b9ced2987a107b141e"} Oct 30 17:23:57.099164 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:57.099110 4493 scope.go:122] "RemoveContainer" containerID="75c694a4b9d0335d33b9544ceba1690e4f05a135e5cb26ce021dd775702cdaa9" Oct 30 17:23:57.111548 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:57.111529 4493 scope.go:122] "RemoveContainer" containerID="75c694a4b9d0335d33b9544ceba1690e4f05a135e5cb26ce021dd775702cdaa9" Oct 30 17:23:57.111878 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:23:57.111844 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75c694a4b9d0335d33b9544ceba1690e4f05a135e5cb26ce021dd775702cdaa9\": container with ID starting with 75c694a4b9d0335d33b9544ceba1690e4f05a135e5cb26ce021dd775702cdaa9 not found: ID does not exist" containerID="75c694a4b9d0335d33b9544ceba1690e4f05a135e5cb26ce021dd775702cdaa9" Oct 30 17:23:57.111994 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:57.111882 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75c694a4b9d0335d33b9544ceba1690e4f05a135e5cb26ce021dd775702cdaa9"} err="failed to get container status \"75c694a4b9d0335d33b9544ceba1690e4f05a135e5cb26ce021dd775702cdaa9\": rpc error: code = NotFound desc = could not find container \"75c694a4b9d0335d33b9544ceba1690e4f05a135e5cb26ce021dd775702cdaa9\": container with ID starting with 75c694a4b9d0335d33b9544ceba1690e4f05a135e5cb26ce021dd775702cdaa9 not found: ID does not exist" Oct 30 17:23:57.163607 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:57.163571 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br"] Oct 30 17:23:57.168709 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:57.168678 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-wf6br"] Oct 30 17:23:58.953276 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:58.953233 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b30c2fa0-a3d3-4942-9596-bb6b245a238f" path="/var/lib/kubelet/pods/b30c2fa0-a3d3-4942-9596-bb6b245a238f/volumes" Oct 30 17:23:58.953729 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:58.953709 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e15d1f-9pk85"] Oct 30 17:23:58.953822 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:23:58.953754 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e15d1f-9pk85"] Oct 30 17:24:00.343927 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:24:00.343883 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 17:24:00.935864 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:00.935822 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d783148-4efc-4e09-b3d5-449262bd5647" path="/var/lib/kubelet/pods/9d783148-4efc-4e09-b3d5-449262bd5647/volumes" Oct 30 17:24:06.432144 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:06.432106 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6"] Oct 30 17:24:06.432656 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:06.432395 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6" podUID="ba3aa602-ca3b-4d37-9156-7c95b17a99cd" containerName="istio-proxy" containerID="cri-o://0eb9a58c2bc15c366b9af7e475d1124a7155c0befceb1171492d5d6143e969c2" gracePeriod=30 Oct 30 17:24:07.034384 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:07.034339 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-725577c0-cg2v9"] Oct 30 17:24:07.044337 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:07.044303 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-725577c0-cg2v9"] Oct 30 17:24:07.086322 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:07.086289 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-85eb87-bhqxk"] Oct 30 17:24:07.086945 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:07.086928 4493 cpu_manager.go:414] "containerMap: removing container" podUID="b30c2fa0-a3d3-4942-9596-bb6b245a238f" containerName="istio-proxy" Oct 30 17:24:07.086945 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:07.086945 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b30c2fa0-a3d3-4942-9596-bb6b245a238f" containerName="istio-proxy" Oct 30 17:24:07.087125 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:07.086971 4493 cpu_manager.go:414] "containerMap: removing container" podUID="a94190b0-d6ef-4227-9b06-d10cbf89c567" containerName="main" Oct 30 17:24:07.087125 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:07.086976 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a94190b0-d6ef-4227-9b06-d10cbf89c567" containerName="main" Oct 30 17:24:07.087125 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:07.087003 4493 cpu_manager.go:414] "containerMap: removing container" podUID="ac8c430b-0861-4ccd-b4ab-ac0a02716019" containerName="revoke-keys" Oct 30 17:24:07.087125 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:07.087008 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac8c430b-0861-4ccd-b4ab-ac0a02716019" containerName="revoke-keys" Oct 30 17:24:07.087125 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:07.087017 4493 cpu_manager.go:414] "containerMap: removing container" podUID="86707cad-e7dd-4a77-b04b-3bf99540cc18" containerName="main" Oct 30 17:24:07.087125 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:07.087022 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86707cad-e7dd-4a77-b04b-3bf99540cc18" containerName="main" Oct 30 17:24:07.087413 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:07.087138 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="a94190b0-d6ef-4227-9b06-d10cbf89c567" containerName="main" Oct 30 17:24:07.087413 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:07.087150 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac8c430b-0861-4ccd-b4ab-ac0a02716019" containerName="revoke-keys" Oct 30 17:24:07.087413 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:07.087159 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="86707cad-e7dd-4a77-b04b-3bf99540cc18" containerName="main" Oct 30 17:24:07.087413 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:07.087168 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="b30c2fa0-a3d3-4942-9596-bb6b245a238f" containerName="istio-proxy" Oct 30 17:24:07.092648 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:07.092619 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-85eb87-bhqxk" Oct 30 17:24:07.107583 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:07.107558 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:24:07.107737 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:07.107636 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:24:07.113990 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:07.113963 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-85eb87-bhqxk"] Oct 30 17:24:07.195744 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:07.195708 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/822b2b48-39df-417d-8985-647a0a0e1313-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-85eb87-bhqxk\" (UID: \"822b2b48-39df-417d-8985-647a0a0e1313\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-85eb87-bhqxk" Oct 30 17:24:07.196031 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:07.195772 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb7jk\" (UniqueName: \"kubernetes.io/projected/822b2b48-39df-417d-8985-647a0a0e1313-kube-api-access-kb7jk\") pod \"maas-api-revoke-keys-e2e-inf-b-85eb87-bhqxk\" (UID: \"822b2b48-39df-417d-8985-647a0a0e1313\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-85eb87-bhqxk" Oct 30 17:24:07.196031 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:07.195994 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/822b2b48-39df-417d-8985-647a0a0e1313-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-85eb87-bhqxk\" (UID: \"822b2b48-39df-417d-8985-647a0a0e1313\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-85eb87-bhqxk" Oct 30 17:24:07.197503 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:07.197462 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:24:07.197503 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:07.197476 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 17:24:07.297223 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:07.297129 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/822b2b48-39df-417d-8985-647a0a0e1313-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-85eb87-bhqxk\" (UID: \"822b2b48-39df-417d-8985-647a0a0e1313\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-85eb87-bhqxk" Oct 30 17:24:07.297409 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:07.297284 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/822b2b48-39df-417d-8985-647a0a0e1313-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-85eb87-bhqxk\" (UID: \"822b2b48-39df-417d-8985-647a0a0e1313\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-85eb87-bhqxk" Oct 30 17:24:07.297409 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:07.297368 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb7jk\" (UniqueName: \"kubernetes.io/projected/822b2b48-39df-417d-8985-647a0a0e1313-kube-api-access-kb7jk\") pod \"maas-api-revoke-keys-e2e-inf-b-85eb87-bhqxk\" (UID: \"822b2b48-39df-417d-8985-647a0a0e1313\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-85eb87-bhqxk" Oct 30 17:24:07.297886 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:07.297863 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/822b2b48-39df-417d-8985-647a0a0e1313-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-85eb87-bhqxk\" (UID: \"822b2b48-39df-417d-8985-647a0a0e1313\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-85eb87-bhqxk" Oct 30 17:24:07.299826 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:07.299803 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/822b2b48-39df-417d-8985-647a0a0e1313-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-85eb87-bhqxk\" (UID: \"822b2b48-39df-417d-8985-647a0a0e1313\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-85eb87-bhqxk" Oct 30 17:24:07.320368 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:07.320339 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb7jk\" (UniqueName: \"kubernetes.io/projected/822b2b48-39df-417d-8985-647a0a0e1313-kube-api-access-kb7jk\") pod \"maas-api-revoke-keys-e2e-inf-b-85eb87-bhqxk\" (UID: \"822b2b48-39df-417d-8985-647a0a0e1313\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-85eb87-bhqxk" Oct 30 17:24:07.408187 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:07.408150 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-85eb87-bhqxk" Oct 30 17:24:07.720051 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:07.720017 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-85eb87-bhqxk"] Oct 30 17:24:08.197048 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:08.197014 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-85eb87-bhqxk" event={"ID":"822b2b48-39df-417d-8985-647a0a0e1313","Type":"ContainerStarted","Data":"bd14a189bbb2b040d66d9faddd8b4d267c08a3f082ad85ca77fda7bed5de791f"} Oct 30 17:24:08.197235 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:08.197070 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-85eb87-bhqxk" event={"ID":"822b2b48-39df-417d-8985-647a0a0e1313","Type":"ContainerStarted","Data":"fed93ddd4c36dbb7ef13f3f3fc47e1cd2db323e1856bb6eb454e2831aab7a408"} Oct 30 17:24:08.238108 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:08.238047 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-85eb87-bhqxk" podStartSLOduration=1.238029512 podStartE2EDuration="1.238029512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:24:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:24:08.235937802 +0000 UTC m=+3675.463112901" watchObservedRunningTime="2026-10-30 17:24:08.238029512 +0000 UTC m=+3675.465204658" Oct 30 17:24:08.936209 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:08.936173 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04ae6844-f50a-401d-b174-3cc4edfd7d18" path="/var/lib/kubelet/pods/04ae6844-f50a-401d-b174-3cc4edfd7d18/volumes" Oct 30 17:24:09.206589 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:09.206550 4493 generic.go:356] "Generic (PLEG): container finished" podID="822b2b48-39df-417d-8985-647a0a0e1313" containerID="bd14a189bbb2b040d66d9faddd8b4d267c08a3f082ad85ca77fda7bed5de791f" exitCode=0 Oct 30 17:24:09.206854 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:09.206633 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-85eb87-bhqxk" event={"ID":"822b2b48-39df-417d-8985-647a0a0e1313","Type":"ContainerDied","Data":"bd14a189bbb2b040d66d9faddd8b4d267c08a3f082ad85ca77fda7bed5de791f"} Oct 30 17:24:10.403197 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:10.403172 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-85eb87-bhqxk" Oct 30 17:24:10.455719 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:10.455684 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/822b2b48-39df-417d-8985-647a0a0e1313-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/822b2b48-39df-417d-8985-647a0a0e1313-maas-api-openshift-ca\") pod \"822b2b48-39df-417d-8985-647a0a0e1313\" (UID: \"822b2b48-39df-417d-8985-647a0a0e1313\") " Oct 30 17:24:10.455944 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:10.455919 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/822b2b48-39df-417d-8985-647a0a0e1313-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/822b2b48-39df-417d-8985-647a0a0e1313-maas-api-xks-ca\") pod \"822b2b48-39df-417d-8985-647a0a0e1313\" (UID: \"822b2b48-39df-417d-8985-647a0a0e1313\") " Oct 30 17:24:10.456075 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:10.456057 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/822b2b48-39df-417d-8985-647a0a0e1313-kube-api-access-kb7jk\" (UniqueName: \"kubernetes.io/projected/822b2b48-39df-417d-8985-647a0a0e1313-kube-api-access-kb7jk\") pod \"822b2b48-39df-417d-8985-647a0a0e1313\" (UID: \"822b2b48-39df-417d-8985-647a0a0e1313\") " Oct 30 17:24:10.456139 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:10.456117 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/822b2b48-39df-417d-8985-647a0a0e1313-maas-api-openshift-ca" pod "822b2b48-39df-417d-8985-647a0a0e1313" (UID: "822b2b48-39df-417d-8985-647a0a0e1313"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:24:10.456650 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:10.456603 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/822b2b48-39df-417d-8985-647a0a0e1313-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:24:10.458887 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:10.458862 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/822b2b48-39df-417d-8985-647a0a0e1313-maas-api-xks-ca" pod "822b2b48-39df-417d-8985-647a0a0e1313" (UID: "822b2b48-39df-417d-8985-647a0a0e1313"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:24:10.458992 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:10.458948 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/822b2b48-39df-417d-8985-647a0a0e1313-kube-api-access-kb7jk" pod "822b2b48-39df-417d-8985-647a0a0e1313" (UID: "822b2b48-39df-417d-8985-647a0a0e1313"). InnerVolumeSpecName "kube-api-access-kb7jk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:24:10.557543 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:10.557512 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/822b2b48-39df-417d-8985-647a0a0e1313-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:24:10.557543 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:10.557541 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kb7jk\" (UniqueName: \"kubernetes.io/projected/822b2b48-39df-417d-8985-647a0a0e1313-kube-api-access-kb7jk\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:24:11.043404 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:11.043365 4493 patch_prober.go:29] interesting pod/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:24:11.043590 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:11.043422 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6" podUID="ba3aa602-ca3b-4d37-9156-7c95b17a99cd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:24:11.223834 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:11.223723 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-85eb87-bhqxk" event={"ID":"822b2b48-39df-417d-8985-647a0a0e1313","Type":"ContainerDied","Data":"fed93ddd4c36dbb7ef13f3f3fc47e1cd2db323e1856bb6eb454e2831aab7a408"} Oct 30 17:24:11.223834 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:11.223776 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fed93ddd4c36dbb7ef13f3f3fc47e1cd2db323e1856bb6eb454e2831aab7a408" Oct 30 17:24:11.223834 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:11.223742 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-85eb87-bhqxk" Oct 30 17:24:11.728891 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:11.728862 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6" Oct 30 17:24:11.775285 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:11.775246 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba3aa602-ca3b-4d37-9156-7c95b17a99cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba3aa602-ca3b-4d37-9156-7c95b17a99cd-istio-envoy\") pod \"ba3aa602-ca3b-4d37-9156-7c95b17a99cd\" (UID: \"ba3aa602-ca3b-4d37-9156-7c95b17a99cd\") " Oct 30 17:24:11.775476 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:11.775303 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ba3aa602-ca3b-4d37-9156-7c95b17a99cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba3aa602-ca3b-4d37-9156-7c95b17a99cd-istio-podinfo\") pod \"ba3aa602-ca3b-4d37-9156-7c95b17a99cd\" (UID: \"ba3aa602-ca3b-4d37-9156-7c95b17a99cd\") " Oct 30 17:24:11.775476 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:11.775394 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba3aa602-ca3b-4d37-9156-7c95b17a99cd-kube-api-access-6swd8\" (UniqueName: \"kubernetes.io/projected/ba3aa602-ca3b-4d37-9156-7c95b17a99cd-kube-api-access-6swd8\") pod \"ba3aa602-ca3b-4d37-9156-7c95b17a99cd\" (UID: \"ba3aa602-ca3b-4d37-9156-7c95b17a99cd\") " Oct 30 17:24:11.776137 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:11.775539 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba3aa602-ca3b-4d37-9156-7c95b17a99cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba3aa602-ca3b-4d37-9156-7c95b17a99cd-istio-data\") pod \"ba3aa602-ca3b-4d37-9156-7c95b17a99cd\" (UID: \"ba3aa602-ca3b-4d37-9156-7c95b17a99cd\") " Oct 30 17:24:11.776137 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:11.775624 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba3aa602-ca3b-4d37-9156-7c95b17a99cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba3aa602-ca3b-4d37-9156-7c95b17a99cd-istiod-ca-cert\") pod \"ba3aa602-ca3b-4d37-9156-7c95b17a99cd\" (UID: \"ba3aa602-ca3b-4d37-9156-7c95b17a99cd\") " Oct 30 17:24:11.776137 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:11.775663 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba3aa602-ca3b-4d37-9156-7c95b17a99cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba3aa602-ca3b-4d37-9156-7c95b17a99cd-credential-socket\") pod \"ba3aa602-ca3b-4d37-9156-7c95b17a99cd\" (UID: \"ba3aa602-ca3b-4d37-9156-7c95b17a99cd\") " Oct 30 17:24:11.776137 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:11.775704 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba3aa602-ca3b-4d37-9156-7c95b17a99cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba3aa602-ca3b-4d37-9156-7c95b17a99cd-workload-socket\") pod \"ba3aa602-ca3b-4d37-9156-7c95b17a99cd\" (UID: \"ba3aa602-ca3b-4d37-9156-7c95b17a99cd\") " Oct 30 17:24:11.776137 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:11.775768 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba3aa602-ca3b-4d37-9156-7c95b17a99cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba3aa602-ca3b-4d37-9156-7c95b17a99cd-workload-certs\") pod \"ba3aa602-ca3b-4d37-9156-7c95b17a99cd\" (UID: \"ba3aa602-ca3b-4d37-9156-7c95b17a99cd\") " Oct 30 17:24:11.776137 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:11.775834 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba3aa602-ca3b-4d37-9156-7c95b17a99cd-istio-token\" (UniqueName: \"kubernetes.io/projected/ba3aa602-ca3b-4d37-9156-7c95b17a99cd-istio-token\") pod \"ba3aa602-ca3b-4d37-9156-7c95b17a99cd\" (UID: \"ba3aa602-ca3b-4d37-9156-7c95b17a99cd\") " Oct 30 17:24:11.776137 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:11.775954 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba3aa602-ca3b-4d37-9156-7c95b17a99cd-istio-data" pod "ba3aa602-ca3b-4d37-9156-7c95b17a99cd" (UID: "ba3aa602-ca3b-4d37-9156-7c95b17a99cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:24:11.776592 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:11.776172 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba3aa602-ca3b-4d37-9156-7c95b17a99cd-credential-socket" pod "ba3aa602-ca3b-4d37-9156-7c95b17a99cd" (UID: "ba3aa602-ca3b-4d37-9156-7c95b17a99cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:24:11.776592 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:11.776500 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba3aa602-ca3b-4d37-9156-7c95b17a99cd-istio-data\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:24:11.776592 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:11.776522 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba3aa602-ca3b-4d37-9156-7c95b17a99cd-credential-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:24:11.777039 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:11.777010 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba3aa602-ca3b-4d37-9156-7c95b17a99cd-istiod-ca-cert" pod "ba3aa602-ca3b-4d37-9156-7c95b17a99cd" (UID: "ba3aa602-ca3b-4d37-9156-7c95b17a99cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:24:11.777146 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:11.777083 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba3aa602-ca3b-4d37-9156-7c95b17a99cd-workload-certs" pod "ba3aa602-ca3b-4d37-9156-7c95b17a99cd" (UID: "ba3aa602-ca3b-4d37-9156-7c95b17a99cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:24:11.777393 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:11.777366 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba3aa602-ca3b-4d37-9156-7c95b17a99cd-workload-socket" pod "ba3aa602-ca3b-4d37-9156-7c95b17a99cd" (UID: "ba3aa602-ca3b-4d37-9156-7c95b17a99cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:24:11.779938 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:11.779913 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba3aa602-ca3b-4d37-9156-7c95b17a99cd-istio-envoy" pod "ba3aa602-ca3b-4d37-9156-7c95b17a99cd" (UID: "ba3aa602-ca3b-4d37-9156-7c95b17a99cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:24:11.780031 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:11.779970 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba3aa602-ca3b-4d37-9156-7c95b17a99cd-istio-token" pod "ba3aa602-ca3b-4d37-9156-7c95b17a99cd" (UID: "ba3aa602-ca3b-4d37-9156-7c95b17a99cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:24:11.780623 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:11.780598 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba3aa602-ca3b-4d37-9156-7c95b17a99cd-istio-podinfo" pod "ba3aa602-ca3b-4d37-9156-7c95b17a99cd" (UID: "ba3aa602-ca3b-4d37-9156-7c95b17a99cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:24:11.782308 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:11.782284 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba3aa602-ca3b-4d37-9156-7c95b17a99cd-kube-api-access-6swd8" pod "ba3aa602-ca3b-4d37-9156-7c95b17a99cd" (UID: "ba3aa602-ca3b-4d37-9156-7c95b17a99cd"). InnerVolumeSpecName "kube-api-access-6swd8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:24:11.882062 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:11.881372 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6swd8\" (UniqueName: \"kubernetes.io/projected/ba3aa602-ca3b-4d37-9156-7c95b17a99cd-kube-api-access-6swd8\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:24:11.882062 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:11.881418 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba3aa602-ca3b-4d37-9156-7c95b17a99cd-istiod-ca-cert\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:24:11.882062 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:11.881433 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba3aa602-ca3b-4d37-9156-7c95b17a99cd-workload-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:24:11.882062 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:11.881448 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba3aa602-ca3b-4d37-9156-7c95b17a99cd-workload-certs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:24:11.882062 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:11.881463 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba3aa602-ca3b-4d37-9156-7c95b17a99cd-istio-token\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:24:11.882062 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:11.881475 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba3aa602-ca3b-4d37-9156-7c95b17a99cd-istio-envoy\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:24:11.882062 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:11.881487 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba3aa602-ca3b-4d37-9156-7c95b17a99cd-istio-podinfo\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:24:12.237205 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:12.237169 4493 generic.go:356] "Generic (PLEG): container finished" podID="ba3aa602-ca3b-4d37-9156-7c95b17a99cd" containerID="0eb9a58c2bc15c366b9af7e475d1124a7155c0befceb1171492d5d6143e969c2" exitCode=0 Oct 30 17:24:12.237397 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:12.237236 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6" Oct 30 17:24:12.237397 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:12.237258 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6" event={"ID":"ba3aa602-ca3b-4d37-9156-7c95b17a99cd","Type":"ContainerDied","Data":"0eb9a58c2bc15c366b9af7e475d1124a7155c0befceb1171492d5d6143e969c2"} Oct 30 17:24:12.237397 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:12.237321 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6" event={"ID":"ba3aa602-ca3b-4d37-9156-7c95b17a99cd","Type":"ContainerDied","Data":"a8ca382523d5cf30997edd3a04af93a751610bf00b9f9aea0fa50f186c099de4"} Oct 30 17:24:12.237397 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:12.237348 4493 scope.go:122] "RemoveContainer" containerID="0eb9a58c2bc15c366b9af7e475d1124a7155c0befceb1171492d5d6143e969c2" Oct 30 17:24:12.252070 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:12.252042 4493 scope.go:122] "RemoveContainer" containerID="33f6e9b7a3fe74fad17b82952f1bd31a08384ffaf3a4a9c54df250e09ebb4b54" Oct 30 17:24:12.266713 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:12.266693 4493 scope.go:122] "RemoveContainer" containerID="0eb9a58c2bc15c366b9af7e475d1124a7155c0befceb1171492d5d6143e969c2" Oct 30 17:24:12.267059 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:24:12.267035 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0eb9a58c2bc15c366b9af7e475d1124a7155c0befceb1171492d5d6143e969c2\": container with ID starting with 0eb9a58c2bc15c366b9af7e475d1124a7155c0befceb1171492d5d6143e969c2 not found: ID does not exist" containerID="0eb9a58c2bc15c366b9af7e475d1124a7155c0befceb1171492d5d6143e969c2" Oct 30 17:24:12.267164 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:12.267066 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0eb9a58c2bc15c366b9af7e475d1124a7155c0befceb1171492d5d6143e969c2"} err="failed to get container status \"0eb9a58c2bc15c366b9af7e475d1124a7155c0befceb1171492d5d6143e969c2\": rpc error: code = NotFound desc = could not find container \"0eb9a58c2bc15c366b9af7e475d1124a7155c0befceb1171492d5d6143e969c2\": container with ID starting with 0eb9a58c2bc15c366b9af7e475d1124a7155c0befceb1171492d5d6143e969c2 not found: ID does not exist" Oct 30 17:24:12.267164 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:12.267083 4493 scope.go:122] "RemoveContainer" containerID="33f6e9b7a3fe74fad17b82952f1bd31a08384ffaf3a4a9c54df250e09ebb4b54" Oct 30 17:24:12.267366 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:24:12.267351 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33f6e9b7a3fe74fad17b82952f1bd31a08384ffaf3a4a9c54df250e09ebb4b54\": container with ID starting with 33f6e9b7a3fe74fad17b82952f1bd31a08384ffaf3a4a9c54df250e09ebb4b54 not found: ID does not exist" containerID="33f6e9b7a3fe74fad17b82952f1bd31a08384ffaf3a4a9c54df250e09ebb4b54" Oct 30 17:24:12.267422 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:12.267370 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33f6e9b7a3fe74fad17b82952f1bd31a08384ffaf3a4a9c54df250e09ebb4b54"} err="failed to get container status \"33f6e9b7a3fe74fad17b82952f1bd31a08384ffaf3a4a9c54df250e09ebb4b54\": rpc error: code = NotFound desc = could not find container \"33f6e9b7a3fe74fad17b82952f1bd31a08384ffaf3a4a9c54df250e09ebb4b54\": container with ID starting with 33f6e9b7a3fe74fad17b82952f1bd31a08384ffaf3a4a9c54df250e09ebb4b54 not found: ID does not exist" Oct 30 17:24:12.304301 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:12.304273 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6"] Oct 30 17:24:12.313961 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:12.313932 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-e15d1f-openshift-default-55c7d89dc9-4g9k6"] Oct 30 17:24:12.933658 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:12.933628 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba3aa602-ca3b-4d37-9156-7c95b17a99cd" path="/var/lib/kubelet/pods/ba3aa602-ca3b-4d37-9156-7c95b17a99cd/volumes" Oct 30 17:24:26.606814 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:26.603924 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8bff6e-zps26"] Oct 30 17:24:26.750462 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:26.750420 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8bff6e-zps26"] Oct 30 17:24:26.999841 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:26.999189 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7521ace7-b975-4772-a31d-75b30cf36283" path="/var/lib/kubelet/pods/7521ace7-b975-4772-a31d-75b30cf36283/volumes" Oct 30 17:24:27.170347 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:27.170311 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6e08bd-cg4q6"] Oct 30 17:24:27.188212 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:27.188051 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6e08bd-cg4q6"] Oct 30 17:24:28.501144 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:28.501109 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 17:24:28.939796 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:28.939754 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b56c14bc-2ce9-4e40-9a24-f86e3e6f79d3" path="/var/lib/kubelet/pods/b56c14bc-2ce9-4e40-9a24-f86e3e6f79d3/volumes" Oct 30 17:24:29.562617 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:29.562580 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9afbb2-6njdf"] Oct 30 17:24:29.575563 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:29.575516 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9afbb2-6njdf"] Oct 30 17:24:30.343253 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:24:30.343211 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 17:24:30.934138 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:30.934097 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac8c430b-0861-4ccd-b4ab-ac0a02716019" path="/var/lib/kubelet/pods/ac8c430b-0861-4ccd-b4ab-ac0a02716019/volumes" Oct 30 17:24:34.808596 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:34.808558 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2d365c-skfpg"] Oct 30 17:24:34.809290 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:34.809271 4493 cpu_manager.go:414] "containerMap: removing container" podUID="ba3aa602-ca3b-4d37-9156-7c95b17a99cd" containerName="istio-proxy" Oct 30 17:24:34.809343 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:34.809292 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba3aa602-ca3b-4d37-9156-7c95b17a99cd" containerName="istio-proxy" Oct 30 17:24:34.809343 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:34.809308 4493 cpu_manager.go:414] "containerMap: removing container" podUID="822b2b48-39df-417d-8985-647a0a0e1313" containerName="revoke-keys" Oct 30 17:24:34.809343 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:34.809313 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="822b2b48-39df-417d-8985-647a0a0e1313" containerName="revoke-keys" Oct 30 17:24:34.809451 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:34.809438 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba3aa602-ca3b-4d37-9156-7c95b17a99cd" containerName="istio-proxy" Oct 30 17:24:34.809451 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:34.809450 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="822b2b48-39df-417d-8985-647a0a0e1313" containerName="revoke-keys" Oct 30 17:24:34.813521 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:34.813499 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2d365c-skfpg" Oct 30 17:24:34.826095 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:34.826059 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2d365c-skfpg"] Oct 30 17:24:34.831864 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:34.831829 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:24:34.831864 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:34.831845 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:24:34.848156 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:34.848133 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:24:34.848376 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:34.848359 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 17:24:34.913690 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:34.913348 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qm9x\" (UniqueName: \"kubernetes.io/projected/47bf14f7-1ba3-4416-a1ff-2bf301c2e98d-kube-api-access-8qm9x\") pod \"maas-api-revoke-keys-e2e-worker-w3-2d365c-skfpg\" (UID: \"47bf14f7-1ba3-4416-a1ff-2bf301c2e98d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2d365c-skfpg" Oct 30 17:24:34.913690 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:34.913586 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47bf14f7-1ba3-4416-a1ff-2bf301c2e98d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-2d365c-skfpg\" (UID: \"47bf14f7-1ba3-4416-a1ff-2bf301c2e98d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2d365c-skfpg" Oct 30 17:24:34.914128 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:34.913879 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47bf14f7-1ba3-4416-a1ff-2bf301c2e98d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-2d365c-skfpg\" (UID: \"47bf14f7-1ba3-4416-a1ff-2bf301c2e98d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2d365c-skfpg" Oct 30 17:24:35.015199 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:35.015154 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47bf14f7-1ba3-4416-a1ff-2bf301c2e98d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-2d365c-skfpg\" (UID: \"47bf14f7-1ba3-4416-a1ff-2bf301c2e98d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2d365c-skfpg" Oct 30 17:24:35.015408 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:35.015282 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47bf14f7-1ba3-4416-a1ff-2bf301c2e98d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-2d365c-skfpg\" (UID: \"47bf14f7-1ba3-4416-a1ff-2bf301c2e98d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2d365c-skfpg" Oct 30 17:24:35.015488 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:35.015456 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qm9x\" (UniqueName: \"kubernetes.io/projected/47bf14f7-1ba3-4416-a1ff-2bf301c2e98d-kube-api-access-8qm9x\") pod \"maas-api-revoke-keys-e2e-worker-w3-2d365c-skfpg\" (UID: \"47bf14f7-1ba3-4416-a1ff-2bf301c2e98d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2d365c-skfpg" Oct 30 17:24:35.016276 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:35.016248 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47bf14f7-1ba3-4416-a1ff-2bf301c2e98d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-2d365c-skfpg\" (UID: \"47bf14f7-1ba3-4416-a1ff-2bf301c2e98d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2d365c-skfpg" Oct 30 17:24:35.017903 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:35.017873 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47bf14f7-1ba3-4416-a1ff-2bf301c2e98d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-2d365c-skfpg\" (UID: \"47bf14f7-1ba3-4416-a1ff-2bf301c2e98d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2d365c-skfpg" Oct 30 17:24:35.038196 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:35.038169 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qm9x\" (UniqueName: \"kubernetes.io/projected/47bf14f7-1ba3-4416-a1ff-2bf301c2e98d-kube-api-access-8qm9x\") pod \"maas-api-revoke-keys-e2e-worker-w3-2d365c-skfpg\" (UID: \"47bf14f7-1ba3-4416-a1ff-2bf301c2e98d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2d365c-skfpg" Oct 30 17:24:35.133577 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:35.133496 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2d365c-skfpg" Oct 30 17:24:35.699810 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:35.699487 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2d365c-skfpg"] Oct 30 17:24:35.722922 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:35.722891 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bbc36f-wqwqs"] Oct 30 17:24:35.723815 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:35.723770 4493 cpu_manager.go:414] "containerMap: removing container" podUID="ba3aa602-ca3b-4d37-9156-7c95b17a99cd" containerName="istio-proxy" Oct 30 17:24:35.723815 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:35.723811 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba3aa602-ca3b-4d37-9156-7c95b17a99cd" containerName="istio-proxy" Oct 30 17:24:35.723985 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:35.723974 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba3aa602-ca3b-4d37-9156-7c95b17a99cd" containerName="istio-proxy" Oct 30 17:24:35.729074 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:35.729041 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bbc36f-wqwqs" Oct 30 17:24:35.750009 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:35.749987 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bbc36f-wqwqs"] Oct 30 17:24:35.891141 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:35.891099 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb00f070-a308-43a8-866a-438505dc7600-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-bbc36f-wqwqs\" (UID: \"fb00f070-a308-43a8-866a-438505dc7600\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bbc36f-wqwqs" Oct 30 17:24:35.891543 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:35.891255 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb00f070-a308-43a8-866a-438505dc7600-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-bbc36f-wqwqs\" (UID: \"fb00f070-a308-43a8-866a-438505dc7600\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bbc36f-wqwqs" Oct 30 17:24:35.891543 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:35.891313 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67bwf\" (UniqueName: \"kubernetes.io/projected/fb00f070-a308-43a8-866a-438505dc7600-kube-api-access-67bwf\") pod \"maas-api-revoke-keys-e2e-inf-b-bbc36f-wqwqs\" (UID: \"fb00f070-a308-43a8-866a-438505dc7600\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bbc36f-wqwqs" Oct 30 17:24:35.950922 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:35.950841 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2d365c-skfpg" event={"ID":"47bf14f7-1ba3-4416-a1ff-2bf301c2e98d","Type":"ContainerStarted","Data":"e7d7e578a77e728a38893d04e42ac4c72a68431f9b9aeec3c444e84a09bbb025"} Oct 30 17:24:35.992747 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:35.992709 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb00f070-a308-43a8-866a-438505dc7600-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-bbc36f-wqwqs\" (UID: \"fb00f070-a308-43a8-866a-438505dc7600\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bbc36f-wqwqs" Oct 30 17:24:35.992951 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:35.992811 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67bwf\" (UniqueName: \"kubernetes.io/projected/fb00f070-a308-43a8-866a-438505dc7600-kube-api-access-67bwf\") pod \"maas-api-revoke-keys-e2e-inf-b-bbc36f-wqwqs\" (UID: \"fb00f070-a308-43a8-866a-438505dc7600\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bbc36f-wqwqs" Oct 30 17:24:35.993005 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:35.992968 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb00f070-a308-43a8-866a-438505dc7600-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-bbc36f-wqwqs\" (UID: \"fb00f070-a308-43a8-866a-438505dc7600\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bbc36f-wqwqs" Oct 30 17:24:35.993669 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:35.993588 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb00f070-a308-43a8-866a-438505dc7600-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-bbc36f-wqwqs\" (UID: \"fb00f070-a308-43a8-866a-438505dc7600\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bbc36f-wqwqs" Oct 30 17:24:35.995351 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:35.995331 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb00f070-a308-43a8-866a-438505dc7600-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-bbc36f-wqwqs\" (UID: \"fb00f070-a308-43a8-866a-438505dc7600\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bbc36f-wqwqs" Oct 30 17:24:36.015588 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:36.015563 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67bwf\" (UniqueName: \"kubernetes.io/projected/fb00f070-a308-43a8-866a-438505dc7600-kube-api-access-67bwf\") pod \"maas-api-revoke-keys-e2e-inf-b-bbc36f-wqwqs\" (UID: \"fb00f070-a308-43a8-866a-438505dc7600\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bbc36f-wqwqs" Oct 30 17:24:36.065649 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:36.065617 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bbc36f-wqwqs" Oct 30 17:24:36.384765 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:36.384733 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bbc36f-wqwqs"] Oct 30 17:24:36.385944 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:24:36.385921 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb00f070_a308_43a8_866a_438505dc7600.slice/crio-9e5f8d9ec421ac90a776822ccc4b5211dc362311a062f046be30fe8e3bdd50d3 WatchSource:0}: Error finding container 9e5f8d9ec421ac90a776822ccc4b5211dc362311a062f046be30fe8e3bdd50d3: Status 404 returned error can't find the container with id 9e5f8d9ec421ac90a776822ccc4b5211dc362311a062f046be30fe8e3bdd50d3 Oct 30 17:24:36.972160 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:36.972077 4493 generic.go:356] "Generic (PLEG): container finished" podID="47bf14f7-1ba3-4416-a1ff-2bf301c2e98d" containerID="ada67f7ce3d0784e0aa99aadd007419be124d442cadf3cc5646e5f8d751b1b04" exitCode=0 Oct 30 17:24:36.972644 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:36.972242 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2d365c-skfpg" event={"ID":"47bf14f7-1ba3-4416-a1ff-2bf301c2e98d","Type":"ContainerDied","Data":"ada67f7ce3d0784e0aa99aadd007419be124d442cadf3cc5646e5f8d751b1b04"} Oct 30 17:24:36.977503 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:36.975165 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bbc36f-wqwqs" event={"ID":"fb00f070-a308-43a8-866a-438505dc7600","Type":"ContainerStarted","Data":"544621adfd9cabbda23b23e68b0df2ac7faf0a99a8efa3c593182522ca8117d2"} Oct 30 17:24:36.977503 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:36.975220 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bbc36f-wqwqs" event={"ID":"fb00f070-a308-43a8-866a-438505dc7600","Type":"ContainerStarted","Data":"9e5f8d9ec421ac90a776822ccc4b5211dc362311a062f046be30fe8e3bdd50d3"} Oct 30 17:24:36.982862 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:36.982761 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x"] Oct 30 17:24:36.983301 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:36.983238 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x" podUID="273838d6-034c-4763-a75f-5a357a4953f4" containerName="istio-proxy" containerID="cri-o://f9eba3fa07ffe13d21325f081b011b36a7f097ec03725b08099c1f83423407e4" gracePeriod=30 Oct 30 17:24:37.002880 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:37.002841 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt"] Oct 30 17:24:37.003140 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:37.003087 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt" podUID="46c8b953-69a7-4e2e-8695-a0b9d745529f" containerName="istio-proxy" containerID="cri-o://83c7dfe4cc1f60ab3d1349a28f17511fe13e8aa88b83ef1ef858ddb9d83a0559" gracePeriod=30 Oct 30 17:24:37.162809 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:37.157766 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bbc36f-wqwqs" podStartSLOduration=2.157744514 podStartE2EDuration="2.157744514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:24:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:24:37.139039669 +0000 UTC m=+3704.366214779" watchObservedRunningTime="2026-10-30 17:24:37.157744514 +0000 UTC m=+3704.384919614" Oct 30 17:24:37.564150 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:37.564119 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4c6596-xdd2z"] Oct 30 17:24:37.568461 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:37.568429 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4c6596-xdd2z" Oct 30 17:24:37.591007 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:37.590963 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4c6596-xdd2z"] Oct 30 17:24:37.722775 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:37.722690 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/335a2fc4-633e-4cbb-9d6b-d2a52e435ee4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4c6596-xdd2z\" (UID: \"335a2fc4-633e-4cbb-9d6b-d2a52e435ee4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4c6596-xdd2z" Oct 30 17:24:37.723056 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:37.723033 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhvlc\" (UniqueName: \"kubernetes.io/projected/335a2fc4-633e-4cbb-9d6b-d2a52e435ee4-kube-api-access-rhvlc\") pod \"maas-api-revoke-keys-e2e-inf-b-4c6596-xdd2z\" (UID: \"335a2fc4-633e-4cbb-9d6b-d2a52e435ee4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4c6596-xdd2z" Oct 30 17:24:37.723196 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:37.723183 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/335a2fc4-633e-4cbb-9d6b-d2a52e435ee4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4c6596-xdd2z\" (UID: \"335a2fc4-633e-4cbb-9d6b-d2a52e435ee4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4c6596-xdd2z" Oct 30 17:24:37.824118 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:37.824083 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhvlc\" (UniqueName: \"kubernetes.io/projected/335a2fc4-633e-4cbb-9d6b-d2a52e435ee4-kube-api-access-rhvlc\") pod \"maas-api-revoke-keys-e2e-inf-b-4c6596-xdd2z\" (UID: \"335a2fc4-633e-4cbb-9d6b-d2a52e435ee4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4c6596-xdd2z" Oct 30 17:24:37.824296 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:37.824203 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/335a2fc4-633e-4cbb-9d6b-d2a52e435ee4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4c6596-xdd2z\" (UID: \"335a2fc4-633e-4cbb-9d6b-d2a52e435ee4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4c6596-xdd2z" Oct 30 17:24:37.824296 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:37.824255 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/335a2fc4-633e-4cbb-9d6b-d2a52e435ee4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4c6596-xdd2z\" (UID: \"335a2fc4-633e-4cbb-9d6b-d2a52e435ee4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4c6596-xdd2z" Oct 30 17:24:37.824860 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:37.824816 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/335a2fc4-633e-4cbb-9d6b-d2a52e435ee4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4c6596-xdd2z\" (UID: \"335a2fc4-633e-4cbb-9d6b-d2a52e435ee4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4c6596-xdd2z" Oct 30 17:24:37.826626 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:37.826604 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/335a2fc4-633e-4cbb-9d6b-d2a52e435ee4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4c6596-xdd2z\" (UID: \"335a2fc4-633e-4cbb-9d6b-d2a52e435ee4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4c6596-xdd2z" Oct 30 17:24:37.848827 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:37.848779 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhvlc\" (UniqueName: \"kubernetes.io/projected/335a2fc4-633e-4cbb-9d6b-d2a52e435ee4-kube-api-access-rhvlc\") pod \"maas-api-revoke-keys-e2e-inf-b-4c6596-xdd2z\" (UID: \"335a2fc4-633e-4cbb-9d6b-d2a52e435ee4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4c6596-xdd2z" Oct 30 17:24:37.883859 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:37.883826 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4c6596-xdd2z" Oct 30 17:24:37.990950 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:37.990869 4493 generic.go:356] "Generic (PLEG): container finished" podID="fb00f070-a308-43a8-866a-438505dc7600" containerID="544621adfd9cabbda23b23e68b0df2ac7faf0a99a8efa3c593182522ca8117d2" exitCode=0 Oct 30 17:24:37.991594 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:37.991570 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bbc36f-wqwqs" event={"ID":"fb00f070-a308-43a8-866a-438505dc7600","Type":"ContainerDied","Data":"544621adfd9cabbda23b23e68b0df2ac7faf0a99a8efa3c593182522ca8117d2"} Oct 30 17:24:38.174029 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:38.174005 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2d365c-skfpg" Oct 30 17:24:38.199293 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:38.199263 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4c6596-xdd2z"] Oct 30 17:24:38.199917 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:24:38.199875 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod335a2fc4_633e_4cbb_9d6b_d2a52e435ee4.slice/crio-6accba5f0306275681e1da9054f47fb0fd6ae1f4775b6a9e07d33de535aa4c1c WatchSource:0}: Error finding container 6accba5f0306275681e1da9054f47fb0fd6ae1f4775b6a9e07d33de535aa4c1c: Status 404 returned error can't find the container with id 6accba5f0306275681e1da9054f47fb0fd6ae1f4775b6a9e07d33de535aa4c1c Oct 30 17:24:38.333985 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:38.333958 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47bf14f7-1ba3-4416-a1ff-2bf301c2e98d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47bf14f7-1ba3-4416-a1ff-2bf301c2e98d-maas-api-openshift-ca\") pod \"47bf14f7-1ba3-4416-a1ff-2bf301c2e98d\" (UID: \"47bf14f7-1ba3-4416-a1ff-2bf301c2e98d\") " Oct 30 17:24:38.334157 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:38.334010 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47bf14f7-1ba3-4416-a1ff-2bf301c2e98d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47bf14f7-1ba3-4416-a1ff-2bf301c2e98d-maas-api-xks-ca\") pod \"47bf14f7-1ba3-4416-a1ff-2bf301c2e98d\" (UID: \"47bf14f7-1ba3-4416-a1ff-2bf301c2e98d\") " Oct 30 17:24:38.334202 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:38.334168 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47bf14f7-1ba3-4416-a1ff-2bf301c2e98d-kube-api-access-8qm9x\" (UniqueName: \"kubernetes.io/projected/47bf14f7-1ba3-4416-a1ff-2bf301c2e98d-kube-api-access-8qm9x\") pod \"47bf14f7-1ba3-4416-a1ff-2bf301c2e98d\" (UID: \"47bf14f7-1ba3-4416-a1ff-2bf301c2e98d\") " Oct 30 17:24:38.334322 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:38.334297 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47bf14f7-1ba3-4416-a1ff-2bf301c2e98d-maas-api-openshift-ca" pod "47bf14f7-1ba3-4416-a1ff-2bf301c2e98d" (UID: "47bf14f7-1ba3-4416-a1ff-2bf301c2e98d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:24:38.334773 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:38.334752 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47bf14f7-1ba3-4416-a1ff-2bf301c2e98d-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:24:38.336334 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:38.336303 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47bf14f7-1ba3-4416-a1ff-2bf301c2e98d-maas-api-xks-ca" pod "47bf14f7-1ba3-4416-a1ff-2bf301c2e98d" (UID: "47bf14f7-1ba3-4416-a1ff-2bf301c2e98d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:24:38.336431 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:38.336366 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47bf14f7-1ba3-4416-a1ff-2bf301c2e98d-kube-api-access-8qm9x" pod "47bf14f7-1ba3-4416-a1ff-2bf301c2e98d" (UID: "47bf14f7-1ba3-4416-a1ff-2bf301c2e98d"). InnerVolumeSpecName "kube-api-access-8qm9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:24:38.435977 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:38.435929 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47bf14f7-1ba3-4416-a1ff-2bf301c2e98d-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:24:38.435977 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:38.435972 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qm9x\" (UniqueName: \"kubernetes.io/projected/47bf14f7-1ba3-4416-a1ff-2bf301c2e98d-kube-api-access-8qm9x\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:24:38.999864 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:38.999826 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4c6596-xdd2z" event={"ID":"335a2fc4-633e-4cbb-9d6b-d2a52e435ee4","Type":"ContainerStarted","Data":"79fa4153c32ccb452843128262a9a0c65421122c019d9bbfc9fd9be2dd9167d8"} Oct 30 17:24:39.000317 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:38.999886 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4c6596-xdd2z" event={"ID":"335a2fc4-633e-4cbb-9d6b-d2a52e435ee4","Type":"ContainerStarted","Data":"6accba5f0306275681e1da9054f47fb0fd6ae1f4775b6a9e07d33de535aa4c1c"} Oct 30 17:24:39.001356 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:39.001336 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2d365c-skfpg" Oct 30 17:24:39.001471 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:39.001378 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2d365c-skfpg" event={"ID":"47bf14f7-1ba3-4416-a1ff-2bf301c2e98d","Type":"ContainerDied","Data":"e7d7e578a77e728a38893d04e42ac4c72a68431f9b9aeec3c444e84a09bbb025"} Oct 30 17:24:39.001471 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:39.001429 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7d7e578a77e728a38893d04e42ac4c72a68431f9b9aeec3c444e84a09bbb025" Oct 30 17:24:39.045054 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:39.045002 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4c6596-xdd2z" podStartSLOduration=2.04498869 podStartE2EDuration="2.04498869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:24:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:24:39.041294297 +0000 UTC m=+3706.268469396" watchObservedRunningTime="2026-10-30 17:24:39.04498869 +0000 UTC m=+3706.272163790" Oct 30 17:24:39.172662 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:39.172636 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bbc36f-wqwqs" Oct 30 17:24:39.248060 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:39.246761 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb00f070-a308-43a8-866a-438505dc7600-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb00f070-a308-43a8-866a-438505dc7600-maas-api-openshift-ca\") pod \"fb00f070-a308-43a8-866a-438505dc7600\" (UID: \"fb00f070-a308-43a8-866a-438505dc7600\") " Oct 30 17:24:39.248060 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:39.246988 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb00f070-a308-43a8-866a-438505dc7600-kube-api-access-67bwf\" (UniqueName: \"kubernetes.io/projected/fb00f070-a308-43a8-866a-438505dc7600-kube-api-access-67bwf\") pod \"fb00f070-a308-43a8-866a-438505dc7600\" (UID: \"fb00f070-a308-43a8-866a-438505dc7600\") " Oct 30 17:24:39.248060 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:39.247041 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb00f070-a308-43a8-866a-438505dc7600-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb00f070-a308-43a8-866a-438505dc7600-maas-api-xks-ca\") pod \"fb00f070-a308-43a8-866a-438505dc7600\" (UID: \"fb00f070-a308-43a8-866a-438505dc7600\") " Oct 30 17:24:39.249093 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:39.249066 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb00f070-a308-43a8-866a-438505dc7600-maas-api-openshift-ca" pod "fb00f070-a308-43a8-866a-438505dc7600" (UID: "fb00f070-a308-43a8-866a-438505dc7600"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:24:39.251638 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:39.251603 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb00f070-a308-43a8-866a-438505dc7600-kube-api-access-67bwf" pod "fb00f070-a308-43a8-866a-438505dc7600" (UID: "fb00f070-a308-43a8-866a-438505dc7600"). InnerVolumeSpecName "kube-api-access-67bwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:24:39.254132 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:39.254104 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb00f070-a308-43a8-866a-438505dc7600-maas-api-xks-ca" pod "fb00f070-a308-43a8-866a-438505dc7600" (UID: "fb00f070-a308-43a8-866a-438505dc7600"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:24:39.348700 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:39.348668 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67bwf\" (UniqueName: \"kubernetes.io/projected/fb00f070-a308-43a8-866a-438505dc7600-kube-api-access-67bwf\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:24:39.348700 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:39.348696 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb00f070-a308-43a8-866a-438505dc7600-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:24:39.348700 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:39.348706 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb00f070-a308-43a8-866a-438505dc7600-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:24:39.502750 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:39.502669 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw"] Oct 30 17:24:39.529641 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:39.529608 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl"] Oct 30 17:24:40.022550 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:40.022515 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bbc36f-wqwqs" Oct 30 17:24:40.022550 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:40.022526 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bbc36f-wqwqs" event={"ID":"fb00f070-a308-43a8-866a-438505dc7600","Type":"ContainerDied","Data":"9e5f8d9ec421ac90a776822ccc4b5211dc362311a062f046be30fe8e3bdd50d3"} Oct 30 17:24:40.023069 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:40.022602 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e5f8d9ec421ac90a776822ccc4b5211dc362311a062f046be30fe8e3bdd50d3" Oct 30 17:24:40.027603 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:40.027574 4493 generic.go:356] "Generic (PLEG): container finished" podID="335a2fc4-633e-4cbb-9d6b-d2a52e435ee4" containerID="79fa4153c32ccb452843128262a9a0c65421122c019d9bbfc9fd9be2dd9167d8" exitCode=0 Oct 30 17:24:40.027729 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:40.027654 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4c6596-xdd2z" event={"ID":"335a2fc4-633e-4cbb-9d6b-d2a52e435ee4","Type":"ContainerDied","Data":"79fa4153c32ccb452843128262a9a0c65421122c019d9bbfc9fd9be2dd9167d8"} Oct 30 17:24:40.733512 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:40.733476 4493 patch_prober.go:29] interesting pod/e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:24:40.733716 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:40.733538 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt" podUID="46c8b953-69a7-4e2e-8695-a0b9d745529f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:24:40.823310 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:40.823274 4493 patch_prober.go:29] interesting pod/e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:24:40.823494 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:40.823338 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x" podUID="273838d6-034c-4763-a75f-5a357a4953f4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:24:41.213553 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:41.213523 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4c6596-xdd2z" Oct 30 17:24:41.277587 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:41.277554 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/335a2fc4-633e-4cbb-9d6b-d2a52e435ee4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/335a2fc4-633e-4cbb-9d6b-d2a52e435ee4-maas-api-xks-ca\") pod \"335a2fc4-633e-4cbb-9d6b-d2a52e435ee4\" (UID: \"335a2fc4-633e-4cbb-9d6b-d2a52e435ee4\") " Oct 30 17:24:41.277830 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:41.277636 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/335a2fc4-633e-4cbb-9d6b-d2a52e435ee4-kube-api-access-rhvlc\" (UniqueName: \"kubernetes.io/projected/335a2fc4-633e-4cbb-9d6b-d2a52e435ee4-kube-api-access-rhvlc\") pod \"335a2fc4-633e-4cbb-9d6b-d2a52e435ee4\" (UID: \"335a2fc4-633e-4cbb-9d6b-d2a52e435ee4\") " Oct 30 17:24:41.277830 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:41.277729 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/335a2fc4-633e-4cbb-9d6b-d2a52e435ee4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/335a2fc4-633e-4cbb-9d6b-d2a52e435ee4-maas-api-openshift-ca\") pod \"335a2fc4-633e-4cbb-9d6b-d2a52e435ee4\" (UID: \"335a2fc4-633e-4cbb-9d6b-d2a52e435ee4\") " Oct 30 17:24:41.278420 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:41.278370 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/335a2fc4-633e-4cbb-9d6b-d2a52e435ee4-maas-api-openshift-ca" pod "335a2fc4-633e-4cbb-9d6b-d2a52e435ee4" (UID: "335a2fc4-633e-4cbb-9d6b-d2a52e435ee4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:24:41.280434 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:41.280398 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/335a2fc4-633e-4cbb-9d6b-d2a52e435ee4-kube-api-access-rhvlc" pod "335a2fc4-633e-4cbb-9d6b-d2a52e435ee4" (UID: "335a2fc4-633e-4cbb-9d6b-d2a52e435ee4"). InnerVolumeSpecName "kube-api-access-rhvlc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:24:41.280708 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:41.280680 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/335a2fc4-633e-4cbb-9d6b-d2a52e435ee4-maas-api-xks-ca" pod "335a2fc4-633e-4cbb-9d6b-d2a52e435ee4" (UID: "335a2fc4-633e-4cbb-9d6b-d2a52e435ee4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:24:41.380811 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:41.380223 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/335a2fc4-633e-4cbb-9d6b-d2a52e435ee4-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:24:41.380811 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:41.380262 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhvlc\" (UniqueName: \"kubernetes.io/projected/335a2fc4-633e-4cbb-9d6b-d2a52e435ee4-kube-api-access-rhvlc\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:24:41.380811 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:41.380277 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/335a2fc4-633e-4cbb-9d6b-d2a52e435ee4-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:24:42.049947 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:42.049914 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4c6596-xdd2z" event={"ID":"335a2fc4-633e-4cbb-9d6b-d2a52e435ee4","Type":"ContainerDied","Data":"6accba5f0306275681e1da9054f47fb0fd6ae1f4775b6a9e07d33de535aa4c1c"} Oct 30 17:24:42.050054 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:42.049954 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4c6596-xdd2z" Oct 30 17:24:42.050054 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:42.049974 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6accba5f0306275681e1da9054f47fb0fd6ae1f4775b6a9e07d33de535aa4c1c" Oct 30 17:24:42.297354 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:42.297326 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x" Oct 30 17:24:42.383268 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:42.383230 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt" Oct 30 17:24:42.393491 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:42.393466 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46c8b953-69a7-4e2e-8695-a0b9d745529f-kube-api-access-flv5q\" (UniqueName: \"kubernetes.io/projected/46c8b953-69a7-4e2e-8695-a0b9d745529f-kube-api-access-flv5q\") pod \"46c8b953-69a7-4e2e-8695-a0b9d745529f\" (UID: \"46c8b953-69a7-4e2e-8695-a0b9d745529f\") " Oct 30 17:24:42.393647 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:42.393526 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/273838d6-034c-4763-a75f-5a357a4953f4-istio-token\" (UniqueName: \"kubernetes.io/projected/273838d6-034c-4763-a75f-5a357a4953f4-istio-token\") pod \"273838d6-034c-4763-a75f-5a357a4953f4\" (UID: \"273838d6-034c-4763-a75f-5a357a4953f4\") " Oct 30 17:24:42.393647 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:42.393562 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/273838d6-034c-4763-a75f-5a357a4953f4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/273838d6-034c-4763-a75f-5a357a4953f4-istio-data\") pod \"273838d6-034c-4763-a75f-5a357a4953f4\" (UID: \"273838d6-034c-4763-a75f-5a357a4953f4\") " Oct 30 17:24:42.393647 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:42.393589 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46c8b953-69a7-4e2e-8695-a0b9d745529f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46c8b953-69a7-4e2e-8695-a0b9d745529f-istio-data\") pod \"46c8b953-69a7-4e2e-8695-a0b9d745529f\" (UID: \"46c8b953-69a7-4e2e-8695-a0b9d745529f\") " Oct 30 17:24:42.393647 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:42.393624 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46c8b953-69a7-4e2e-8695-a0b9d745529f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46c8b953-69a7-4e2e-8695-a0b9d745529f-istiod-ca-cert\") pod \"46c8b953-69a7-4e2e-8695-a0b9d745529f\" (UID: \"46c8b953-69a7-4e2e-8695-a0b9d745529f\") " Oct 30 17:24:42.393959 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:42.393796 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/273838d6-034c-4763-a75f-5a357a4953f4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/273838d6-034c-4763-a75f-5a357a4953f4-credential-socket\") pod \"273838d6-034c-4763-a75f-5a357a4953f4\" (UID: \"273838d6-034c-4763-a75f-5a357a4953f4\") " Oct 30 17:24:42.393959 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:42.393840 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/273838d6-034c-4763-a75f-5a357a4953f4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/273838d6-034c-4763-a75f-5a357a4953f4-istiod-ca-cert\") pod \"273838d6-034c-4763-a75f-5a357a4953f4\" (UID: \"273838d6-034c-4763-a75f-5a357a4953f4\") " Oct 30 17:24:42.393959 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:42.393910 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46c8b953-69a7-4e2e-8695-a0b9d745529f-istio-token\" (UniqueName: \"kubernetes.io/projected/46c8b953-69a7-4e2e-8695-a0b9d745529f-istio-token\") pod \"46c8b953-69a7-4e2e-8695-a0b9d745529f\" (UID: \"46c8b953-69a7-4e2e-8695-a0b9d745529f\") " Oct 30 17:24:42.393959 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:42.393915 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46c8b953-69a7-4e2e-8695-a0b9d745529f-istio-data" pod "46c8b953-69a7-4e2e-8695-a0b9d745529f" (UID: "46c8b953-69a7-4e2e-8695-a0b9d745529f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:24:42.393959 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:42.393922 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/273838d6-034c-4763-a75f-5a357a4953f4-istio-data" pod "273838d6-034c-4763-a75f-5a357a4953f4" (UID: "273838d6-034c-4763-a75f-5a357a4953f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:24:42.394237 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:42.393963 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/273838d6-034c-4763-a75f-5a357a4953f4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/273838d6-034c-4763-a75f-5a357a4953f4-istio-podinfo\") pod \"273838d6-034c-4763-a75f-5a357a4953f4\" (UID: \"273838d6-034c-4763-a75f-5a357a4953f4\") " Oct 30 17:24:42.394237 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:42.394019 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46c8b953-69a7-4e2e-8695-a0b9d745529f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46c8b953-69a7-4e2e-8695-a0b9d745529f-credential-socket\") pod \"46c8b953-69a7-4e2e-8695-a0b9d745529f\" (UID: \"46c8b953-69a7-4e2e-8695-a0b9d745529f\") " Oct 30 17:24:42.394237 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:42.394062 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46c8b953-69a7-4e2e-8695-a0b9d745529f-istiod-ca-cert" pod "46c8b953-69a7-4e2e-8695-a0b9d745529f" (UID: "46c8b953-69a7-4e2e-8695-a0b9d745529f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:24:42.394237 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:42.394099 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/273838d6-034c-4763-a75f-5a357a4953f4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/273838d6-034c-4763-a75f-5a357a4953f4-workload-certs\") pod \"273838d6-034c-4763-a75f-5a357a4953f4\" (UID: \"273838d6-034c-4763-a75f-5a357a4953f4\") " Oct 30 17:24:42.394237 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:42.394134 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46c8b953-69a7-4e2e-8695-a0b9d745529f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46c8b953-69a7-4e2e-8695-a0b9d745529f-istio-envoy\") pod \"46c8b953-69a7-4e2e-8695-a0b9d745529f\" (UID: \"46c8b953-69a7-4e2e-8695-a0b9d745529f\") " Oct 30 17:24:42.394237 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:42.394188 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/273838d6-034c-4763-a75f-5a357a4953f4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/273838d6-034c-4763-a75f-5a357a4953f4-workload-socket\") pod \"273838d6-034c-4763-a75f-5a357a4953f4\" (UID: \"273838d6-034c-4763-a75f-5a357a4953f4\") " Oct 30 17:24:42.394586 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:42.394244 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/273838d6-034c-4763-a75f-5a357a4953f4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/273838d6-034c-4763-a75f-5a357a4953f4-istio-envoy\") pod \"273838d6-034c-4763-a75f-5a357a4953f4\" (UID: \"273838d6-034c-4763-a75f-5a357a4953f4\") " Oct 30 17:24:42.394586 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:42.394286 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/46c8b953-69a7-4e2e-8695-a0b9d745529f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46c8b953-69a7-4e2e-8695-a0b9d745529f-istio-podinfo\") pod \"46c8b953-69a7-4e2e-8695-a0b9d745529f\" (UID: \"46c8b953-69a7-4e2e-8695-a0b9d745529f\") " Oct 30 17:24:42.394586 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:42.394296 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46c8b953-69a7-4e2e-8695-a0b9d745529f-credential-socket" pod "46c8b953-69a7-4e2e-8695-a0b9d745529f" (UID: "46c8b953-69a7-4e2e-8695-a0b9d745529f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:24:42.394586 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:42.394318 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46c8b953-69a7-4e2e-8695-a0b9d745529f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46c8b953-69a7-4e2e-8695-a0b9d745529f-workload-certs\") pod \"46c8b953-69a7-4e2e-8695-a0b9d745529f\" (UID: \"46c8b953-69a7-4e2e-8695-a0b9d745529f\") " Oct 30 17:24:42.394586 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:42.394316 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/273838d6-034c-4763-a75f-5a357a4953f4-credential-socket" pod "273838d6-034c-4763-a75f-5a357a4953f4" (UID: "273838d6-034c-4763-a75f-5a357a4953f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:24:42.394586 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:42.394323 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/273838d6-034c-4763-a75f-5a357a4953f4-istiod-ca-cert" pod "273838d6-034c-4763-a75f-5a357a4953f4" (UID: "273838d6-034c-4763-a75f-5a357a4953f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:24:42.394586 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:42.394392 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/273838d6-034c-4763-a75f-5a357a4953f4-kube-api-access-t78wv\" (UniqueName: \"kubernetes.io/projected/273838d6-034c-4763-a75f-5a357a4953f4-kube-api-access-t78wv\") pod \"273838d6-034c-4763-a75f-5a357a4953f4\" (UID: \"273838d6-034c-4763-a75f-5a357a4953f4\") " Oct 30 17:24:42.394586 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:42.394452 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46c8b953-69a7-4e2e-8695-a0b9d745529f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46c8b953-69a7-4e2e-8695-a0b9d745529f-workload-socket\") pod \"46c8b953-69a7-4e2e-8695-a0b9d745529f\" (UID: \"46c8b953-69a7-4e2e-8695-a0b9d745529f\") " Oct 30 17:24:42.395330 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:42.395306 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46c8b953-69a7-4e2e-8695-a0b9d745529f-istiod-ca-cert\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:24:42.395427 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:42.395335 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/273838d6-034c-4763-a75f-5a357a4953f4-credential-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:24:42.395427 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:42.395350 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/273838d6-034c-4763-a75f-5a357a4953f4-istiod-ca-cert\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:24:42.395427 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:42.395363 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46c8b953-69a7-4e2e-8695-a0b9d745529f-credential-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:24:42.395427 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:42.395375 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/273838d6-034c-4763-a75f-5a357a4953f4-istio-data\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:24:42.395427 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:42.395388 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46c8b953-69a7-4e2e-8695-a0b9d745529f-istio-data\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:24:42.395706 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:42.395619 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46c8b953-69a7-4e2e-8695-a0b9d745529f-workload-socket" pod "46c8b953-69a7-4e2e-8695-a0b9d745529f" (UID: "46c8b953-69a7-4e2e-8695-a0b9d745529f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:24:42.395848 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:42.395826 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46c8b953-69a7-4e2e-8695-a0b9d745529f-workload-certs" pod "46c8b953-69a7-4e2e-8695-a0b9d745529f" (UID: "46c8b953-69a7-4e2e-8695-a0b9d745529f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:24:42.396758 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:42.396734 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/273838d6-034c-4763-a75f-5a357a4953f4-istio-token" pod "273838d6-034c-4763-a75f-5a357a4953f4" (UID: "273838d6-034c-4763-a75f-5a357a4953f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:24:42.396979 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:42.396956 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/273838d6-034c-4763-a75f-5a357a4953f4-workload-socket" pod "273838d6-034c-4763-a75f-5a357a4953f4" (UID: "273838d6-034c-4763-a75f-5a357a4953f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:24:42.397128 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:42.397106 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/273838d6-034c-4763-a75f-5a357a4953f4-workload-certs" pod "273838d6-034c-4763-a75f-5a357a4953f4" (UID: "273838d6-034c-4763-a75f-5a357a4953f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:24:42.397398 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:42.397373 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46c8b953-69a7-4e2e-8695-a0b9d745529f-istio-token" pod "46c8b953-69a7-4e2e-8695-a0b9d745529f" (UID: "46c8b953-69a7-4e2e-8695-a0b9d745529f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:24:42.397471 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:42.397254 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46c8b953-69a7-4e2e-8695-a0b9d745529f-istio-envoy" pod "46c8b953-69a7-4e2e-8695-a0b9d745529f" (UID: "46c8b953-69a7-4e2e-8695-a0b9d745529f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:24:42.398044 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:42.398020 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46c8b953-69a7-4e2e-8695-a0b9d745529f-istio-podinfo" pod "46c8b953-69a7-4e2e-8695-a0b9d745529f" (UID: "46c8b953-69a7-4e2e-8695-a0b9d745529f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:24:42.398266 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:42.398247 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/273838d6-034c-4763-a75f-5a357a4953f4-kube-api-access-t78wv" pod "273838d6-034c-4763-a75f-5a357a4953f4" (UID: "273838d6-034c-4763-a75f-5a357a4953f4"). InnerVolumeSpecName "kube-api-access-t78wv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:24:42.398471 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:42.398451 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/273838d6-034c-4763-a75f-5a357a4953f4-istio-envoy" pod "273838d6-034c-4763-a75f-5a357a4953f4" (UID: "273838d6-034c-4763-a75f-5a357a4953f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:24:42.398878 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:42.398857 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/273838d6-034c-4763-a75f-5a357a4953f4-istio-podinfo" pod "273838d6-034c-4763-a75f-5a357a4953f4" (UID: "273838d6-034c-4763-a75f-5a357a4953f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:24:42.399887 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:42.399865 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46c8b953-69a7-4e2e-8695-a0b9d745529f-kube-api-access-flv5q" pod "46c8b953-69a7-4e2e-8695-a0b9d745529f" (UID: "46c8b953-69a7-4e2e-8695-a0b9d745529f"). InnerVolumeSpecName "kube-api-access-flv5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:24:42.497056 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:42.497023 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46c8b953-69a7-4e2e-8695-a0b9d745529f-istio-token\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:24:42.497056 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:42.497051 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/273838d6-034c-4763-a75f-5a357a4953f4-istio-podinfo\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:24:42.497056 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:42.497060 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/273838d6-034c-4763-a75f-5a357a4953f4-workload-certs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:24:42.497288 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:42.497070 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46c8b953-69a7-4e2e-8695-a0b9d745529f-istio-envoy\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:24:42.497288 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:42.497078 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/273838d6-034c-4763-a75f-5a357a4953f4-workload-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:24:42.497288 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:42.497086 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/273838d6-034c-4763-a75f-5a357a4953f4-istio-envoy\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:24:42.497288 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:42.497096 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46c8b953-69a7-4e2e-8695-a0b9d745529f-istio-podinfo\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:24:42.497288 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:42.497103 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46c8b953-69a7-4e2e-8695-a0b9d745529f-workload-certs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:24:42.497288 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:42.497112 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t78wv\" (UniqueName: \"kubernetes.io/projected/273838d6-034c-4763-a75f-5a357a4953f4-kube-api-access-t78wv\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:24:42.497288 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:42.497120 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46c8b953-69a7-4e2e-8695-a0b9d745529f-workload-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:24:42.497288 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:42.497128 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flv5q\" (UniqueName: \"kubernetes.io/projected/46c8b953-69a7-4e2e-8695-a0b9d745529f-kube-api-access-flv5q\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:24:42.497288 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:42.497135 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/273838d6-034c-4763-a75f-5a357a4953f4-istio-token\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:24:43.064973 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:43.064931 4493 generic.go:356] "Generic (PLEG): container finished" podID="273838d6-034c-4763-a75f-5a357a4953f4" containerID="f9eba3fa07ffe13d21325f081b011b36a7f097ec03725b08099c1f83423407e4" exitCode=0 Oct 30 17:24:43.065246 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:43.065221 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x" event={"ID":"273838d6-034c-4763-a75f-5a357a4953f4","Type":"ContainerDied","Data":"f9eba3fa07ffe13d21325f081b011b36a7f097ec03725b08099c1f83423407e4"} Oct 30 17:24:43.065354 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:43.065323 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x" event={"ID":"273838d6-034c-4763-a75f-5a357a4953f4","Type":"ContainerDied","Data":"62d1d03cc3477a59810bd27cd321c9cf18d865d5ff11ec0c0bc0aa4a1bca1735"} Oct 30 17:24:43.065417 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:43.065362 4493 scope.go:122] "RemoveContainer" containerID="f9eba3fa07ffe13d21325f081b011b36a7f097ec03725b08099c1f83423407e4" Oct 30 17:24:43.065585 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:43.065567 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x" Oct 30 17:24:43.068867 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:43.068838 4493 generic.go:356] "Generic (PLEG): container finished" podID="46c8b953-69a7-4e2e-8695-a0b9d745529f" containerID="83c7dfe4cc1f60ab3d1349a28f17511fe13e8aa88b83ef1ef858ddb9d83a0559" exitCode=0 Oct 30 17:24:43.068996 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:43.068976 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt" event={"ID":"46c8b953-69a7-4e2e-8695-a0b9d745529f","Type":"ContainerDied","Data":"83c7dfe4cc1f60ab3d1349a28f17511fe13e8aa88b83ef1ef858ddb9d83a0559"} Oct 30 17:24:43.069081 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:43.069039 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt" event={"ID":"46c8b953-69a7-4e2e-8695-a0b9d745529f","Type":"ContainerDied","Data":"8494b0d3ea0b168f3afdba6fc34020090b497d291c1e9b6d5403c886ea6a4d44"} Oct 30 17:24:43.069243 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:43.069221 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt" Oct 30 17:24:43.085011 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:43.084982 4493 scope.go:122] "RemoveContainer" containerID="f9eba3fa07ffe13d21325f081b011b36a7f097ec03725b08099c1f83423407e4" Oct 30 17:24:43.085305 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:24:43.085270 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9eba3fa07ffe13d21325f081b011b36a7f097ec03725b08099c1f83423407e4\": container with ID starting with f9eba3fa07ffe13d21325f081b011b36a7f097ec03725b08099c1f83423407e4 not found: ID does not exist" containerID="f9eba3fa07ffe13d21325f081b011b36a7f097ec03725b08099c1f83423407e4" Oct 30 17:24:43.085410 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:43.085307 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9eba3fa07ffe13d21325f081b011b36a7f097ec03725b08099c1f83423407e4"} err="failed to get container status \"f9eba3fa07ffe13d21325f081b011b36a7f097ec03725b08099c1f83423407e4\": rpc error: code = NotFound desc = could not find container \"f9eba3fa07ffe13d21325f081b011b36a7f097ec03725b08099c1f83423407e4\": container with ID starting with f9eba3fa07ffe13d21325f081b011b36a7f097ec03725b08099c1f83423407e4 not found: ID does not exist" Oct 30 17:24:43.085410 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:43.085330 4493 scope.go:122] "RemoveContainer" containerID="83c7dfe4cc1f60ab3d1349a28f17511fe13e8aa88b83ef1ef858ddb9d83a0559" Oct 30 17:24:43.096262 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:43.096243 4493 scope.go:122] "RemoveContainer" containerID="83c7dfe4cc1f60ab3d1349a28f17511fe13e8aa88b83ef1ef858ddb9d83a0559" Oct 30 17:24:43.096545 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:24:43.096528 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83c7dfe4cc1f60ab3d1349a28f17511fe13e8aa88b83ef1ef858ddb9d83a0559\": container with ID starting with 83c7dfe4cc1f60ab3d1349a28f17511fe13e8aa88b83ef1ef858ddb9d83a0559 not found: ID does not exist" containerID="83c7dfe4cc1f60ab3d1349a28f17511fe13e8aa88b83ef1ef858ddb9d83a0559" Oct 30 17:24:43.096607 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:43.096552 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83c7dfe4cc1f60ab3d1349a28f17511fe13e8aa88b83ef1ef858ddb9d83a0559"} err="failed to get container status \"83c7dfe4cc1f60ab3d1349a28f17511fe13e8aa88b83ef1ef858ddb9d83a0559\": rpc error: code = NotFound desc = could not find container \"83c7dfe4cc1f60ab3d1349a28f17511fe13e8aa88b83ef1ef858ddb9d83a0559\": container with ID starting with 83c7dfe4cc1f60ab3d1349a28f17511fe13e8aa88b83ef1ef858ddb9d83a0559 not found: ID does not exist" Oct 30 17:24:43.118285 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:43.118253 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x"] Oct 30 17:24:43.130028 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:43.130001 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-f579x"] Oct 30 17:24:43.178809 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:43.178120 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt"] Oct 30 17:24:43.188736 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:43.183155 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-rqfrt"] Oct 30 17:24:44.983636 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:44.983592 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="273838d6-034c-4763-a75f-5a357a4953f4" path="/var/lib/kubelet/pods/273838d6-034c-4763-a75f-5a357a4953f4/volumes" Oct 30 17:24:44.984316 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:44.984297 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46c8b953-69a7-4e2e-8695-a0b9d745529f" path="/var/lib/kubelet/pods/46c8b953-69a7-4e2e-8695-a0b9d745529f/volumes" Oct 30 17:24:52.546383 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:52.546348 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-85eb87-bhqxk"] Oct 30 17:24:52.555945 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:52.555911 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-85eb87-bhqxk"] Oct 30 17:24:52.934498 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:52.934467 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="822b2b48-39df-417d-8985-647a0a0e1313" path="/var/lib/kubelet/pods/822b2b48-39df-417d-8985-647a0a0e1313/volumes" Oct 30 17:24:54.528209 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:54.528145 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" podUID="b5abe703-fee0-4185-aa22-88d1db06392b" containerName="main" containerID="cri-o://bc6fea780408c43d6c2c4a80863af8d452c26a9fcf578f9110d9d64c4fb7a422" gracePeriod=45 Oct 30 17:24:54.559243 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:54.559202 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" podUID="e43738f3-d55f-4c0a-a364-bf2ac76605c6" containerName="main" containerID="cri-o://c6a0c1fe11d893b0b5c92c409685d11fefe2250cba9742fb8acc3ffc9d7ff5ce" gracePeriod=45 Oct 30 17:24:54.850235 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:54.850204 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" Oct 30 17:24:54.878897 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:54.878865 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" Oct 30 17:24:54.947598 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:54.947562 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5abe703-fee0-4185-aa22-88d1db06392b-tls-certs\" (UniqueName: \"kubernetes.io/secret/b5abe703-fee0-4185-aa22-88d1db06392b-tls-certs\") pod \"b5abe703-fee0-4185-aa22-88d1db06392b\" (UID: \"b5abe703-fee0-4185-aa22-88d1db06392b\") " Oct 30 17:24:54.947766 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:54.947636 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5abe703-fee0-4185-aa22-88d1db06392b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5abe703-fee0-4185-aa22-88d1db06392b-model-cache\") pod \"b5abe703-fee0-4185-aa22-88d1db06392b\" (UID: \"b5abe703-fee0-4185-aa22-88d1db06392b\") " Oct 30 17:24:54.947766 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:54.947703 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e43738f3-d55f-4c0a-a364-bf2ac76605c6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e43738f3-d55f-4c0a-a364-bf2ac76605c6-model-cache\") pod \"e43738f3-d55f-4c0a-a364-bf2ac76605c6\" (UID: \"e43738f3-d55f-4c0a-a364-bf2ac76605c6\") " Oct 30 17:24:54.947766 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:54.947731 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e43738f3-d55f-4c0a-a364-bf2ac76605c6-tls-certs\" (UniqueName: \"kubernetes.io/secret/e43738f3-d55f-4c0a-a364-bf2ac76605c6-tls-certs\") pod \"e43738f3-d55f-4c0a-a364-bf2ac76605c6\" (UID: \"e43738f3-d55f-4c0a-a364-bf2ac76605c6\") " Oct 30 17:24:54.947994 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:54.947820 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e43738f3-d55f-4c0a-a364-bf2ac76605c6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e43738f3-d55f-4c0a-a364-bf2ac76605c6-tmp-dir\") pod \"e43738f3-d55f-4c0a-a364-bf2ac76605c6\" (UID: \"e43738f3-d55f-4c0a-a364-bf2ac76605c6\") " Oct 30 17:24:54.947994 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:54.947867 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5abe703-fee0-4185-aa22-88d1db06392b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5abe703-fee0-4185-aa22-88d1db06392b-tmp-dir\") pod \"b5abe703-fee0-4185-aa22-88d1db06392b\" (UID: \"b5abe703-fee0-4185-aa22-88d1db06392b\") " Oct 30 17:24:54.947994 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:54.947952 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5abe703-fee0-4185-aa22-88d1db06392b-kube-api-access-hqcbx\" (UniqueName: \"kubernetes.io/projected/b5abe703-fee0-4185-aa22-88d1db06392b-kube-api-access-hqcbx\") pod \"b5abe703-fee0-4185-aa22-88d1db06392b\" (UID: \"b5abe703-fee0-4185-aa22-88d1db06392b\") " Oct 30 17:24:54.948166 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:54.948013 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e43738f3-d55f-4c0a-a364-bf2ac76605c6-kube-api-access-fsrnh\" (UniqueName: \"kubernetes.io/projected/e43738f3-d55f-4c0a-a364-bf2ac76605c6-kube-api-access-fsrnh\") pod \"e43738f3-d55f-4c0a-a364-bf2ac76605c6\" (UID: \"e43738f3-d55f-4c0a-a364-bf2ac76605c6\") " Oct 30 17:24:54.948166 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:54.948048 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e43738f3-d55f-4c0a-a364-bf2ac76605c6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e43738f3-d55f-4c0a-a364-bf2ac76605c6-dshm\") pod \"e43738f3-d55f-4c0a-a364-bf2ac76605c6\" (UID: \"e43738f3-d55f-4c0a-a364-bf2ac76605c6\") " Oct 30 17:24:54.948282 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:54.948257 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5abe703-fee0-4185-aa22-88d1db06392b-home\" (UniqueName: \"kubernetes.io/empty-dir/b5abe703-fee0-4185-aa22-88d1db06392b-home\") pod \"b5abe703-fee0-4185-aa22-88d1db06392b\" (UID: \"b5abe703-fee0-4185-aa22-88d1db06392b\") " Oct 30 17:24:54.948346 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:54.948304 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5abe703-fee0-4185-aa22-88d1db06392b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5abe703-fee0-4185-aa22-88d1db06392b-dshm\") pod \"b5abe703-fee0-4185-aa22-88d1db06392b\" (UID: \"b5abe703-fee0-4185-aa22-88d1db06392b\") " Oct 30 17:24:54.949183 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:54.948358 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e43738f3-d55f-4c0a-a364-bf2ac76605c6-home\" (UniqueName: \"kubernetes.io/empty-dir/e43738f3-d55f-4c0a-a364-bf2ac76605c6-home\") pod \"e43738f3-d55f-4c0a-a364-bf2ac76605c6\" (UID: \"e43738f3-d55f-4c0a-a364-bf2ac76605c6\") " Oct 30 17:24:54.949183 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:54.948647 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5abe703-fee0-4185-aa22-88d1db06392b-model-cache" pod "b5abe703-fee0-4185-aa22-88d1db06392b" (UID: "b5abe703-fee0-4185-aa22-88d1db06392b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:24:54.949183 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:54.948940 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e43738f3-d55f-4c0a-a364-bf2ac76605c6-home" pod "e43738f3-d55f-4c0a-a364-bf2ac76605c6" (UID: "e43738f3-d55f-4c0a-a364-bf2ac76605c6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:24:54.949183 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:54.949058 4493 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5abe703-fee0-4185-aa22-88d1db06392b-model-cache\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:24:54.949183 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:54.949078 4493 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e43738f3-d55f-4c0a-a364-bf2ac76605c6-home\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:24:54.949500 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:54.949317 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e43738f3-d55f-4c0a-a364-bf2ac76605c6-tmp-dir" pod "e43738f3-d55f-4c0a-a364-bf2ac76605c6" (UID: "e43738f3-d55f-4c0a-a364-bf2ac76605c6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:24:54.949911 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:54.949605 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5abe703-fee0-4185-aa22-88d1db06392b-home" pod "b5abe703-fee0-4185-aa22-88d1db06392b" (UID: "b5abe703-fee0-4185-aa22-88d1db06392b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:24:54.949911 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:54.949862 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e43738f3-d55f-4c0a-a364-bf2ac76605c6-model-cache" pod "e43738f3-d55f-4c0a-a364-bf2ac76605c6" (UID: "e43738f3-d55f-4c0a-a364-bf2ac76605c6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:24:54.952067 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:54.952041 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5abe703-fee0-4185-aa22-88d1db06392b-tmp-dir" pod "b5abe703-fee0-4185-aa22-88d1db06392b" (UID: "b5abe703-fee0-4185-aa22-88d1db06392b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:24:54.952258 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:54.952229 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e43738f3-d55f-4c0a-a364-bf2ac76605c6-dshm" pod "e43738f3-d55f-4c0a-a364-bf2ac76605c6" (UID: "e43738f3-d55f-4c0a-a364-bf2ac76605c6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:24:54.952258 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:54.952246 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5abe703-fee0-4185-aa22-88d1db06392b-dshm" pod "b5abe703-fee0-4185-aa22-88d1db06392b" (UID: "b5abe703-fee0-4185-aa22-88d1db06392b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:24:54.954208 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:54.954186 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e43738f3-d55f-4c0a-a364-bf2ac76605c6-tls-certs" pod "e43738f3-d55f-4c0a-a364-bf2ac76605c6" (UID: "e43738f3-d55f-4c0a-a364-bf2ac76605c6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:24:54.954331 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:54.954221 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5abe703-fee0-4185-aa22-88d1db06392b-tls-certs" pod "b5abe703-fee0-4185-aa22-88d1db06392b" (UID: "b5abe703-fee0-4185-aa22-88d1db06392b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:24:54.955576 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:54.955550 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5abe703-fee0-4185-aa22-88d1db06392b-kube-api-access-hqcbx" pod "b5abe703-fee0-4185-aa22-88d1db06392b" (UID: "b5abe703-fee0-4185-aa22-88d1db06392b"). InnerVolumeSpecName "kube-api-access-hqcbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:24:54.956260 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:54.956240 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e43738f3-d55f-4c0a-a364-bf2ac76605c6-kube-api-access-fsrnh" pod "e43738f3-d55f-4c0a-a364-bf2ac76605c6" (UID: "e43738f3-d55f-4c0a-a364-bf2ac76605c6"). InnerVolumeSpecName "kube-api-access-fsrnh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:24:55.062809 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:55.057755 4493 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5abe703-fee0-4185-aa22-88d1db06392b-tls-certs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:24:55.062809 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:55.057816 4493 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e43738f3-d55f-4c0a-a364-bf2ac76605c6-model-cache\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:24:55.062809 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:55.057830 4493 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e43738f3-d55f-4c0a-a364-bf2ac76605c6-tls-certs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:24:55.062809 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:55.057844 4493 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e43738f3-d55f-4c0a-a364-bf2ac76605c6-tmp-dir\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:24:55.062809 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:55.057856 4493 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5abe703-fee0-4185-aa22-88d1db06392b-tmp-dir\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:24:55.062809 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:55.057869 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqcbx\" (UniqueName: \"kubernetes.io/projected/b5abe703-fee0-4185-aa22-88d1db06392b-kube-api-access-hqcbx\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:24:55.062809 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:55.057883 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsrnh\" (UniqueName: \"kubernetes.io/projected/e43738f3-d55f-4c0a-a364-bf2ac76605c6-kube-api-access-fsrnh\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:24:55.062809 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:55.057902 4493 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e43738f3-d55f-4c0a-a364-bf2ac76605c6-dshm\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:24:55.062809 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:55.057914 4493 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5abe703-fee0-4185-aa22-88d1db06392b-home\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:24:55.062809 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:55.057925 4493 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5abe703-fee0-4185-aa22-88d1db06392b-dshm\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:24:55.214991 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:55.214957 4493 generic.go:356] "Generic (PLEG): container finished" podID="e43738f3-d55f-4c0a-a364-bf2ac76605c6" containerID="c6a0c1fe11d893b0b5c92c409685d11fefe2250cba9742fb8acc3ffc9d7ff5ce" exitCode=0 Oct 30 17:24:55.215219 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:55.215062 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" Oct 30 17:24:55.215219 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:55.215069 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" event={"ID":"e43738f3-d55f-4c0a-a364-bf2ac76605c6","Type":"ContainerDied","Data":"c6a0c1fe11d893b0b5c92c409685d11fefe2250cba9742fb8acc3ffc9d7ff5ce"} Oct 30 17:24:55.215219 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:55.215107 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl" event={"ID":"e43738f3-d55f-4c0a-a364-bf2ac76605c6","Type":"ContainerDied","Data":"705ff074bda8b6906f62ccb4880e81dfec1d1279a4b87f8d28799444044eda52"} Oct 30 17:24:55.215219 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:55.215128 4493 scope.go:122] "RemoveContainer" containerID="c6a0c1fe11d893b0b5c92c409685d11fefe2250cba9742fb8acc3ffc9d7ff5ce" Oct 30 17:24:55.217655 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:55.217624 4493 generic.go:356] "Generic (PLEG): container finished" podID="b5abe703-fee0-4185-aa22-88d1db06392b" containerID="bc6fea780408c43d6c2c4a80863af8d452c26a9fcf578f9110d9d64c4fb7a422" exitCode=0 Oct 30 17:24:55.218070 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:55.218046 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" event={"ID":"b5abe703-fee0-4185-aa22-88d1db06392b","Type":"ContainerDied","Data":"bc6fea780408c43d6c2c4a80863af8d452c26a9fcf578f9110d9d64c4fb7a422"} Oct 30 17:24:55.218192 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:55.218088 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" event={"ID":"b5abe703-fee0-4185-aa22-88d1db06392b","Type":"ContainerDied","Data":"3b536a4979e61f5b3b04ab6e8894c76d14b5bc2e3b690004c5346b5fd09a40bb"} Oct 30 17:24:55.218192 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:55.218153 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw" Oct 30 17:24:55.233867 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:55.233844 4493 scope.go:122] "RemoveContainer" containerID="c6a0c1fe11d893b0b5c92c409685d11fefe2250cba9742fb8acc3ffc9d7ff5ce" Oct 30 17:24:55.234193 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:24:55.234173 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6a0c1fe11d893b0b5c92c409685d11fefe2250cba9742fb8acc3ffc9d7ff5ce\": container with ID starting with c6a0c1fe11d893b0b5c92c409685d11fefe2250cba9742fb8acc3ffc9d7ff5ce not found: ID does not exist" containerID="c6a0c1fe11d893b0b5c92c409685d11fefe2250cba9742fb8acc3ffc9d7ff5ce" Oct 30 17:24:55.234323 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:55.234201 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6a0c1fe11d893b0b5c92c409685d11fefe2250cba9742fb8acc3ffc9d7ff5ce"} err="failed to get container status \"c6a0c1fe11d893b0b5c92c409685d11fefe2250cba9742fb8acc3ffc9d7ff5ce\": rpc error: code = NotFound desc = could not find container \"c6a0c1fe11d893b0b5c92c409685d11fefe2250cba9742fb8acc3ffc9d7ff5ce\": container with ID starting with c6a0c1fe11d893b0b5c92c409685d11fefe2250cba9742fb8acc3ffc9d7ff5ce not found: ID does not exist" Oct 30 17:24:55.234323 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:55.234218 4493 scope.go:122] "RemoveContainer" containerID="bc6fea780408c43d6c2c4a80863af8d452c26a9fcf578f9110d9d64c4fb7a422" Oct 30 17:24:55.246510 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:55.246484 4493 scope.go:122] "RemoveContainer" containerID="bc6fea780408c43d6c2c4a80863af8d452c26a9fcf578f9110d9d64c4fb7a422" Oct 30 17:24:55.246861 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:24:55.246836 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc6fea780408c43d6c2c4a80863af8d452c26a9fcf578f9110d9d64c4fb7a422\": container with ID starting with bc6fea780408c43d6c2c4a80863af8d452c26a9fcf578f9110d9d64c4fb7a422 not found: ID does not exist" containerID="bc6fea780408c43d6c2c4a80863af8d452c26a9fcf578f9110d9d64c4fb7a422" Oct 30 17:24:55.246961 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:55.246871 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc6fea780408c43d6c2c4a80863af8d452c26a9fcf578f9110d9d64c4fb7a422"} err="failed to get container status \"bc6fea780408c43d6c2c4a80863af8d452c26a9fcf578f9110d9d64c4fb7a422\": rpc error: code = NotFound desc = could not find container \"bc6fea780408c43d6c2c4a80863af8d452c26a9fcf578f9110d9d64c4fb7a422\": container with ID starting with bc6fea780408c43d6c2c4a80863af8d452c26a9fcf578f9110d9d64c4fb7a422 not found: ID does not exist" Oct 30 17:24:55.267514 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:55.267487 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl"] Oct 30 17:24:55.271200 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:55.271177 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-2d365c/simulator-w3-2d365c-kserve-9f985c58c-zzjfl"] Oct 30 17:24:55.319410 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:55.319381 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw"] Oct 30 17:24:55.325212 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:55.325183 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-2d365c/distinct-2-w3-2d365c-kserve-6d899f8666-f9sqw"] Oct 30 17:24:56.934382 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:56.934348 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5abe703-fee0-4185-aa22-88d1db06392b" path="/var/lib/kubelet/pods/b5abe703-fee0-4185-aa22-88d1db06392b/volumes" Oct 30 17:24:56.935115 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:56.935090 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e43738f3-d55f-4c0a-a364-bf2ac76605c6" path="/var/lib/kubelet/pods/e43738f3-d55f-4c0a-a364-bf2ac76605c6/volumes" Oct 30 17:24:57.861826 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:57.861778 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95f0efc4-qnx55"] Oct 30 17:24:57.862357 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:57.862343 4493 cpu_manager.go:414] "containerMap: removing container" podUID="46c8b953-69a7-4e2e-8695-a0b9d745529f" containerName="istio-proxy" Oct 30 17:24:57.862421 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:57.862359 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46c8b953-69a7-4e2e-8695-a0b9d745529f" containerName="istio-proxy" Oct 30 17:24:57.862421 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:57.862373 4493 cpu_manager.go:414] "containerMap: removing container" podUID="335a2fc4-633e-4cbb-9d6b-d2a52e435ee4" containerName="revoke-keys" Oct 30 17:24:57.862421 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:57.862379 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="335a2fc4-633e-4cbb-9d6b-d2a52e435ee4" containerName="revoke-keys" Oct 30 17:24:57.862421 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:57.862388 4493 cpu_manager.go:414] "containerMap: removing container" podUID="273838d6-034c-4763-a75f-5a357a4953f4" containerName="istio-proxy" Oct 30 17:24:57.862421 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:57.862393 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="273838d6-034c-4763-a75f-5a357a4953f4" containerName="istio-proxy" Oct 30 17:24:57.862421 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:57.862421 4493 cpu_manager.go:414] "containerMap: removing container" podUID="e43738f3-d55f-4c0a-a364-bf2ac76605c6" containerName="main" Oct 30 17:24:57.862627 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:57.862428 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e43738f3-d55f-4c0a-a364-bf2ac76605c6" containerName="main" Oct 30 17:24:57.862627 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:57.862436 4493 cpu_manager.go:414] "containerMap: removing container" podUID="fb00f070-a308-43a8-866a-438505dc7600" containerName="revoke-keys" Oct 30 17:24:57.862627 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:57.862440 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb00f070-a308-43a8-866a-438505dc7600" containerName="revoke-keys" Oct 30 17:24:57.862627 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:57.862450 4493 cpu_manager.go:414] "containerMap: removing container" podUID="b5abe703-fee0-4185-aa22-88d1db06392b" containerName="main" Oct 30 17:24:57.862627 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:57.862454 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5abe703-fee0-4185-aa22-88d1db06392b" containerName="main" Oct 30 17:24:57.862627 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:57.862464 4493 cpu_manager.go:414] "containerMap: removing container" podUID="47bf14f7-1ba3-4416-a1ff-2bf301c2e98d" containerName="revoke-keys" Oct 30 17:24:57.862627 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:57.862468 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47bf14f7-1ba3-4416-a1ff-2bf301c2e98d" containerName="revoke-keys" Oct 30 17:24:57.862627 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:57.862544 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="e43738f3-d55f-4c0a-a364-bf2ac76605c6" containerName="main" Oct 30 17:24:57.862627 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:57.862551 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb00f070-a308-43a8-866a-438505dc7600" containerName="revoke-keys" Oct 30 17:24:57.862627 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:57.862558 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5abe703-fee0-4185-aa22-88d1db06392b" containerName="main" Oct 30 17:24:57.862627 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:57.862564 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="47bf14f7-1ba3-4416-a1ff-2bf301c2e98d" containerName="revoke-keys" Oct 30 17:24:57.862627 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:57.862572 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="46c8b953-69a7-4e2e-8695-a0b9d745529f" containerName="istio-proxy" Oct 30 17:24:57.862627 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:57.862578 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="335a2fc4-633e-4cbb-9d6b-d2a52e435ee4" containerName="revoke-keys" Oct 30 17:24:57.862627 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:57.862584 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="273838d6-034c-4763-a75f-5a357a4953f4" containerName="istio-proxy" Oct 30 17:24:57.866290 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:57.866272 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95f0efc4-qnx55" Oct 30 17:24:57.874250 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:57.874227 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1beb74a-dcl87"] Oct 30 17:24:57.878072 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:57.878055 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1beb74a-dcl87" Oct 30 17:24:57.908841 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:57.895488 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:24:57.908841 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:57.895560 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:24:57.908841 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:57.895621 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 17:24:57.908841 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:57.904844 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95f0efc4-qnx55"] Oct 30 17:24:57.908841 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:57.905917 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:24:57.908841 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:57.908705 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1beb74a-dcl87"] Oct 30 17:24:57.993931 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:57.993904 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99364c73-370a-443d-a082-454bc4fda069-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1beb74a-dcl87\" (UID: \"99364c73-370a-443d-a082-454bc4fda069\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1beb74a-dcl87" Oct 30 17:24:57.994331 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:57.994027 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee384285-d605-493f-bbb1-eec6e15d1549-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95f0efc4-qnx55\" (UID: \"ee384285-d605-493f-bbb1-eec6e15d1549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95f0efc4-qnx55" Oct 30 17:24:57.994331 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:57.994070 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgqm2\" (UniqueName: \"kubernetes.io/projected/99364c73-370a-443d-a082-454bc4fda069-kube-api-access-hgqm2\") pod \"maas-api-revoke-sub-a1beb74a-dcl87\" (UID: \"99364c73-370a-443d-a082-454bc4fda069\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1beb74a-dcl87" Oct 30 17:24:57.994331 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:57.994126 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wm2b\" (UniqueName: \"kubernetes.io/projected/ee384285-d605-493f-bbb1-eec6e15d1549-kube-api-access-7wm2b\") pod \"maas-api-revoke-sub-95f0efc4-qnx55\" (UID: \"ee384285-d605-493f-bbb1-eec6e15d1549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95f0efc4-qnx55" Oct 30 17:24:57.994331 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:57.994208 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee384285-d605-493f-bbb1-eec6e15d1549-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95f0efc4-qnx55\" (UID: \"ee384285-d605-493f-bbb1-eec6e15d1549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95f0efc4-qnx55" Oct 30 17:24:57.994331 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:57.994232 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99364c73-370a-443d-a082-454bc4fda069-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1beb74a-dcl87\" (UID: \"99364c73-370a-443d-a082-454bc4fda069\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1beb74a-dcl87" Oct 30 17:24:58.095660 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:58.095630 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee384285-d605-493f-bbb1-eec6e15d1549-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95f0efc4-qnx55\" (UID: \"ee384285-d605-493f-bbb1-eec6e15d1549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95f0efc4-qnx55" Oct 30 17:24:58.095891 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:58.095670 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99364c73-370a-443d-a082-454bc4fda069-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1beb74a-dcl87\" (UID: \"99364c73-370a-443d-a082-454bc4fda069\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1beb74a-dcl87" Oct 30 17:24:58.095891 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:58.095751 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99364c73-370a-443d-a082-454bc4fda069-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1beb74a-dcl87\" (UID: \"99364c73-370a-443d-a082-454bc4fda069\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1beb74a-dcl87" Oct 30 17:24:58.095891 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:58.095868 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee384285-d605-493f-bbb1-eec6e15d1549-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95f0efc4-qnx55\" (UID: \"ee384285-d605-493f-bbb1-eec6e15d1549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95f0efc4-qnx55" Oct 30 17:24:58.096045 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:58.095904 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgqm2\" (UniqueName: \"kubernetes.io/projected/99364c73-370a-443d-a082-454bc4fda069-kube-api-access-hgqm2\") pod \"maas-api-revoke-sub-a1beb74a-dcl87\" (UID: \"99364c73-370a-443d-a082-454bc4fda069\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1beb74a-dcl87" Oct 30 17:24:58.096045 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:58.095941 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wm2b\" (UniqueName: \"kubernetes.io/projected/ee384285-d605-493f-bbb1-eec6e15d1549-kube-api-access-7wm2b\") pod \"maas-api-revoke-sub-95f0efc4-qnx55\" (UID: \"ee384285-d605-493f-bbb1-eec6e15d1549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95f0efc4-qnx55" Oct 30 17:24:58.096467 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:58.096443 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99364c73-370a-443d-a082-454bc4fda069-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1beb74a-dcl87\" (UID: \"99364c73-370a-443d-a082-454bc4fda069\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1beb74a-dcl87" Oct 30 17:24:58.096711 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:58.096445 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee384285-d605-493f-bbb1-eec6e15d1549-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95f0efc4-qnx55\" (UID: \"ee384285-d605-493f-bbb1-eec6e15d1549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95f0efc4-qnx55" Oct 30 17:24:58.098477 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:58.098452 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee384285-d605-493f-bbb1-eec6e15d1549-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95f0efc4-qnx55\" (UID: \"ee384285-d605-493f-bbb1-eec6e15d1549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95f0efc4-qnx55" Oct 30 17:24:58.098558 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:58.098504 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99364c73-370a-443d-a082-454bc4fda069-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1beb74a-dcl87\" (UID: \"99364c73-370a-443d-a082-454bc4fda069\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1beb74a-dcl87" Oct 30 17:24:58.110600 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:58.110572 4493 scope.go:122] "RemoveContainer" containerID="0f30add92d2aca98bb755b6276bba5f85cafe5d3e31cfd4c0c5af97ae0ce6fec" Oct 30 17:24:58.119775 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:58.119714 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgqm2\" (UniqueName: \"kubernetes.io/projected/99364c73-370a-443d-a082-454bc4fda069-kube-api-access-hgqm2\") pod \"maas-api-revoke-sub-a1beb74a-dcl87\" (UID: \"99364c73-370a-443d-a082-454bc4fda069\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1beb74a-dcl87" Oct 30 17:24:58.119775 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:58.119737 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wm2b\" (UniqueName: \"kubernetes.io/projected/ee384285-d605-493f-bbb1-eec6e15d1549-kube-api-access-7wm2b\") pod \"maas-api-revoke-sub-95f0efc4-qnx55\" (UID: \"ee384285-d605-493f-bbb1-eec6e15d1549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95f0efc4-qnx55" Oct 30 17:24:58.180761 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:58.180740 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95f0efc4-qnx55" Oct 30 17:24:58.191659 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:58.191640 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1beb74a-dcl87" Oct 30 17:24:58.545734 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:58.545697 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95f0efc4-qnx55"] Oct 30 17:24:58.546237 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:24:58.546206 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee384285_d605_493f_bbb1_eec6e15d1549.slice/crio-dcdc92e184fdef939f5877771658f735a01eb381ccf031fecc29ac2a43ad4ddc WatchSource:0}: Error finding container dcdc92e184fdef939f5877771658f735a01eb381ccf031fecc29ac2a43ad4ddc: Status 404 returned error can't find the container with id dcdc92e184fdef939f5877771658f735a01eb381ccf031fecc29ac2a43ad4ddc Oct 30 17:24:58.584669 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:58.584643 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1beb74a-dcl87"] Oct 30 17:24:58.584915 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:24:58.584893 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99364c73_370a_443d_a082_454bc4fda069.slice/crio-f650617240f0033627c49b8c82ab491b1ff9b71b444a087f04c685059424cd9a WatchSource:0}: Error finding container f650617240f0033627c49b8c82ab491b1ff9b71b444a087f04c685059424cd9a: Status 404 returned error can't find the container with id f650617240f0033627c49b8c82ab491b1ff9b71b444a087f04c685059424cd9a Oct 30 17:24:59.257529 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:59.257489 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1beb74a-dcl87" event={"ID":"99364c73-370a-443d-a082-454bc4fda069","Type":"ContainerStarted","Data":"a4fc17b690a48e466dae3e69dc0c35641a067f3ef08df8e39207a09c9abcb9dd"} Oct 30 17:24:59.258016 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:59.257556 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1beb74a-dcl87" event={"ID":"99364c73-370a-443d-a082-454bc4fda069","Type":"ContainerStarted","Data":"f650617240f0033627c49b8c82ab491b1ff9b71b444a087f04c685059424cd9a"} Oct 30 17:24:59.259422 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:59.259394 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95f0efc4-qnx55" event={"ID":"ee384285-d605-493f-bbb1-eec6e15d1549","Type":"ContainerStarted","Data":"054aa3159ea0ffc88b9e2059505aef53c7448b7325b4dc6e749970b15dade833"} Oct 30 17:24:59.259586 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:59.259450 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95f0efc4-qnx55" event={"ID":"ee384285-d605-493f-bbb1-eec6e15d1549","Type":"ContainerStarted","Data":"dcdc92e184fdef939f5877771658f735a01eb381ccf031fecc29ac2a43ad4ddc"} Oct 30 17:24:59.297360 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:59.297315 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1beb74a-dcl87" podStartSLOduration=2.297301931 podStartE2EDuration="2.297301931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:24:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:24:59.296527223 +0000 UTC m=+3726.523702322" watchObservedRunningTime="2026-10-30 17:24:59.297301931 +0000 UTC m=+3726.524477032" Oct 30 17:24:59.340398 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:24:59.340354 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95f0efc4-qnx55" podStartSLOduration=2.340340651 podStartE2EDuration="2.340340651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:24:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:24:59.337223096 +0000 UTC m=+3726.564398206" watchObservedRunningTime="2026-10-30 17:24:59.340340651 +0000 UTC m=+3726.567515755" Oct 30 17:25:00.270294 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:00.270262 4493 generic.go:356] "Generic (PLEG): container finished" podID="ee384285-d605-493f-bbb1-eec6e15d1549" containerID="054aa3159ea0ffc88b9e2059505aef53c7448b7325b4dc6e749970b15dade833" exitCode=0 Oct 30 17:25:00.270798 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:00.270337 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95f0efc4-qnx55" event={"ID":"ee384285-d605-493f-bbb1-eec6e15d1549","Type":"ContainerDied","Data":"054aa3159ea0ffc88b9e2059505aef53c7448b7325b4dc6e749970b15dade833"} Oct 30 17:25:00.272334 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:00.272310 4493 generic.go:356] "Generic (PLEG): container finished" podID="99364c73-370a-443d-a082-454bc4fda069" containerID="a4fc17b690a48e466dae3e69dc0c35641a067f3ef08df8e39207a09c9abcb9dd" exitCode=0 Oct 30 17:25:00.272452 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:00.272344 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1beb74a-dcl87" event={"ID":"99364c73-370a-443d-a082-454bc4fda069","Type":"ContainerDied","Data":"a4fc17b690a48e466dae3e69dc0c35641a067f3ef08df8e39207a09c9abcb9dd"} Oct 30 17:25:00.511915 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:25:00.511196 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 17:25:00.808896 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:00.808858 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2589b5-8ntsr"] Oct 30 17:25:00.813124 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:00.813106 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2589b5-8ntsr" Oct 30 17:25:00.835090 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:00.834950 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2589b5-8ntsr"] Oct 30 17:25:00.871857 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:00.871822 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mp5s\" (UniqueName: \"kubernetes.io/projected/8200bff9-b1ac-41e9-812d-75ff15466ca4-kube-api-access-6mp5s\") pod \"maas-api-revoke-keys-e2e-extmod-a-2589b5-8ntsr\" (UID: \"8200bff9-b1ac-41e9-812d-75ff15466ca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2589b5-8ntsr" Oct 30 17:25:00.872039 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:00.871905 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8200bff9-b1ac-41e9-812d-75ff15466ca4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-2589b5-8ntsr\" (UID: \"8200bff9-b1ac-41e9-812d-75ff15466ca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2589b5-8ntsr" Oct 30 17:25:00.872240 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:00.872212 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8200bff9-b1ac-41e9-812d-75ff15466ca4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-2589b5-8ntsr\" (UID: \"8200bff9-b1ac-41e9-812d-75ff15466ca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2589b5-8ntsr" Oct 30 17:25:00.973349 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:00.973302 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8200bff9-b1ac-41e9-812d-75ff15466ca4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-2589b5-8ntsr\" (UID: \"8200bff9-b1ac-41e9-812d-75ff15466ca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2589b5-8ntsr" Oct 30 17:25:00.973349 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:00.973359 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mp5s\" (UniqueName: \"kubernetes.io/projected/8200bff9-b1ac-41e9-812d-75ff15466ca4-kube-api-access-6mp5s\") pod \"maas-api-revoke-keys-e2e-extmod-a-2589b5-8ntsr\" (UID: \"8200bff9-b1ac-41e9-812d-75ff15466ca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2589b5-8ntsr" Oct 30 17:25:00.973658 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:00.973486 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8200bff9-b1ac-41e9-812d-75ff15466ca4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-2589b5-8ntsr\" (UID: \"8200bff9-b1ac-41e9-812d-75ff15466ca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2589b5-8ntsr" Oct 30 17:25:00.974343 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:00.974318 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8200bff9-b1ac-41e9-812d-75ff15466ca4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-2589b5-8ntsr\" (UID: \"8200bff9-b1ac-41e9-812d-75ff15466ca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2589b5-8ntsr" Oct 30 17:25:00.976365 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:00.976340 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8200bff9-b1ac-41e9-812d-75ff15466ca4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-2589b5-8ntsr\" (UID: \"8200bff9-b1ac-41e9-812d-75ff15466ca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2589b5-8ntsr" Oct 30 17:25:00.997134 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:00.997111 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mp5s\" (UniqueName: \"kubernetes.io/projected/8200bff9-b1ac-41e9-812d-75ff15466ca4-kube-api-access-6mp5s\") pod \"maas-api-revoke-keys-e2e-extmod-a-2589b5-8ntsr\" (UID: \"8200bff9-b1ac-41e9-812d-75ff15466ca4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2589b5-8ntsr" Oct 30 17:25:01.127186 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:01.127098 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2589b5-8ntsr" Oct 30 17:25:01.443794 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:01.443743 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2589b5-8ntsr"] Oct 30 17:25:01.454815 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:25:01.449348 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8200bff9_b1ac_41e9_812d_75ff15466ca4.slice/crio-e1ee62fd3a6043c234372629aebc43d626d253172c788b834200bde843d8426c WatchSource:0}: Error finding container e1ee62fd3a6043c234372629aebc43d626d253172c788b834200bde843d8426c: Status 404 returned error can't find the container with id e1ee62fd3a6043c234372629aebc43d626d253172c788b834200bde843d8426c Oct 30 17:25:01.473139 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:01.473116 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95f0efc4-qnx55" Oct 30 17:25:01.495262 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:01.495239 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1beb74a-dcl87" Oct 30 17:25:01.582203 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:01.582171 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee384285-d605-493f-bbb1-eec6e15d1549-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee384285-d605-493f-bbb1-eec6e15d1549-maas-api-xks-ca\") pod \"ee384285-d605-493f-bbb1-eec6e15d1549\" (UID: \"ee384285-d605-493f-bbb1-eec6e15d1549\") " Oct 30 17:25:01.582363 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:01.582324 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99364c73-370a-443d-a082-454bc4fda069-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99364c73-370a-443d-a082-454bc4fda069-maas-api-openshift-ca\") pod \"99364c73-370a-443d-a082-454bc4fda069\" (UID: \"99364c73-370a-443d-a082-454bc4fda069\") " Oct 30 17:25:01.582446 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:01.582362 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99364c73-370a-443d-a082-454bc4fda069-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99364c73-370a-443d-a082-454bc4fda069-maas-api-xks-ca\") pod \"99364c73-370a-443d-a082-454bc4fda069\" (UID: \"99364c73-370a-443d-a082-454bc4fda069\") " Oct 30 17:25:01.582545 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:01.582525 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99364c73-370a-443d-a082-454bc4fda069-kube-api-access-hgqm2\" (UniqueName: \"kubernetes.io/projected/99364c73-370a-443d-a082-454bc4fda069-kube-api-access-hgqm2\") pod \"99364c73-370a-443d-a082-454bc4fda069\" (UID: \"99364c73-370a-443d-a082-454bc4fda069\") " Oct 30 17:25:01.582629 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:01.582572 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee384285-d605-493f-bbb1-eec6e15d1549-kube-api-access-7wm2b\" (UniqueName: \"kubernetes.io/projected/ee384285-d605-493f-bbb1-eec6e15d1549-kube-api-access-7wm2b\") pod \"ee384285-d605-493f-bbb1-eec6e15d1549\" (UID: \"ee384285-d605-493f-bbb1-eec6e15d1549\") " Oct 30 17:25:01.582629 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:01.582610 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee384285-d605-493f-bbb1-eec6e15d1549-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee384285-d605-493f-bbb1-eec6e15d1549-maas-api-openshift-ca\") pod \"ee384285-d605-493f-bbb1-eec6e15d1549\" (UID: \"ee384285-d605-493f-bbb1-eec6e15d1549\") " Oct 30 17:25:01.582770 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:01.582744 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99364c73-370a-443d-a082-454bc4fda069-maas-api-openshift-ca" pod "99364c73-370a-443d-a082-454bc4fda069" (UID: "99364c73-370a-443d-a082-454bc4fda069"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:25:01.583136 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:01.583110 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99364c73-370a-443d-a082-454bc4fda069-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:25:01.583328 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:01.583300 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee384285-d605-493f-bbb1-eec6e15d1549-maas-api-openshift-ca" pod "ee384285-d605-493f-bbb1-eec6e15d1549" (UID: "ee384285-d605-493f-bbb1-eec6e15d1549"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:25:01.585152 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:01.585130 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee384285-d605-493f-bbb1-eec6e15d1549-maas-api-xks-ca" pod "ee384285-d605-493f-bbb1-eec6e15d1549" (UID: "ee384285-d605-493f-bbb1-eec6e15d1549"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:25:01.585347 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:01.585324 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99364c73-370a-443d-a082-454bc4fda069-maas-api-xks-ca" pod "99364c73-370a-443d-a082-454bc4fda069" (UID: "99364c73-370a-443d-a082-454bc4fda069"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:25:01.585535 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:01.585513 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99364c73-370a-443d-a082-454bc4fda069-kube-api-access-hgqm2" pod "99364c73-370a-443d-a082-454bc4fda069" (UID: "99364c73-370a-443d-a082-454bc4fda069"). InnerVolumeSpecName "kube-api-access-hgqm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:25:01.585621 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:01.585574 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee384285-d605-493f-bbb1-eec6e15d1549-kube-api-access-7wm2b" pod "ee384285-d605-493f-bbb1-eec6e15d1549" (UID: "ee384285-d605-493f-bbb1-eec6e15d1549"). InnerVolumeSpecName "kube-api-access-7wm2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:25:01.684288 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:01.684198 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99364c73-370a-443d-a082-454bc4fda069-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:25:01.684288 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:01.684240 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgqm2\" (UniqueName: \"kubernetes.io/projected/99364c73-370a-443d-a082-454bc4fda069-kube-api-access-hgqm2\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:25:01.684288 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:01.684258 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wm2b\" (UniqueName: \"kubernetes.io/projected/ee384285-d605-493f-bbb1-eec6e15d1549-kube-api-access-7wm2b\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:25:01.684288 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:01.684275 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee384285-d605-493f-bbb1-eec6e15d1549-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:25:01.684288 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:01.684289 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee384285-d605-493f-bbb1-eec6e15d1549-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:25:02.295422 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:02.295327 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1beb74a-dcl87" Oct 30 17:25:02.295422 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:02.295398 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1beb74a-dcl87" event={"ID":"99364c73-370a-443d-a082-454bc4fda069","Type":"ContainerDied","Data":"f650617240f0033627c49b8c82ab491b1ff9b71b444a087f04c685059424cd9a"} Oct 30 17:25:02.295666 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:02.295461 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f650617240f0033627c49b8c82ab491b1ff9b71b444a087f04c685059424cd9a" Oct 30 17:25:02.297110 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:02.297085 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95f0efc4-qnx55" event={"ID":"ee384285-d605-493f-bbb1-eec6e15d1549","Type":"ContainerDied","Data":"dcdc92e184fdef939f5877771658f735a01eb381ccf031fecc29ac2a43ad4ddc"} Oct 30 17:25:02.297238 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:02.297128 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dcdc92e184fdef939f5877771658f735a01eb381ccf031fecc29ac2a43ad4ddc" Oct 30 17:25:02.297238 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:02.297090 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95f0efc4-qnx55" Oct 30 17:25:02.299128 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:02.299101 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2589b5-8ntsr" event={"ID":"8200bff9-b1ac-41e9-812d-75ff15466ca4","Type":"ContainerStarted","Data":"6708ce7d2e14273938fe5e184a48d9420f6d03aa8de74e095c59860c6b68713a"} Oct 30 17:25:02.299259 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:02.299148 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2589b5-8ntsr" event={"ID":"8200bff9-b1ac-41e9-812d-75ff15466ca4","Type":"ContainerStarted","Data":"e1ee62fd3a6043c234372629aebc43d626d253172c788b834200bde843d8426c"} Oct 30 17:25:02.349764 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:02.349701 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2589b5-8ntsr" podStartSLOduration=2.349681939 podStartE2EDuration="2.349681939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:25:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:25:02.348693556 +0000 UTC m=+3729.575868659" watchObservedRunningTime="2026-10-30 17:25:02.349681939 +0000 UTC m=+3729.576857044" Oct 30 17:25:03.308565 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:03.308530 4493 generic.go:356] "Generic (PLEG): container finished" podID="8200bff9-b1ac-41e9-812d-75ff15466ca4" containerID="6708ce7d2e14273938fe5e184a48d9420f6d03aa8de74e095c59860c6b68713a" exitCode=0 Oct 30 17:25:03.309012 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:03.308595 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2589b5-8ntsr" event={"ID":"8200bff9-b1ac-41e9-812d-75ff15466ca4","Type":"ContainerDied","Data":"6708ce7d2e14273938fe5e184a48d9420f6d03aa8de74e095c59860c6b68713a"} Oct 30 17:25:04.490734 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:04.490708 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2589b5-8ntsr" Oct 30 17:25:04.625926 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:04.625821 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8200bff9-b1ac-41e9-812d-75ff15466ca4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8200bff9-b1ac-41e9-812d-75ff15466ca4-maas-api-xks-ca\") pod \"8200bff9-b1ac-41e9-812d-75ff15466ca4\" (UID: \"8200bff9-b1ac-41e9-812d-75ff15466ca4\") " Oct 30 17:25:04.626108 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:04.626007 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8200bff9-b1ac-41e9-812d-75ff15466ca4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8200bff9-b1ac-41e9-812d-75ff15466ca4-maas-api-openshift-ca\") pod \"8200bff9-b1ac-41e9-812d-75ff15466ca4\" (UID: \"8200bff9-b1ac-41e9-812d-75ff15466ca4\") " Oct 30 17:25:04.626108 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:04.626073 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8200bff9-b1ac-41e9-812d-75ff15466ca4-kube-api-access-6mp5s\" (UniqueName: \"kubernetes.io/projected/8200bff9-b1ac-41e9-812d-75ff15466ca4-kube-api-access-6mp5s\") pod \"8200bff9-b1ac-41e9-812d-75ff15466ca4\" (UID: \"8200bff9-b1ac-41e9-812d-75ff15466ca4\") " Oct 30 17:25:04.626464 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:04.626431 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8200bff9-b1ac-41e9-812d-75ff15466ca4-maas-api-openshift-ca" pod "8200bff9-b1ac-41e9-812d-75ff15466ca4" (UID: "8200bff9-b1ac-41e9-812d-75ff15466ca4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:25:04.626753 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:04.626703 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8200bff9-b1ac-41e9-812d-75ff15466ca4-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:25:04.628269 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:04.628237 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8200bff9-b1ac-41e9-812d-75ff15466ca4-maas-api-xks-ca" pod "8200bff9-b1ac-41e9-812d-75ff15466ca4" (UID: "8200bff9-b1ac-41e9-812d-75ff15466ca4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:25:04.628511 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:04.628491 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8200bff9-b1ac-41e9-812d-75ff15466ca4-kube-api-access-6mp5s" pod "8200bff9-b1ac-41e9-812d-75ff15466ca4" (UID: "8200bff9-b1ac-41e9-812d-75ff15466ca4"). InnerVolumeSpecName "kube-api-access-6mp5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:25:04.728336 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:04.728289 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mp5s\" (UniqueName: \"kubernetes.io/projected/8200bff9-b1ac-41e9-812d-75ff15466ca4-kube-api-access-6mp5s\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:25:04.728336 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:04.728332 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8200bff9-b1ac-41e9-812d-75ff15466ca4-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:25:05.328559 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:05.328476 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2589b5-8ntsr" Oct 30 17:25:05.328559 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:05.328497 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2589b5-8ntsr" event={"ID":"8200bff9-b1ac-41e9-812d-75ff15466ca4","Type":"ContainerDied","Data":"e1ee62fd3a6043c234372629aebc43d626d253172c788b834200bde843d8426c"} Oct 30 17:25:05.328559 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:05.328545 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1ee62fd3a6043c234372629aebc43d626d253172c788b834200bde843d8426c" Oct 30 17:25:16.872972 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:16.872907 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bbc36f-wqwqs"] Oct 30 17:25:16.881529 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:16.881492 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bbc36f-wqwqs"] Oct 30 17:25:16.947886 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:16.947314 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb00f070-a308-43a8-866a-438505dc7600" path="/var/lib/kubelet/pods/fb00f070-a308-43a8-866a-438505dc7600/volumes" Oct 30 17:25:17.318807 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:17.318385 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2589b5-8ntsr"] Oct 30 17:25:17.329273 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:17.329236 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-2589b5-8ntsr"] Oct 30 17:25:17.780857 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:17.780815 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4c6596-xdd2z"] Oct 30 17:25:17.791820 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:17.791769 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4c6596-xdd2z"] Oct 30 17:25:18.935307 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:18.935276 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="335a2fc4-633e-4cbb-9d6b-d2a52e435ee4" path="/var/lib/kubelet/pods/335a2fc4-633e-4cbb-9d6b-d2a52e435ee4/volumes" Oct 30 17:25:18.935768 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:18.935720 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8200bff9-b1ac-41e9-812d-75ff15466ca4" path="/var/lib/kubelet/pods/8200bff9-b1ac-41e9-812d-75ff15466ca4/volumes" Oct 30 17:25:24.777946 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:24.777893 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x"] Oct 30 17:25:24.778557 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:24.778526 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x" podUID="18c895a8-b0d7-425c-81eb-282dd1766d59" containerName="istio-proxy" containerID="cri-o://9b6d7c415fb7a23bb4445a3949732b1afcd83c421b71f8bb96ab82d02f129d21" gracePeriod=30 Oct 30 17:25:25.379998 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:25.371628 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9bb5b1-rmwwt"] Oct 30 17:25:25.380972 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:25.380944 4493 cpu_manager.go:414] "containerMap: removing container" podUID="8200bff9-b1ac-41e9-812d-75ff15466ca4" containerName="revoke-keys" Oct 30 17:25:25.381147 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:25.381122 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8200bff9-b1ac-41e9-812d-75ff15466ca4" containerName="revoke-keys" Oct 30 17:25:25.381227 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:25.381159 4493 cpu_manager.go:414] "containerMap: removing container" podUID="99364c73-370a-443d-a082-454bc4fda069" containerName="revoke-keys" Oct 30 17:25:25.381227 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:25.381172 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99364c73-370a-443d-a082-454bc4fda069" containerName="revoke-keys" Oct 30 17:25:25.381227 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:25.381184 4493 cpu_manager.go:414] "containerMap: removing container" podUID="ee384285-d605-493f-bbb1-eec6e15d1549" containerName="revoke-keys" Oct 30 17:25:25.381227 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:25.381191 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee384285-d605-493f-bbb1-eec6e15d1549" containerName="revoke-keys" Oct 30 17:25:25.381457 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:25.381390 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="8200bff9-b1ac-41e9-812d-75ff15466ca4" containerName="revoke-keys" Oct 30 17:25:25.381457 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:25.381404 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="99364c73-370a-443d-a082-454bc4fda069" containerName="revoke-keys" Oct 30 17:25:25.381457 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:25.381414 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee384285-d605-493f-bbb1-eec6e15d1549" containerName="revoke-keys" Oct 30 17:25:25.386502 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:25.386468 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9bb5b1-rmwwt" Oct 30 17:25:25.407804 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:25.406823 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:25:25.407804 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:25.407024 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:25:25.407804 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:25.407159 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 17:25:25.408679 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:25.408651 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9bb5b1-rmwwt"] Oct 30 17:25:25.409112 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:25.409096 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:25:25.500452 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:25.500418 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6nm8\" (UniqueName: \"kubernetes.io/projected/712bd7c8-327a-42a8-b0a4-2ef420590395-kube-api-access-x6nm8\") pod \"maas-api-revoke-keys-e2e-extmod-a-9bb5b1-rmwwt\" (UID: \"712bd7c8-327a-42a8-b0a4-2ef420590395\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9bb5b1-rmwwt" Oct 30 17:25:25.500638 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:25.500483 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/712bd7c8-327a-42a8-b0a4-2ef420590395-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-9bb5b1-rmwwt\" (UID: \"712bd7c8-327a-42a8-b0a4-2ef420590395\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9bb5b1-rmwwt" Oct 30 17:25:25.500638 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:25.500579 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/712bd7c8-327a-42a8-b0a4-2ef420590395-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-9bb5b1-rmwwt\" (UID: \"712bd7c8-327a-42a8-b0a4-2ef420590395\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9bb5b1-rmwwt" Oct 30 17:25:25.602759 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:25.602717 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6nm8\" (UniqueName: \"kubernetes.io/projected/712bd7c8-327a-42a8-b0a4-2ef420590395-kube-api-access-x6nm8\") pod \"maas-api-revoke-keys-e2e-extmod-a-9bb5b1-rmwwt\" (UID: \"712bd7c8-327a-42a8-b0a4-2ef420590395\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9bb5b1-rmwwt" Oct 30 17:25:25.603033 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:25.602837 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/712bd7c8-327a-42a8-b0a4-2ef420590395-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-9bb5b1-rmwwt\" (UID: \"712bd7c8-327a-42a8-b0a4-2ef420590395\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9bb5b1-rmwwt" Oct 30 17:25:25.603033 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:25.602899 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/712bd7c8-327a-42a8-b0a4-2ef420590395-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-9bb5b1-rmwwt\" (UID: \"712bd7c8-327a-42a8-b0a4-2ef420590395\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9bb5b1-rmwwt" Oct 30 17:25:25.604275 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:25.604239 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/712bd7c8-327a-42a8-b0a4-2ef420590395-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-9bb5b1-rmwwt\" (UID: \"712bd7c8-327a-42a8-b0a4-2ef420590395\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9bb5b1-rmwwt" Oct 30 17:25:25.612466 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:25.612434 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/712bd7c8-327a-42a8-b0a4-2ef420590395-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-9bb5b1-rmwwt\" (UID: \"712bd7c8-327a-42a8-b0a4-2ef420590395\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9bb5b1-rmwwt" Oct 30 17:25:25.678176 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:25.678138 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 17:25:25.693262 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:25.692122 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6nm8\" (UniqueName: \"kubernetes.io/projected/712bd7c8-327a-42a8-b0a4-2ef420590395-kube-api-access-x6nm8\") pod \"maas-api-revoke-keys-e2e-extmod-a-9bb5b1-rmwwt\" (UID: \"712bd7c8-327a-42a8-b0a4-2ef420590395\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9bb5b1-rmwwt" Oct 30 17:25:25.706792 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:25.706762 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9bb5b1-rmwwt" Oct 30 17:25:26.204394 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:25:26.204358 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod712bd7c8_327a_42a8_b0a4_2ef420590395.slice/crio-063dba89ae57dc262d2c3c42ca9bfb8dbf1e02ff97336918a2d9b7fe92173b68 WatchSource:0}: Error finding container 063dba89ae57dc262d2c3c42ca9bfb8dbf1e02ff97336918a2d9b7fe92173b68: Status 404 returned error can't find the container with id 063dba89ae57dc262d2c3c42ca9bfb8dbf1e02ff97336918a2d9b7fe92173b68 Oct 30 17:25:26.204967 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:26.204892 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9bb5b1-rmwwt"] Oct 30 17:25:26.262491 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:26.262454 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5c6b3c-gh6cm"] Oct 30 17:25:26.268300 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:26.268274 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5c6b3c-gh6cm" Oct 30 17:25:26.346810 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:26.343644 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5c6b3c-gh6cm"] Oct 30 17:25:26.357808 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:26.354035 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80635646-db40-4d39-b3f4-e3a35108ebb3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-5c6b3c-gh6cm\" (UID: \"80635646-db40-4d39-b3f4-e3a35108ebb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5c6b3c-gh6cm" Oct 30 17:25:26.357808 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:26.354155 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4trp\" (UniqueName: \"kubernetes.io/projected/80635646-db40-4d39-b3f4-e3a35108ebb3-kube-api-access-p4trp\") pod \"maas-api-revoke-keys-e2e-extmod-b-5c6b3c-gh6cm\" (UID: \"80635646-db40-4d39-b3f4-e3a35108ebb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5c6b3c-gh6cm" Oct 30 17:25:26.357808 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:26.354199 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80635646-db40-4d39-b3f4-e3a35108ebb3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-5c6b3c-gh6cm\" (UID: \"80635646-db40-4d39-b3f4-e3a35108ebb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5c6b3c-gh6cm" Oct 30 17:25:26.456755 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:26.456662 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80635646-db40-4d39-b3f4-e3a35108ebb3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-5c6b3c-gh6cm\" (UID: \"80635646-db40-4d39-b3f4-e3a35108ebb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5c6b3c-gh6cm" Oct 30 17:25:26.498884 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:26.461140 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80635646-db40-4d39-b3f4-e3a35108ebb3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-5c6b3c-gh6cm\" (UID: \"80635646-db40-4d39-b3f4-e3a35108ebb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5c6b3c-gh6cm" Oct 30 17:25:26.499092 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:26.499012 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4trp\" (UniqueName: \"kubernetes.io/projected/80635646-db40-4d39-b3f4-e3a35108ebb3-kube-api-access-p4trp\") pod \"maas-api-revoke-keys-e2e-extmod-b-5c6b3c-gh6cm\" (UID: \"80635646-db40-4d39-b3f4-e3a35108ebb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5c6b3c-gh6cm" Oct 30 17:25:26.499201 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:26.499087 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80635646-db40-4d39-b3f4-e3a35108ebb3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-5c6b3c-gh6cm\" (UID: \"80635646-db40-4d39-b3f4-e3a35108ebb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5c6b3c-gh6cm" Oct 30 17:25:26.503474 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:26.503448 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80635646-db40-4d39-b3f4-e3a35108ebb3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-5c6b3c-gh6cm\" (UID: \"80635646-db40-4d39-b3f4-e3a35108ebb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5c6b3c-gh6cm" Oct 30 17:25:26.526316 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:26.526276 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4trp\" (UniqueName: \"kubernetes.io/projected/80635646-db40-4d39-b3f4-e3a35108ebb3-kube-api-access-p4trp\") pod \"maas-api-revoke-keys-e2e-extmod-b-5c6b3c-gh6cm\" (UID: \"80635646-db40-4d39-b3f4-e3a35108ebb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5c6b3c-gh6cm" Oct 30 17:25:26.600743 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:26.600706 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5c6b3c-gh6cm" Oct 30 17:25:26.706876 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:26.705407 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9bb5b1-rmwwt" event={"ID":"712bd7c8-327a-42a8-b0a4-2ef420590395","Type":"ContainerStarted","Data":"84713e584055094b178976ca16987fbcb7d29575ec65854bb95606f8e20e8cb4"} Oct 30 17:25:26.706876 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:26.705505 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9bb5b1-rmwwt" event={"ID":"712bd7c8-327a-42a8-b0a4-2ef420590395","Type":"ContainerStarted","Data":"063dba89ae57dc262d2c3c42ca9bfb8dbf1e02ff97336918a2d9b7fe92173b68"} Oct 30 17:25:26.751870 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:26.751663 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9bb5b1-rmwwt" podStartSLOduration=1.751642997 podStartE2EDuration="1.751642997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:25:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:25:26.74667536 +0000 UTC m=+3753.973850461" watchObservedRunningTime="2026-10-30 17:25:26.751642997 +0000 UTC m=+3753.978818099" Oct 30 17:25:26.927753 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:25:26.927068 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80635646_db40_4d39_b3f4_e3a35108ebb3.slice/crio-c070cfe470b9785023b0c7f837cd441819ed04528c86f88d01158550e30ff525 WatchSource:0}: Error finding container c070cfe470b9785023b0c7f837cd441819ed04528c86f88d01158550e30ff525: Status 404 returned error can't find the container with id c070cfe470b9785023b0c7f837cd441819ed04528c86f88d01158550e30ff525 Oct 30 17:25:26.937096 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:26.937026 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5c6b3c-gh6cm"] Oct 30 17:25:27.194807 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:27.192769 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk"] Oct 30 17:25:27.194807 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:27.193085 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk" podUID="e934b3f5-e168-4760-9ea6-3b9cab799328" containerName="istio-proxy" containerID="cri-o://53b1681933dfad608e75adebf0a3b5e227685beec5a62059d65ef8dcc36d2c56" gracePeriod=30 Oct 30 17:25:27.716602 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:27.716559 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5c6b3c-gh6cm" event={"ID":"80635646-db40-4d39-b3f4-e3a35108ebb3","Type":"ContainerStarted","Data":"2cedcab2947e2be4feac9cffc2138df7fe9b0b5e8a1933c84a5cd3e4f0a24672"} Oct 30 17:25:27.717140 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:27.716632 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5c6b3c-gh6cm" event={"ID":"80635646-db40-4d39-b3f4-e3a35108ebb3","Type":"ContainerStarted","Data":"c070cfe470b9785023b0c7f837cd441819ed04528c86f88d01158550e30ff525"} Oct 30 17:25:27.718524 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:27.718493 4493 generic.go:356] "Generic (PLEG): container finished" podID="712bd7c8-327a-42a8-b0a4-2ef420590395" containerID="84713e584055094b178976ca16987fbcb7d29575ec65854bb95606f8e20e8cb4" exitCode=0 Oct 30 17:25:27.718666 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:27.718569 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9bb5b1-rmwwt" event={"ID":"712bd7c8-327a-42a8-b0a4-2ef420590395","Type":"ContainerDied","Data":"84713e584055094b178976ca16987fbcb7d29575ec65854bb95606f8e20e8cb4"} Oct 30 17:25:27.763122 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:27.763069 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5c6b3c-gh6cm" podStartSLOduration=1.763056467 podStartE2EDuration="1.763056467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:25:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:25:27.758559753 +0000 UTC m=+3754.985734852" watchObservedRunningTime="2026-10-30 17:25:27.763056467 +0000 UTC m=+3754.990231567" Oct 30 17:25:27.814997 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:27.814966 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3106d6-j6rcg"] Oct 30 17:25:27.819464 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:27.819440 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3106d6-j6rcg" Oct 30 17:25:27.839637 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:27.839583 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3106d6-j6rcg"] Oct 30 17:25:27.841724 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:27.841700 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d891422-969c-4fb0-b35b-4c4adeb61f41-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-3106d6-j6rcg\" (UID: \"2d891422-969c-4fb0-b35b-4c4adeb61f41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3106d6-j6rcg" Oct 30 17:25:27.841852 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:27.841764 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d891422-969c-4fb0-b35b-4c4adeb61f41-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-3106d6-j6rcg\" (UID: \"2d891422-969c-4fb0-b35b-4c4adeb61f41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3106d6-j6rcg" Oct 30 17:25:27.841950 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:27.841924 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfhbv\" (UniqueName: \"kubernetes.io/projected/2d891422-969c-4fb0-b35b-4c4adeb61f41-kube-api-access-bfhbv\") pod \"maas-api-revoke-keys-e2e-extmod-a-3106d6-j6rcg\" (UID: \"2d891422-969c-4fb0-b35b-4c4adeb61f41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3106d6-j6rcg" Oct 30 17:25:27.943047 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:27.943017 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d891422-969c-4fb0-b35b-4c4adeb61f41-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-3106d6-j6rcg\" (UID: \"2d891422-969c-4fb0-b35b-4c4adeb61f41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3106d6-j6rcg" Oct 30 17:25:27.943218 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:27.943096 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d891422-969c-4fb0-b35b-4c4adeb61f41-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-3106d6-j6rcg\" (UID: \"2d891422-969c-4fb0-b35b-4c4adeb61f41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3106d6-j6rcg" Oct 30 17:25:27.943218 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:27.943137 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfhbv\" (UniqueName: \"kubernetes.io/projected/2d891422-969c-4fb0-b35b-4c4adeb61f41-kube-api-access-bfhbv\") pod \"maas-api-revoke-keys-e2e-extmod-a-3106d6-j6rcg\" (UID: \"2d891422-969c-4fb0-b35b-4c4adeb61f41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3106d6-j6rcg" Oct 30 17:25:27.943840 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:27.943810 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d891422-969c-4fb0-b35b-4c4adeb61f41-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-3106d6-j6rcg\" (UID: \"2d891422-969c-4fb0-b35b-4c4adeb61f41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3106d6-j6rcg" Oct 30 17:25:27.946201 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:27.946171 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d891422-969c-4fb0-b35b-4c4adeb61f41-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-3106d6-j6rcg\" (UID: \"2d891422-969c-4fb0-b35b-4c4adeb61f41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3106d6-j6rcg" Oct 30 17:25:27.966036 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:27.966009 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfhbv\" (UniqueName: \"kubernetes.io/projected/2d891422-969c-4fb0-b35b-4c4adeb61f41-kube-api-access-bfhbv\") pod \"maas-api-revoke-keys-e2e-extmod-a-3106d6-j6rcg\" (UID: \"2d891422-969c-4fb0-b35b-4c4adeb61f41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3106d6-j6rcg" Oct 30 17:25:28.136362 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:28.136269 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3106d6-j6rcg" Oct 30 17:25:28.458429 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:28.457259 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3106d6-j6rcg"] Oct 30 17:25:28.460137 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:25:28.460106 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d891422_969c_4fb0_b35b_4c4adeb61f41.slice/crio-c5c71f338fa70c8ca6fdeb8509ffed11636bf8283ffccccd5dc3c5b76a35b400 WatchSource:0}: Error finding container c5c71f338fa70c8ca6fdeb8509ffed11636bf8283ffccccd5dc3c5b76a35b400: Status 404 returned error can't find the container with id c5c71f338fa70c8ca6fdeb8509ffed11636bf8283ffccccd5dc3c5b76a35b400 Oct 30 17:25:28.730556 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:28.730461 4493 generic.go:356] "Generic (PLEG): container finished" podID="80635646-db40-4d39-b3f4-e3a35108ebb3" containerID="2cedcab2947e2be4feac9cffc2138df7fe9b0b5e8a1933c84a5cd3e4f0a24672" exitCode=0 Oct 30 17:25:28.730556 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:28.730544 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5c6b3c-gh6cm" event={"ID":"80635646-db40-4d39-b3f4-e3a35108ebb3","Type":"ContainerDied","Data":"2cedcab2947e2be4feac9cffc2138df7fe9b0b5e8a1933c84a5cd3e4f0a24672"} Oct 30 17:25:28.732178 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:28.732151 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3106d6-j6rcg" event={"ID":"2d891422-969c-4fb0-b35b-4c4adeb61f41","Type":"ContainerStarted","Data":"c5c71f338fa70c8ca6fdeb8509ffed11636bf8283ffccccd5dc3c5b76a35b400"} Oct 30 17:25:28.907324 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:28.907296 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9bb5b1-rmwwt" Oct 30 17:25:28.964065 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:28.964024 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/712bd7c8-327a-42a8-b0a4-2ef420590395-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/712bd7c8-327a-42a8-b0a4-2ef420590395-maas-api-openshift-ca\") pod \"712bd7c8-327a-42a8-b0a4-2ef420590395\" (UID: \"712bd7c8-327a-42a8-b0a4-2ef420590395\") " Oct 30 17:25:28.964218 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:28.964125 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/712bd7c8-327a-42a8-b0a4-2ef420590395-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/712bd7c8-327a-42a8-b0a4-2ef420590395-maas-api-xks-ca\") pod \"712bd7c8-327a-42a8-b0a4-2ef420590395\" (UID: \"712bd7c8-327a-42a8-b0a4-2ef420590395\") " Oct 30 17:25:28.964293 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:28.964244 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/712bd7c8-327a-42a8-b0a4-2ef420590395-kube-api-access-x6nm8\" (UniqueName: \"kubernetes.io/projected/712bd7c8-327a-42a8-b0a4-2ef420590395-kube-api-access-x6nm8\") pod \"712bd7c8-327a-42a8-b0a4-2ef420590395\" (UID: \"712bd7c8-327a-42a8-b0a4-2ef420590395\") " Oct 30 17:25:28.964457 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:28.964415 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/712bd7c8-327a-42a8-b0a4-2ef420590395-maas-api-openshift-ca" pod "712bd7c8-327a-42a8-b0a4-2ef420590395" (UID: "712bd7c8-327a-42a8-b0a4-2ef420590395"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:25:28.965632 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:28.964940 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/712bd7c8-327a-42a8-b0a4-2ef420590395-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:25:28.967292 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:28.967259 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/712bd7c8-327a-42a8-b0a4-2ef420590395-kube-api-access-x6nm8" pod "712bd7c8-327a-42a8-b0a4-2ef420590395" (UID: "712bd7c8-327a-42a8-b0a4-2ef420590395"). InnerVolumeSpecName "kube-api-access-x6nm8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:25:28.969573 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:28.969548 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/712bd7c8-327a-42a8-b0a4-2ef420590395-maas-api-xks-ca" pod "712bd7c8-327a-42a8-b0a4-2ef420590395" (UID: "712bd7c8-327a-42a8-b0a4-2ef420590395"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:25:29.066106 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:29.066028 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/712bd7c8-327a-42a8-b0a4-2ef420590395-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:25:29.066106 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:29.066056 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6nm8\" (UniqueName: \"kubernetes.io/projected/712bd7c8-327a-42a8-b0a4-2ef420590395-kube-api-access-x6nm8\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:25:29.741971 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:29.741920 4493 generic.go:356] "Generic (PLEG): container finished" podID="2d891422-969c-4fb0-b35b-4c4adeb61f41" containerID="05a20345b5fd389429df7995ee399b3e7dd2ccecbb1ddc2c86d3d5d8930eb835" exitCode=0 Oct 30 17:25:29.742541 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:29.741968 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3106d6-j6rcg" event={"ID":"2d891422-969c-4fb0-b35b-4c4adeb61f41","Type":"ContainerDied","Data":"05a20345b5fd389429df7995ee399b3e7dd2ccecbb1ddc2c86d3d5d8930eb835"} Oct 30 17:25:29.743934 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:29.743914 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9bb5b1-rmwwt" Oct 30 17:25:29.744077 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:29.743952 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9bb5b1-rmwwt" event={"ID":"712bd7c8-327a-42a8-b0a4-2ef420590395","Type":"ContainerDied","Data":"063dba89ae57dc262d2c3c42ca9bfb8dbf1e02ff97336918a2d9b7fe92173b68"} Oct 30 17:25:29.744077 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:29.743997 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="063dba89ae57dc262d2c3c42ca9bfb8dbf1e02ff97336918a2d9b7fe92173b68" Oct 30 17:25:29.770703 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:29.770674 4493 patch_prober.go:29] interesting pod/e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:25:29.770878 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:29.770732 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x" podUID="18c895a8-b0d7-425c-81eb-282dd1766d59" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:25:29.908070 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:29.908047 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5c6b3c-gh6cm" Oct 30 17:25:29.978861 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:29.978814 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80635646-db40-4d39-b3f4-e3a35108ebb3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80635646-db40-4d39-b3f4-e3a35108ebb3-maas-api-xks-ca\") pod \"80635646-db40-4d39-b3f4-e3a35108ebb3\" (UID: \"80635646-db40-4d39-b3f4-e3a35108ebb3\") " Oct 30 17:25:29.979101 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:29.979077 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80635646-db40-4d39-b3f4-e3a35108ebb3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80635646-db40-4d39-b3f4-e3a35108ebb3-maas-api-openshift-ca\") pod \"80635646-db40-4d39-b3f4-e3a35108ebb3\" (UID: \"80635646-db40-4d39-b3f4-e3a35108ebb3\") " Oct 30 17:25:29.979235 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:29.979194 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80635646-db40-4d39-b3f4-e3a35108ebb3-kube-api-access-p4trp\" (UniqueName: \"kubernetes.io/projected/80635646-db40-4d39-b3f4-e3a35108ebb3-kube-api-access-p4trp\") pod \"80635646-db40-4d39-b3f4-e3a35108ebb3\" (UID: \"80635646-db40-4d39-b3f4-e3a35108ebb3\") " Oct 30 17:25:29.979866 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:29.979592 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80635646-db40-4d39-b3f4-e3a35108ebb3-maas-api-openshift-ca" pod "80635646-db40-4d39-b3f4-e3a35108ebb3" (UID: "80635646-db40-4d39-b3f4-e3a35108ebb3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:25:29.979866 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:29.979761 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80635646-db40-4d39-b3f4-e3a35108ebb3-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:25:29.982422 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:29.982394 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80635646-db40-4d39-b3f4-e3a35108ebb3-maas-api-xks-ca" pod "80635646-db40-4d39-b3f4-e3a35108ebb3" (UID: "80635646-db40-4d39-b3f4-e3a35108ebb3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:25:29.982623 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:29.982599 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80635646-db40-4d39-b3f4-e3a35108ebb3-kube-api-access-p4trp" pod "80635646-db40-4d39-b3f4-e3a35108ebb3" (UID: "80635646-db40-4d39-b3f4-e3a35108ebb3"). InnerVolumeSpecName "kube-api-access-p4trp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:25:30.081256 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:30.081222 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4trp\" (UniqueName: \"kubernetes.io/projected/80635646-db40-4d39-b3f4-e3a35108ebb3-kube-api-access-p4trp\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:25:30.081256 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:30.081251 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80635646-db40-4d39-b3f4-e3a35108ebb3-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:25:30.113652 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:30.113460 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x" Oct 30 17:25:30.182684 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:30.182649 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18c895a8-b0d7-425c-81eb-282dd1766d59-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c895a8-b0d7-425c-81eb-282dd1766d59-credential-socket\") pod \"18c895a8-b0d7-425c-81eb-282dd1766d59\" (UID: \"18c895a8-b0d7-425c-81eb-282dd1766d59\") " Oct 30 17:25:30.182855 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:30.182702 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18c895a8-b0d7-425c-81eb-282dd1766d59-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c895a8-b0d7-425c-81eb-282dd1766d59-workload-socket\") pod \"18c895a8-b0d7-425c-81eb-282dd1766d59\" (UID: \"18c895a8-b0d7-425c-81eb-282dd1766d59\") " Oct 30 17:25:30.182855 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:30.182758 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18c895a8-b0d7-425c-81eb-282dd1766d59-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18c895a8-b0d7-425c-81eb-282dd1766d59-istio-envoy\") pod \"18c895a8-b0d7-425c-81eb-282dd1766d59\" (UID: \"18c895a8-b0d7-425c-81eb-282dd1766d59\") " Oct 30 17:25:30.182855 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:30.182838 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18c895a8-b0d7-425c-81eb-282dd1766d59-istio-token\" (UniqueName: \"kubernetes.io/projected/18c895a8-b0d7-425c-81eb-282dd1766d59-istio-token\") pod \"18c895a8-b0d7-425c-81eb-282dd1766d59\" (UID: \"18c895a8-b0d7-425c-81eb-282dd1766d59\") " Oct 30 17:25:30.183025 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:30.182935 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/18c895a8-b0d7-425c-81eb-282dd1766d59-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18c895a8-b0d7-425c-81eb-282dd1766d59-istio-podinfo\") pod \"18c895a8-b0d7-425c-81eb-282dd1766d59\" (UID: \"18c895a8-b0d7-425c-81eb-282dd1766d59\") " Oct 30 17:25:30.183081 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:30.183022 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18c895a8-b0d7-425c-81eb-282dd1766d59-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18c895a8-b0d7-425c-81eb-282dd1766d59-workload-certs\") pod \"18c895a8-b0d7-425c-81eb-282dd1766d59\" (UID: \"18c895a8-b0d7-425c-81eb-282dd1766d59\") " Oct 30 17:25:30.183081 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:30.183044 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18c895a8-b0d7-425c-81eb-282dd1766d59-credential-socket" pod "18c895a8-b0d7-425c-81eb-282dd1766d59" (UID: "18c895a8-b0d7-425c-81eb-282dd1766d59"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:25:30.183196 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:30.183077 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18c895a8-b0d7-425c-81eb-282dd1766d59-workload-socket" pod "18c895a8-b0d7-425c-81eb-282dd1766d59" (UID: "18c895a8-b0d7-425c-81eb-282dd1766d59"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:25:30.183196 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:30.183139 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18c895a8-b0d7-425c-81eb-282dd1766d59-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18c895a8-b0d7-425c-81eb-282dd1766d59-istio-data\") pod \"18c895a8-b0d7-425c-81eb-282dd1766d59\" (UID: \"18c895a8-b0d7-425c-81eb-282dd1766d59\") " Oct 30 17:25:30.183314 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:30.183194 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18c895a8-b0d7-425c-81eb-282dd1766d59-kube-api-access-88vsm\" (UniqueName: \"kubernetes.io/projected/18c895a8-b0d7-425c-81eb-282dd1766d59-kube-api-access-88vsm\") pod \"18c895a8-b0d7-425c-81eb-282dd1766d59\" (UID: \"18c895a8-b0d7-425c-81eb-282dd1766d59\") " Oct 30 17:25:30.183314 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:30.183228 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18c895a8-b0d7-425c-81eb-282dd1766d59-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18c895a8-b0d7-425c-81eb-282dd1766d59-istiod-ca-cert\") pod \"18c895a8-b0d7-425c-81eb-282dd1766d59\" (UID: \"18c895a8-b0d7-425c-81eb-282dd1766d59\") " Oct 30 17:25:30.183423 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:30.183375 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18c895a8-b0d7-425c-81eb-282dd1766d59-workload-certs" pod "18c895a8-b0d7-425c-81eb-282dd1766d59" (UID: "18c895a8-b0d7-425c-81eb-282dd1766d59"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:25:30.183496 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:30.183474 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18c895a8-b0d7-425c-81eb-282dd1766d59-istio-data" pod "18c895a8-b0d7-425c-81eb-282dd1766d59" (UID: "18c895a8-b0d7-425c-81eb-282dd1766d59"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:25:30.183705 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:30.183684 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18c895a8-b0d7-425c-81eb-282dd1766d59-istiod-ca-cert" pod "18c895a8-b0d7-425c-81eb-282dd1766d59" (UID: "18c895a8-b0d7-425c-81eb-282dd1766d59"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:25:30.183936 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:30.183918 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18c895a8-b0d7-425c-81eb-282dd1766d59-istio-data\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:25:30.184027 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:30.183941 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18c895a8-b0d7-425c-81eb-282dd1766d59-istiod-ca-cert\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:25:30.184027 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:30.183956 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c895a8-b0d7-425c-81eb-282dd1766d59-credential-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:25:30.184027 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:30.183972 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c895a8-b0d7-425c-81eb-282dd1766d59-workload-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:25:30.184027 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:30.183985 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18c895a8-b0d7-425c-81eb-282dd1766d59-workload-certs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:25:30.185937 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:30.185914 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18c895a8-b0d7-425c-81eb-282dd1766d59-istio-envoy" pod "18c895a8-b0d7-425c-81eb-282dd1766d59" (UID: "18c895a8-b0d7-425c-81eb-282dd1766d59"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:25:30.186291 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:30.186267 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/18c895a8-b0d7-425c-81eb-282dd1766d59-istio-podinfo" pod "18c895a8-b0d7-425c-81eb-282dd1766d59" (UID: "18c895a8-b0d7-425c-81eb-282dd1766d59"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:25:30.186366 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:30.186348 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18c895a8-b0d7-425c-81eb-282dd1766d59-kube-api-access-88vsm" pod "18c895a8-b0d7-425c-81eb-282dd1766d59" (UID: "18c895a8-b0d7-425c-81eb-282dd1766d59"). InnerVolumeSpecName "kube-api-access-88vsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:25:30.186425 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:30.186412 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18c895a8-b0d7-425c-81eb-282dd1766d59-istio-token" pod "18c895a8-b0d7-425c-81eb-282dd1766d59" (UID: "18c895a8-b0d7-425c-81eb-282dd1766d59"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:25:30.286064 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:30.285963 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18c895a8-b0d7-425c-81eb-282dd1766d59-istio-podinfo\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:25:30.286064 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:30.286007 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88vsm\" (UniqueName: \"kubernetes.io/projected/18c895a8-b0d7-425c-81eb-282dd1766d59-kube-api-access-88vsm\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:25:30.286064 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:30.286024 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18c895a8-b0d7-425c-81eb-282dd1766d59-istio-envoy\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:25:30.286064 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:30.286036 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18c895a8-b0d7-425c-81eb-282dd1766d59-istio-token\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:25:30.350770 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:25:30.350734 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 17:25:30.753534 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:30.753495 4493 generic.go:356] "Generic (PLEG): container finished" podID="18c895a8-b0d7-425c-81eb-282dd1766d59" containerID="9b6d7c415fb7a23bb4445a3949732b1afcd83c421b71f8bb96ab82d02f129d21" exitCode=0 Oct 30 17:25:30.754049 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:30.753567 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x" Oct 30 17:25:30.754049 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:30.753581 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x" event={"ID":"18c895a8-b0d7-425c-81eb-282dd1766d59","Type":"ContainerDied","Data":"9b6d7c415fb7a23bb4445a3949732b1afcd83c421b71f8bb96ab82d02f129d21"} Oct 30 17:25:30.754049 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:30.753692 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x" event={"ID":"18c895a8-b0d7-425c-81eb-282dd1766d59","Type":"ContainerDied","Data":"dca96e9821492069910280cd9445ad4dc8bfe22805f31a75c13ba19e90f46663"} Oct 30 17:25:30.754049 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:30.753712 4493 scope.go:122] "RemoveContainer" containerID="9b6d7c415fb7a23bb4445a3949732b1afcd83c421b71f8bb96ab82d02f129d21" Oct 30 17:25:30.755407 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:30.755291 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5c6b3c-gh6cm" Oct 30 17:25:30.755407 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:30.755298 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5c6b3c-gh6cm" event={"ID":"80635646-db40-4d39-b3f4-e3a35108ebb3","Type":"ContainerDied","Data":"c070cfe470b9785023b0c7f837cd441819ed04528c86f88d01158550e30ff525"} Oct 30 17:25:30.755407 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:30.755340 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c070cfe470b9785023b0c7f837cd441819ed04528c86f88d01158550e30ff525" Oct 30 17:25:30.777107 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:30.777088 4493 scope.go:122] "RemoveContainer" containerID="1b94d423f4e040453995404381a7c43ce5b63dde548a6758b4e1811a815c94e1" Oct 30 17:25:30.788342 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:30.788317 4493 scope.go:122] "RemoveContainer" containerID="9b6d7c415fb7a23bb4445a3949732b1afcd83c421b71f8bb96ab82d02f129d21" Oct 30 17:25:30.788673 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:25:30.788648 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b6d7c415fb7a23bb4445a3949732b1afcd83c421b71f8bb96ab82d02f129d21\": container with ID starting with 9b6d7c415fb7a23bb4445a3949732b1afcd83c421b71f8bb96ab82d02f129d21 not found: ID does not exist" containerID="9b6d7c415fb7a23bb4445a3949732b1afcd83c421b71f8bb96ab82d02f129d21" Oct 30 17:25:30.788731 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:30.788687 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b6d7c415fb7a23bb4445a3949732b1afcd83c421b71f8bb96ab82d02f129d21"} err="failed to get container status \"9b6d7c415fb7a23bb4445a3949732b1afcd83c421b71f8bb96ab82d02f129d21\": rpc error: code = NotFound desc = could not find container \"9b6d7c415fb7a23bb4445a3949732b1afcd83c421b71f8bb96ab82d02f129d21\": container with ID starting with 9b6d7c415fb7a23bb4445a3949732b1afcd83c421b71f8bb96ab82d02f129d21 not found: ID does not exist" Oct 30 17:25:30.788731 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:30.788710 4493 scope.go:122] "RemoveContainer" containerID="1b94d423f4e040453995404381a7c43ce5b63dde548a6758b4e1811a815c94e1" Oct 30 17:25:30.789042 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:25:30.789023 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b94d423f4e040453995404381a7c43ce5b63dde548a6758b4e1811a815c94e1\": container with ID starting with 1b94d423f4e040453995404381a7c43ce5b63dde548a6758b4e1811a815c94e1 not found: ID does not exist" containerID="1b94d423f4e040453995404381a7c43ce5b63dde548a6758b4e1811a815c94e1" Oct 30 17:25:30.789107 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:30.789056 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b94d423f4e040453995404381a7c43ce5b63dde548a6758b4e1811a815c94e1"} err="failed to get container status \"1b94d423f4e040453995404381a7c43ce5b63dde548a6758b4e1811a815c94e1\": rpc error: code = NotFound desc = could not find container \"1b94d423f4e040453995404381a7c43ce5b63dde548a6758b4e1811a815c94e1\": container with ID starting with 1b94d423f4e040453995404381a7c43ce5b63dde548a6758b4e1811a815c94e1 not found: ID does not exist" Oct 30 17:25:30.815971 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:30.812950 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x"] Oct 30 17:25:30.816806 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:30.816768 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-bbc36f-openshift-default-756b7cbf4f-qrl4x"] Oct 30 17:25:30.900336 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:30.900313 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3106d6-j6rcg" Oct 30 17:25:30.933275 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:30.933250 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18c895a8-b0d7-425c-81eb-282dd1766d59" path="/var/lib/kubelet/pods/18c895a8-b0d7-425c-81eb-282dd1766d59/volumes" Oct 30 17:25:30.996199 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:30.996164 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d891422-969c-4fb0-b35b-4c4adeb61f41-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d891422-969c-4fb0-b35b-4c4adeb61f41-maas-api-xks-ca\") pod \"2d891422-969c-4fb0-b35b-4c4adeb61f41\" (UID: \"2d891422-969c-4fb0-b35b-4c4adeb61f41\") " Oct 30 17:25:30.996362 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:30.996256 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d891422-969c-4fb0-b35b-4c4adeb61f41-kube-api-access-bfhbv\" (UniqueName: \"kubernetes.io/projected/2d891422-969c-4fb0-b35b-4c4adeb61f41-kube-api-access-bfhbv\") pod \"2d891422-969c-4fb0-b35b-4c4adeb61f41\" (UID: \"2d891422-969c-4fb0-b35b-4c4adeb61f41\") " Oct 30 17:25:30.996493 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:30.996476 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d891422-969c-4fb0-b35b-4c4adeb61f41-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d891422-969c-4fb0-b35b-4c4adeb61f41-maas-api-openshift-ca\") pod \"2d891422-969c-4fb0-b35b-4c4adeb61f41\" (UID: \"2d891422-969c-4fb0-b35b-4c4adeb61f41\") " Oct 30 17:25:30.997041 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:30.997007 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d891422-969c-4fb0-b35b-4c4adeb61f41-maas-api-openshift-ca" pod "2d891422-969c-4fb0-b35b-4c4adeb61f41" (UID: "2d891422-969c-4fb0-b35b-4c4adeb61f41"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:25:30.999217 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:30.999190 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d891422-969c-4fb0-b35b-4c4adeb61f41-maas-api-xks-ca" pod "2d891422-969c-4fb0-b35b-4c4adeb61f41" (UID: "2d891422-969c-4fb0-b35b-4c4adeb61f41"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:25:30.999349 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:30.999232 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d891422-969c-4fb0-b35b-4c4adeb61f41-kube-api-access-bfhbv" pod "2d891422-969c-4fb0-b35b-4c4adeb61f41" (UID: "2d891422-969c-4fb0-b35b-4c4adeb61f41"). InnerVolumeSpecName "kube-api-access-bfhbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:25:31.098684 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:31.098654 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d891422-969c-4fb0-b35b-4c4adeb61f41-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:25:31.098684 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:31.098682 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d891422-969c-4fb0-b35b-4c4adeb61f41-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:25:31.098966 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:31.098692 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfhbv\" (UniqueName: \"kubernetes.io/projected/2d891422-969c-4fb0-b35b-4c4adeb61f41-kube-api-access-bfhbv\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:25:31.767109 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:31.767078 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3106d6-j6rcg" Oct 30 17:25:31.767591 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:31.767106 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3106d6-j6rcg" event={"ID":"2d891422-969c-4fb0-b35b-4c4adeb61f41","Type":"ContainerDied","Data":"c5c71f338fa70c8ca6fdeb8509ffed11636bf8283ffccccd5dc3c5b76a35b400"} Oct 30 17:25:31.767591 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:31.767152 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5c71f338fa70c8ca6fdeb8509ffed11636bf8283ffccccd5dc3c5b76a35b400" Oct 30 17:25:32.491344 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:32.491314 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk" Oct 30 17:25:32.617516 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:32.617476 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e934b3f5-e168-4760-9ea6-3b9cab799328-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e934b3f5-e168-4760-9ea6-3b9cab799328-istio-envoy\") pod \"e934b3f5-e168-4760-9ea6-3b9cab799328\" (UID: \"e934b3f5-e168-4760-9ea6-3b9cab799328\") " Oct 30 17:25:32.617705 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:32.617548 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e934b3f5-e168-4760-9ea6-3b9cab799328-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e934b3f5-e168-4760-9ea6-3b9cab799328-istiod-ca-cert\") pod \"e934b3f5-e168-4760-9ea6-3b9cab799328\" (UID: \"e934b3f5-e168-4760-9ea6-3b9cab799328\") " Oct 30 17:25:32.617705 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:32.617612 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e934b3f5-e168-4760-9ea6-3b9cab799328-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e934b3f5-e168-4760-9ea6-3b9cab799328-istio-podinfo\") pod \"e934b3f5-e168-4760-9ea6-3b9cab799328\" (UID: \"e934b3f5-e168-4760-9ea6-3b9cab799328\") " Oct 30 17:25:32.617705 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:32.617648 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e934b3f5-e168-4760-9ea6-3b9cab799328-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e934b3f5-e168-4760-9ea6-3b9cab799328-credential-socket\") pod \"e934b3f5-e168-4760-9ea6-3b9cab799328\" (UID: \"e934b3f5-e168-4760-9ea6-3b9cab799328\") " Oct 30 17:25:32.617705 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:32.617681 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e934b3f5-e168-4760-9ea6-3b9cab799328-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e934b3f5-e168-4760-9ea6-3b9cab799328-workload-socket\") pod \"e934b3f5-e168-4760-9ea6-3b9cab799328\" (UID: \"e934b3f5-e168-4760-9ea6-3b9cab799328\") " Oct 30 17:25:32.618042 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:32.617800 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e934b3f5-e168-4760-9ea6-3b9cab799328-istio-token\" (UniqueName: \"kubernetes.io/projected/e934b3f5-e168-4760-9ea6-3b9cab799328-istio-token\") pod \"e934b3f5-e168-4760-9ea6-3b9cab799328\" (UID: \"e934b3f5-e168-4760-9ea6-3b9cab799328\") " Oct 30 17:25:32.618042 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:32.617844 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e934b3f5-e168-4760-9ea6-3b9cab799328-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e934b3f5-e168-4760-9ea6-3b9cab799328-istio-data\") pod \"e934b3f5-e168-4760-9ea6-3b9cab799328\" (UID: \"e934b3f5-e168-4760-9ea6-3b9cab799328\") " Oct 30 17:25:32.618042 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:32.617904 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e934b3f5-e168-4760-9ea6-3b9cab799328-kube-api-access-8xqnd\" (UniqueName: \"kubernetes.io/projected/e934b3f5-e168-4760-9ea6-3b9cab799328-kube-api-access-8xqnd\") pod \"e934b3f5-e168-4760-9ea6-3b9cab799328\" (UID: \"e934b3f5-e168-4760-9ea6-3b9cab799328\") " Oct 30 17:25:32.618042 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:32.617942 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e934b3f5-e168-4760-9ea6-3b9cab799328-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e934b3f5-e168-4760-9ea6-3b9cab799328-workload-certs\") pod \"e934b3f5-e168-4760-9ea6-3b9cab799328\" (UID: \"e934b3f5-e168-4760-9ea6-3b9cab799328\") " Oct 30 17:25:32.618042 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:32.618029 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e934b3f5-e168-4760-9ea6-3b9cab799328-credential-socket" pod "e934b3f5-e168-4760-9ea6-3b9cab799328" (UID: "e934b3f5-e168-4760-9ea6-3b9cab799328"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:25:32.618332 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:32.618166 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e934b3f5-e168-4760-9ea6-3b9cab799328-istio-data" pod "e934b3f5-e168-4760-9ea6-3b9cab799328" (UID: "e934b3f5-e168-4760-9ea6-3b9cab799328"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:25:32.618332 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:32.618237 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e934b3f5-e168-4760-9ea6-3b9cab799328-workload-certs" pod "e934b3f5-e168-4760-9ea6-3b9cab799328" (UID: "e934b3f5-e168-4760-9ea6-3b9cab799328"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:25:32.618332 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:32.618296 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e934b3f5-e168-4760-9ea6-3b9cab799328-istiod-ca-cert" pod "e934b3f5-e168-4760-9ea6-3b9cab799328" (UID: "e934b3f5-e168-4760-9ea6-3b9cab799328"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:25:32.618679 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:32.618479 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e934b3f5-e168-4760-9ea6-3b9cab799328-workload-socket" pod "e934b3f5-e168-4760-9ea6-3b9cab799328" (UID: "e934b3f5-e168-4760-9ea6-3b9cab799328"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:25:32.618746 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:32.618731 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e934b3f5-e168-4760-9ea6-3b9cab799328-istiod-ca-cert\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:25:32.618828 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:32.618752 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e934b3f5-e168-4760-9ea6-3b9cab799328-credential-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:25:32.618828 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:32.618766 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e934b3f5-e168-4760-9ea6-3b9cab799328-workload-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:25:32.618828 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:32.618795 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e934b3f5-e168-4760-9ea6-3b9cab799328-istio-data\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:25:32.618828 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:32.618811 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e934b3f5-e168-4760-9ea6-3b9cab799328-workload-certs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:25:32.620229 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:32.620130 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e934b3f5-e168-4760-9ea6-3b9cab799328-istio-envoy" pod "e934b3f5-e168-4760-9ea6-3b9cab799328" (UID: "e934b3f5-e168-4760-9ea6-3b9cab799328"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:25:32.620369 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:32.620256 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e934b3f5-e168-4760-9ea6-3b9cab799328-istio-podinfo" pod "e934b3f5-e168-4760-9ea6-3b9cab799328" (UID: "e934b3f5-e168-4760-9ea6-3b9cab799328"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:25:32.620652 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:32.620630 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e934b3f5-e168-4760-9ea6-3b9cab799328-istio-token" pod "e934b3f5-e168-4760-9ea6-3b9cab799328" (UID: "e934b3f5-e168-4760-9ea6-3b9cab799328"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:25:32.620884 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:32.620864 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e934b3f5-e168-4760-9ea6-3b9cab799328-kube-api-access-8xqnd" pod "e934b3f5-e168-4760-9ea6-3b9cab799328" (UID: "e934b3f5-e168-4760-9ea6-3b9cab799328"). InnerVolumeSpecName "kube-api-access-8xqnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:25:32.719671 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:32.719641 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e934b3f5-e168-4760-9ea6-3b9cab799328-istio-token\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:25:32.719671 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:32.719669 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xqnd\" (UniqueName: \"kubernetes.io/projected/e934b3f5-e168-4760-9ea6-3b9cab799328-kube-api-access-8xqnd\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:25:32.719889 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:32.719679 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e934b3f5-e168-4760-9ea6-3b9cab799328-istio-envoy\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:25:32.719889 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:32.719689 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e934b3f5-e168-4760-9ea6-3b9cab799328-istio-podinfo\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:25:32.778148 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:32.778059 4493 generic.go:356] "Generic (PLEG): container finished" podID="e934b3f5-e168-4760-9ea6-3b9cab799328" containerID="53b1681933dfad608e75adebf0a3b5e227685beec5a62059d65ef8dcc36d2c56" exitCode=0 Oct 30 17:25:32.778148 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:32.778125 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk" Oct 30 17:25:32.778625 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:32.778156 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk" event={"ID":"e934b3f5-e168-4760-9ea6-3b9cab799328","Type":"ContainerDied","Data":"53b1681933dfad608e75adebf0a3b5e227685beec5a62059d65ef8dcc36d2c56"} Oct 30 17:25:32.778625 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:32.778219 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk" event={"ID":"e934b3f5-e168-4760-9ea6-3b9cab799328","Type":"ContainerDied","Data":"46eb75ec6470f77beb9a08cdabd83639cc45a67cd6f2b60f344e8c6426032006"} Oct 30 17:25:32.778625 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:32.778243 4493 scope.go:122] "RemoveContainer" containerID="53b1681933dfad608e75adebf0a3b5e227685beec5a62059d65ef8dcc36d2c56" Oct 30 17:25:32.790580 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:32.790560 4493 scope.go:122] "RemoveContainer" containerID="2847df9bcd14d1409887a15fb491be9a31a9ff83b9f5ee7e83ca6d7948d700d8" Oct 30 17:25:32.801422 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:32.801397 4493 scope.go:122] "RemoveContainer" containerID="53b1681933dfad608e75adebf0a3b5e227685beec5a62059d65ef8dcc36d2c56" Oct 30 17:25:32.801683 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:25:32.801661 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53b1681933dfad608e75adebf0a3b5e227685beec5a62059d65ef8dcc36d2c56\": container with ID starting with 53b1681933dfad608e75adebf0a3b5e227685beec5a62059d65ef8dcc36d2c56 not found: ID does not exist" containerID="53b1681933dfad608e75adebf0a3b5e227685beec5a62059d65ef8dcc36d2c56" Oct 30 17:25:32.801739 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:32.801692 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53b1681933dfad608e75adebf0a3b5e227685beec5a62059d65ef8dcc36d2c56"} err="failed to get container status \"53b1681933dfad608e75adebf0a3b5e227685beec5a62059d65ef8dcc36d2c56\": rpc error: code = NotFound desc = could not find container \"53b1681933dfad608e75adebf0a3b5e227685beec5a62059d65ef8dcc36d2c56\": container with ID starting with 53b1681933dfad608e75adebf0a3b5e227685beec5a62059d65ef8dcc36d2c56 not found: ID does not exist" Oct 30 17:25:32.801739 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:32.801710 4493 scope.go:122] "RemoveContainer" containerID="2847df9bcd14d1409887a15fb491be9a31a9ff83b9f5ee7e83ca6d7948d700d8" Oct 30 17:25:32.802032 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:25:32.802008 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2847df9bcd14d1409887a15fb491be9a31a9ff83b9f5ee7e83ca6d7948d700d8\": container with ID starting with 2847df9bcd14d1409887a15fb491be9a31a9ff83b9f5ee7e83ca6d7948d700d8 not found: ID does not exist" containerID="2847df9bcd14d1409887a15fb491be9a31a9ff83b9f5ee7e83ca6d7948d700d8" Oct 30 17:25:32.802074 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:32.802041 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2847df9bcd14d1409887a15fb491be9a31a9ff83b9f5ee7e83ca6d7948d700d8"} err="failed to get container status \"2847df9bcd14d1409887a15fb491be9a31a9ff83b9f5ee7e83ca6d7948d700d8\": rpc error: code = NotFound desc = could not find container \"2847df9bcd14d1409887a15fb491be9a31a9ff83b9f5ee7e83ca6d7948d700d8\": container with ID starting with 2847df9bcd14d1409887a15fb491be9a31a9ff83b9f5ee7e83ca6d7948d700d8 not found: ID does not exist" Oct 30 17:25:32.830462 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:32.830431 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk"] Oct 30 17:25:32.833972 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:32.833952 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-4c6596-openshift-default-b886bc8ff-bp6mk"] Oct 30 17:25:32.933774 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:32.933724 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e934b3f5-e168-4760-9ea6-3b9cab799328" path="/var/lib/kubelet/pods/e934b3f5-e168-4760-9ea6-3b9cab799328/volumes" Oct 30 17:25:35.943380 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:35.943331 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wh8m9"] Oct 30 17:25:35.946034 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:35.945997 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wh8m9" podUID="a84d4da5-c4fd-4099-b994-4f49214532e5" containerName="istio-proxy" containerID="cri-o://d1b71f8998a095adf8e8b79217733bb316b32e8263f238d3ddfe18445654ced7" gracePeriod=30 Oct 30 17:25:37.068990 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:37.068951 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2d365c-skfpg"] Oct 30 17:25:37.080390 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:37.080363 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2d365c-skfpg"] Oct 30 17:25:38.998873 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:38.993561 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47bf14f7-1ba3-4416-a1ff-2bf301c2e98d" path="/var/lib/kubelet/pods/47bf14f7-1ba3-4416-a1ff-2bf301c2e98d/volumes" Oct 30 17:25:39.142330 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:39.142283 4493 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-wh8m9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:25:39.142523 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:39.142364 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wh8m9" podUID="a84d4da5-c4fd-4099-b994-4f49214532e5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:25:41.221610 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:41.221524 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wh8m9" Oct 30 17:25:41.302654 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:41.302624 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a84d4da5-c4fd-4099-b994-4f49214532e5-kube-api-access-m4w9r\" (UniqueName: \"kubernetes.io/projected/a84d4da5-c4fd-4099-b994-4f49214532e5-kube-api-access-m4w9r\") pod \"a84d4da5-c4fd-4099-b994-4f49214532e5\" (UID: \"a84d4da5-c4fd-4099-b994-4f49214532e5\") " Oct 30 17:25:41.302865 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:41.302695 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a84d4da5-c4fd-4099-b994-4f49214532e5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a84d4da5-c4fd-4099-b994-4f49214532e5-workload-socket\") pod \"a84d4da5-c4fd-4099-b994-4f49214532e5\" (UID: \"a84d4da5-c4fd-4099-b994-4f49214532e5\") " Oct 30 17:25:41.302865 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:41.302752 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a84d4da5-c4fd-4099-b994-4f49214532e5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a84d4da5-c4fd-4099-b994-4f49214532e5-credential-socket\") pod \"a84d4da5-c4fd-4099-b994-4f49214532e5\" (UID: \"a84d4da5-c4fd-4099-b994-4f49214532e5\") " Oct 30 17:25:41.302865 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:41.302835 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a84d4da5-c4fd-4099-b994-4f49214532e5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a84d4da5-c4fd-4099-b994-4f49214532e5-istio-podinfo\") pod \"a84d4da5-c4fd-4099-b994-4f49214532e5\" (UID: \"a84d4da5-c4fd-4099-b994-4f49214532e5\") " Oct 30 17:25:41.303056 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:41.302895 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a84d4da5-c4fd-4099-b994-4f49214532e5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a84d4da5-c4fd-4099-b994-4f49214532e5-istio-data\") pod \"a84d4da5-c4fd-4099-b994-4f49214532e5\" (UID: \"a84d4da5-c4fd-4099-b994-4f49214532e5\") " Oct 30 17:25:41.303056 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:41.302987 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a84d4da5-c4fd-4099-b994-4f49214532e5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a84d4da5-c4fd-4099-b994-4f49214532e5-workload-certs\") pod \"a84d4da5-c4fd-4099-b994-4f49214532e5\" (UID: \"a84d4da5-c4fd-4099-b994-4f49214532e5\") " Oct 30 17:25:41.303056 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:41.303031 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a84d4da5-c4fd-4099-b994-4f49214532e5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a84d4da5-c4fd-4099-b994-4f49214532e5-istiod-ca-cert\") pod \"a84d4da5-c4fd-4099-b994-4f49214532e5\" (UID: \"a84d4da5-c4fd-4099-b994-4f49214532e5\") " Oct 30 17:25:41.303227 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:41.303061 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a84d4da5-c4fd-4099-b994-4f49214532e5-istio-token\" (UniqueName: \"kubernetes.io/projected/a84d4da5-c4fd-4099-b994-4f49214532e5-istio-token\") pod \"a84d4da5-c4fd-4099-b994-4f49214532e5\" (UID: \"a84d4da5-c4fd-4099-b994-4f49214532e5\") " Oct 30 17:25:41.303227 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:41.303111 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a84d4da5-c4fd-4099-b994-4f49214532e5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a84d4da5-c4fd-4099-b994-4f49214532e5-istio-envoy\") pod \"a84d4da5-c4fd-4099-b994-4f49214532e5\" (UID: \"a84d4da5-c4fd-4099-b994-4f49214532e5\") " Oct 30 17:25:41.303227 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:41.303117 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a84d4da5-c4fd-4099-b994-4f49214532e5-workload-socket" pod "a84d4da5-c4fd-4099-b994-4f49214532e5" (UID: "a84d4da5-c4fd-4099-b994-4f49214532e5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:25:41.303227 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:41.303130 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a84d4da5-c4fd-4099-b994-4f49214532e5-credential-socket" pod "a84d4da5-c4fd-4099-b994-4f49214532e5" (UID: "a84d4da5-c4fd-4099-b994-4f49214532e5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:25:41.303456 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:41.303274 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a84d4da5-c4fd-4099-b994-4f49214532e5-istio-data" pod "a84d4da5-c4fd-4099-b994-4f49214532e5" (UID: "a84d4da5-c4fd-4099-b994-4f49214532e5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:25:41.303525 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:41.303504 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a84d4da5-c4fd-4099-b994-4f49214532e5-istiod-ca-cert" pod "a84d4da5-c4fd-4099-b994-4f49214532e5" (UID: "a84d4da5-c4fd-4099-b994-4f49214532e5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:25:41.303740 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:41.303716 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a84d4da5-c4fd-4099-b994-4f49214532e5-istio-data\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:25:41.303852 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:41.303745 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a84d4da5-c4fd-4099-b994-4f49214532e5-istiod-ca-cert\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:25:41.303852 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:41.303759 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a84d4da5-c4fd-4099-b994-4f49214532e5-workload-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:25:41.303852 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:41.303771 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a84d4da5-c4fd-4099-b994-4f49214532e5-credential-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:25:41.305936 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:41.305913 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a84d4da5-c4fd-4099-b994-4f49214532e5-workload-certs" pod "a84d4da5-c4fd-4099-b994-4f49214532e5" (UID: "a84d4da5-c4fd-4099-b994-4f49214532e5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:25:41.306078 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:41.306046 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a84d4da5-c4fd-4099-b994-4f49214532e5-istio-podinfo" pod "a84d4da5-c4fd-4099-b994-4f49214532e5" (UID: "a84d4da5-c4fd-4099-b994-4f49214532e5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:25:41.306238 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:41.306218 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a84d4da5-c4fd-4099-b994-4f49214532e5-istio-envoy" pod "a84d4da5-c4fd-4099-b994-4f49214532e5" (UID: "a84d4da5-c4fd-4099-b994-4f49214532e5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:25:41.306366 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:41.306350 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a84d4da5-c4fd-4099-b994-4f49214532e5-istio-token" pod "a84d4da5-c4fd-4099-b994-4f49214532e5" (UID: "a84d4da5-c4fd-4099-b994-4f49214532e5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:25:41.306493 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:41.306478 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a84d4da5-c4fd-4099-b994-4f49214532e5-kube-api-access-m4w9r" pod "a84d4da5-c4fd-4099-b994-4f49214532e5" (UID: "a84d4da5-c4fd-4099-b994-4f49214532e5"). InnerVolumeSpecName "kube-api-access-m4w9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:25:41.404772 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:41.404728 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a84d4da5-c4fd-4099-b994-4f49214532e5-istio-envoy\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:25:41.404772 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:41.404769 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4w9r\" (UniqueName: \"kubernetes.io/projected/a84d4da5-c4fd-4099-b994-4f49214532e5-kube-api-access-m4w9r\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:25:41.405094 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:41.404934 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a84d4da5-c4fd-4099-b994-4f49214532e5-istio-podinfo\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:25:41.405094 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:41.404953 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a84d4da5-c4fd-4099-b994-4f49214532e5-workload-certs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:25:41.405094 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:41.404963 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a84d4da5-c4fd-4099-b994-4f49214532e5-istio-token\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:25:41.858981 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:41.858948 4493 generic.go:356] "Generic (PLEG): container finished" podID="a84d4da5-c4fd-4099-b994-4f49214532e5" containerID="d1b71f8998a095adf8e8b79217733bb316b32e8263f238d3ddfe18445654ced7" exitCode=0 Oct 30 17:25:41.859195 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:41.858998 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wh8m9" Oct 30 17:25:41.859195 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:41.859037 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wh8m9" event={"ID":"a84d4da5-c4fd-4099-b994-4f49214532e5","Type":"ContainerDied","Data":"d1b71f8998a095adf8e8b79217733bb316b32e8263f238d3ddfe18445654ced7"} Oct 30 17:25:41.859195 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:41.859104 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wh8m9" event={"ID":"a84d4da5-c4fd-4099-b994-4f49214532e5","Type":"ContainerDied","Data":"294c2846775003aa4f61e9c4580f5d41e7dd6ba5cd7ede5603a2a869db7f3262"} Oct 30 17:25:41.859195 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:41.859133 4493 scope.go:122] "RemoveContainer" containerID="d1b71f8998a095adf8e8b79217733bb316b32e8263f238d3ddfe18445654ced7" Oct 30 17:25:41.878905 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:41.878884 4493 scope.go:122] "RemoveContainer" containerID="d1b71f8998a095adf8e8b79217733bb316b32e8263f238d3ddfe18445654ced7" Oct 30 17:25:41.879239 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:25:41.879217 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1b71f8998a095adf8e8b79217733bb316b32e8263f238d3ddfe18445654ced7\": container with ID starting with d1b71f8998a095adf8e8b79217733bb316b32e8263f238d3ddfe18445654ced7 not found: ID does not exist" containerID="d1b71f8998a095adf8e8b79217733bb316b32e8263f238d3ddfe18445654ced7" Oct 30 17:25:41.879311 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:41.879251 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1b71f8998a095adf8e8b79217733bb316b32e8263f238d3ddfe18445654ced7"} err="failed to get container status \"d1b71f8998a095adf8e8b79217733bb316b32e8263f238d3ddfe18445654ced7\": rpc error: code = NotFound desc = could not find container \"d1b71f8998a095adf8e8b79217733bb316b32e8263f238d3ddfe18445654ced7\": container with ID starting with d1b71f8998a095adf8e8b79217733bb316b32e8263f238d3ddfe18445654ced7 not found: ID does not exist" Oct 30 17:25:41.925763 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:41.925730 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wh8m9"] Oct 30 17:25:41.934430 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:41.934397 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-wh8m9"] Oct 30 17:25:42.936176 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:42.936140 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a84d4da5-c4fd-4099-b994-4f49214532e5" path="/var/lib/kubelet/pods/a84d4da5-c4fd-4099-b994-4f49214532e5/volumes" Oct 30 17:25:57.493635 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:57.493591 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 17:25:58.249810 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:58.249046 4493 scope.go:122] "RemoveContainer" containerID="975592ab9e7a3bed678359171ae0bcb5a527a05aefbdb1d0498f7ebf813c6644" Oct 30 17:25:58.312882 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:58.312852 4493 scope.go:122] "RemoveContainer" containerID="5057cd19efef10c441a82b5982bd901396945aa8a69fd76c9d434829b9d22a3e" Oct 30 17:25:58.766505 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:58.760855 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3106d6-j6rcg"] Oct 30 17:25:58.766505 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:58.765735 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-3106d6-j6rcg"] Oct 30 17:25:58.937772 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:58.937738 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d891422-969c-4fb0-b35b-4c4adeb61f41" path="/var/lib/kubelet/pods/2d891422-969c-4fb0-b35b-4c4adeb61f41/volumes" Oct 30 17:25:59.322812 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:59.317477 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5c6b3c-gh6cm"] Oct 30 17:25:59.356808 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:25:59.356367 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5c6b3c-gh6cm"] Oct 30 17:26:00.342999 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:26:00.342965 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 17:26:00.934363 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:00.934327 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80635646-db40-4d39-b3f4-e3a35108ebb3" path="/var/lib/kubelet/pods/80635646-db40-4d39-b3f4-e3a35108ebb3/volumes" Oct 30 17:26:03.375402 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:03.375369 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9bb5b1-rmwwt"] Oct 30 17:26:03.388194 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:03.388159 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-9bb5b1-rmwwt"] Oct 30 17:26:04.933266 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:04.933228 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="712bd7c8-327a-42a8-b0a4-2ef420590395" path="/var/lib/kubelet/pods/712bd7c8-327a-42a8-b0a4-2ef420590395/volumes" Oct 30 17:26:07.166628 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:07.166588 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5758b7-d9wls"] Oct 30 17:26:07.167201 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:07.167185 4493 cpu_manager.go:414] "containerMap: removing container" podUID="712bd7c8-327a-42a8-b0a4-2ef420590395" containerName="revoke-keys" Oct 30 17:26:07.167249 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:07.167203 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="712bd7c8-327a-42a8-b0a4-2ef420590395" containerName="revoke-keys" Oct 30 17:26:07.167249 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:07.167222 4493 cpu_manager.go:414] "containerMap: removing container" podUID="e934b3f5-e168-4760-9ea6-3b9cab799328" containerName="istio-proxy" Oct 30 17:26:07.167249 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:07.167227 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e934b3f5-e168-4760-9ea6-3b9cab799328" containerName="istio-proxy" Oct 30 17:26:07.167249 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:07.167240 4493 cpu_manager.go:414] "containerMap: removing container" podUID="18c895a8-b0d7-425c-81eb-282dd1766d59" containerName="istio-proxy" Oct 30 17:26:07.167249 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:07.167245 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18c895a8-b0d7-425c-81eb-282dd1766d59" containerName="istio-proxy" Oct 30 17:26:07.167472 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:07.167256 4493 cpu_manager.go:414] "containerMap: removing container" podUID="2d891422-969c-4fb0-b35b-4c4adeb61f41" containerName="revoke-keys" Oct 30 17:26:07.167472 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:07.167261 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d891422-969c-4fb0-b35b-4c4adeb61f41" containerName="revoke-keys" Oct 30 17:26:07.167472 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:07.167285 4493 cpu_manager.go:414] "containerMap: removing container" podUID="80635646-db40-4d39-b3f4-e3a35108ebb3" containerName="revoke-keys" Oct 30 17:26:07.167472 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:07.167290 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80635646-db40-4d39-b3f4-e3a35108ebb3" containerName="revoke-keys" Oct 30 17:26:07.167472 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:07.167300 4493 cpu_manager.go:414] "containerMap: removing container" podUID="18c895a8-b0d7-425c-81eb-282dd1766d59" containerName="istio-proxy" Oct 30 17:26:07.167472 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:07.167303 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18c895a8-b0d7-425c-81eb-282dd1766d59" containerName="istio-proxy" Oct 30 17:26:07.167472 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:07.167312 4493 cpu_manager.go:414] "containerMap: removing container" podUID="a84d4da5-c4fd-4099-b994-4f49214532e5" containerName="istio-proxy" Oct 30 17:26:07.167472 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:07.167316 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a84d4da5-c4fd-4099-b994-4f49214532e5" containerName="istio-proxy" Oct 30 17:26:07.167472 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:07.167395 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="80635646-db40-4d39-b3f4-e3a35108ebb3" containerName="revoke-keys" Oct 30 17:26:07.167472 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:07.167404 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="18c895a8-b0d7-425c-81eb-282dd1766d59" containerName="istio-proxy" Oct 30 17:26:07.167472 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:07.167412 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="a84d4da5-c4fd-4099-b994-4f49214532e5" containerName="istio-proxy" Oct 30 17:26:07.167472 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:07.167420 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="712bd7c8-327a-42a8-b0a4-2ef420590395" containerName="revoke-keys" Oct 30 17:26:07.167472 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:07.167427 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="e934b3f5-e168-4760-9ea6-3b9cab799328" containerName="istio-proxy" Oct 30 17:26:07.167472 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:07.167432 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="e934b3f5-e168-4760-9ea6-3b9cab799328" containerName="istio-proxy" Oct 30 17:26:07.167472 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:07.167437 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="18c895a8-b0d7-425c-81eb-282dd1766d59" containerName="istio-proxy" Oct 30 17:26:07.167472 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:07.167443 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d891422-969c-4fb0-b35b-4c4adeb61f41" containerName="revoke-keys" Oct 30 17:26:07.176362 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:07.176341 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5758b7-d9wls" Oct 30 17:26:07.192535 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:07.192511 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5758b7-d9wls"] Oct 30 17:26:07.199095 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:07.199071 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:26:07.199215 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:07.199138 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:26:07.254731 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:07.254686 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 17:26:07.255065 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:07.255051 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:26:07.259183 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:07.259142 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f2f8c77-10d4-4a4f-97bf-6a1c50a4702f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-5758b7-d9wls\" (UID: \"4f2f8c77-10d4-4a4f-97bf-6a1c50a4702f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5758b7-d9wls" Oct 30 17:26:07.259482 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:07.259461 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqgrk\" (UniqueName: \"kubernetes.io/projected/4f2f8c77-10d4-4a4f-97bf-6a1c50a4702f-kube-api-access-vqgrk\") pod \"maas-api-revoke-keys-e2e-extmod-b-5758b7-d9wls\" (UID: \"4f2f8c77-10d4-4a4f-97bf-6a1c50a4702f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5758b7-d9wls" Oct 30 17:26:07.259664 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:07.259633 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f2f8c77-10d4-4a4f-97bf-6a1c50a4702f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-5758b7-d9wls\" (UID: \"4f2f8c77-10d4-4a4f-97bf-6a1c50a4702f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5758b7-d9wls" Oct 30 17:26:07.299719 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:07.297441 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w"] Oct 30 17:26:07.299719 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:07.297664 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w" podUID="8e21f846-6c01-496f-998e-ccfe2a18d5cd" containerName="istio-proxy" containerID="cri-o://b00a36923850817ca34a07fff2dc81de8bbd8b84e0459f629104b19a2bce9295" gracePeriod=30 Oct 30 17:26:07.372862 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:07.362700 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqgrk\" (UniqueName: \"kubernetes.io/projected/4f2f8c77-10d4-4a4f-97bf-6a1c50a4702f-kube-api-access-vqgrk\") pod \"maas-api-revoke-keys-e2e-extmod-b-5758b7-d9wls\" (UID: \"4f2f8c77-10d4-4a4f-97bf-6a1c50a4702f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5758b7-d9wls" Oct 30 17:26:07.372862 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:07.362762 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f2f8c77-10d4-4a4f-97bf-6a1c50a4702f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-5758b7-d9wls\" (UID: \"4f2f8c77-10d4-4a4f-97bf-6a1c50a4702f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5758b7-d9wls" Oct 30 17:26:07.372862 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:07.363017 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f2f8c77-10d4-4a4f-97bf-6a1c50a4702f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-5758b7-d9wls\" (UID: \"4f2f8c77-10d4-4a4f-97bf-6a1c50a4702f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5758b7-d9wls" Oct 30 17:26:07.375763 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:07.375507 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f2f8c77-10d4-4a4f-97bf-6a1c50a4702f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-5758b7-d9wls\" (UID: \"4f2f8c77-10d4-4a4f-97bf-6a1c50a4702f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5758b7-d9wls" Oct 30 17:26:07.417857 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:07.417614 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f2f8c77-10d4-4a4f-97bf-6a1c50a4702f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-5758b7-d9wls\" (UID: \"4f2f8c77-10d4-4a4f-97bf-6a1c50a4702f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5758b7-d9wls" Oct 30 17:26:07.455815 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:07.455486 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqgrk\" (UniqueName: \"kubernetes.io/projected/4f2f8c77-10d4-4a4f-97bf-6a1c50a4702f-kube-api-access-vqgrk\") pod \"maas-api-revoke-keys-e2e-extmod-b-5758b7-d9wls\" (UID: \"4f2f8c77-10d4-4a4f-97bf-6a1c50a4702f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5758b7-d9wls" Oct 30 17:26:07.495283 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:07.495237 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5758b7-d9wls" Oct 30 17:26:07.819983 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:07.819950 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5758b7-d9wls"] Oct 30 17:26:07.821160 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:26:07.821116 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f2f8c77_10d4_4a4f_97bf_6a1c50a4702f.slice/crio-2abb2794ad3ecfe2957f6ffd65aa374868173f49e4bbe2ae5ba5bebbd33ea52f WatchSource:0}: Error finding container 2abb2794ad3ecfe2957f6ffd65aa374868173f49e4bbe2ae5ba5bebbd33ea52f: Status 404 returned error can't find the container with id 2abb2794ad3ecfe2957f6ffd65aa374868173f49e4bbe2ae5ba5bebbd33ea52f Oct 30 17:26:07.855613 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:07.855545 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cb6251-mmbcd"] Oct 30 17:26:07.856200 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:07.856185 4493 cpu_manager.go:414] "containerMap: removing container" podUID="e934b3f5-e168-4760-9ea6-3b9cab799328" containerName="istio-proxy" Oct 30 17:26:07.856278 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:07.856201 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e934b3f5-e168-4760-9ea6-3b9cab799328" containerName="istio-proxy" Oct 30 17:26:07.881807 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:07.872093 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cb6251-mmbcd" Oct 30 17:26:07.885838 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:07.884929 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cb6251-mmbcd"] Oct 30 17:26:07.974703 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:07.974666 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96228ff4-3b01-46dd-93da-15465c07776c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-cb6251-mmbcd\" (UID: \"96228ff4-3b01-46dd-93da-15465c07776c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cb6251-mmbcd" Oct 30 17:26:07.974934 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:07.974727 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96228ff4-3b01-46dd-93da-15465c07776c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-cb6251-mmbcd\" (UID: \"96228ff4-3b01-46dd-93da-15465c07776c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cb6251-mmbcd" Oct 30 17:26:07.974934 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:07.974892 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tp4f\" (UniqueName: \"kubernetes.io/projected/96228ff4-3b01-46dd-93da-15465c07776c-kube-api-access-7tp4f\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-cb6251-mmbcd\" (UID: \"96228ff4-3b01-46dd-93da-15465c07776c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cb6251-mmbcd" Oct 30 17:26:08.076399 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:08.076308 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96228ff4-3b01-46dd-93da-15465c07776c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-cb6251-mmbcd\" (UID: \"96228ff4-3b01-46dd-93da-15465c07776c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cb6251-mmbcd" Oct 30 17:26:08.076577 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:08.076408 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tp4f\" (UniqueName: \"kubernetes.io/projected/96228ff4-3b01-46dd-93da-15465c07776c-kube-api-access-7tp4f\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-cb6251-mmbcd\" (UID: \"96228ff4-3b01-46dd-93da-15465c07776c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cb6251-mmbcd" Oct 30 17:26:08.076626 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:08.076576 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96228ff4-3b01-46dd-93da-15465c07776c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-cb6251-mmbcd\" (UID: \"96228ff4-3b01-46dd-93da-15465c07776c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cb6251-mmbcd" Oct 30 17:26:08.077081 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:08.077062 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96228ff4-3b01-46dd-93da-15465c07776c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-cb6251-mmbcd\" (UID: \"96228ff4-3b01-46dd-93da-15465c07776c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cb6251-mmbcd" Oct 30 17:26:08.079182 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:08.079162 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96228ff4-3b01-46dd-93da-15465c07776c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-cb6251-mmbcd\" (UID: \"96228ff4-3b01-46dd-93da-15465c07776c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cb6251-mmbcd" Oct 30 17:26:08.093490 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:08.093452 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5758b7-d9wls" event={"ID":"4f2f8c77-10d4-4a4f-97bf-6a1c50a4702f","Type":"ContainerStarted","Data":"2abb2794ad3ecfe2957f6ffd65aa374868173f49e4bbe2ae5ba5bebbd33ea52f"} Oct 30 17:26:08.099378 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:08.099331 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tp4f\" (UniqueName: \"kubernetes.io/projected/96228ff4-3b01-46dd-93da-15465c07776c-kube-api-access-7tp4f\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-cb6251-mmbcd\" (UID: \"96228ff4-3b01-46dd-93da-15465c07776c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cb6251-mmbcd" Oct 30 17:26:08.198084 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:08.198052 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cb6251-mmbcd" Oct 30 17:26:08.507830 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:08.507797 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cb6251-mmbcd"] Oct 30 17:26:08.509754 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:26:08.509712 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96228ff4_3b01_46dd_93da_15465c07776c.slice/crio-356b98851190dbaa8e8bff41d5c120f61863bf3c24ea68cf4356381a1c36a29b WatchSource:0}: Error finding container 356b98851190dbaa8e8bff41d5c120f61863bf3c24ea68cf4356381a1c36a29b: Status 404 returned error can't find the container with id 356b98851190dbaa8e8bff41d5c120f61863bf3c24ea68cf4356381a1c36a29b Oct 30 17:26:09.102225 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:09.102131 4493 generic.go:356] "Generic (PLEG): container finished" podID="4f2f8c77-10d4-4a4f-97bf-6a1c50a4702f" containerID="93a1b240fba937004c489bea62931eaa802503835766745014ab7cdf34dcec39" exitCode=0 Oct 30 17:26:09.102403 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:09.102221 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5758b7-d9wls" event={"ID":"4f2f8c77-10d4-4a4f-97bf-6a1c50a4702f","Type":"ContainerDied","Data":"93a1b240fba937004c489bea62931eaa802503835766745014ab7cdf34dcec39"} Oct 30 17:26:09.104085 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:09.104059 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cb6251-mmbcd" event={"ID":"96228ff4-3b01-46dd-93da-15465c07776c","Type":"ContainerStarted","Data":"444231345d0308885a0401a12cc70c6bcec14d556060ac16ef66ebab7745a4d9"} Oct 30 17:26:09.104218 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:09.104109 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cb6251-mmbcd" event={"ID":"96228ff4-3b01-46dd-93da-15465c07776c","Type":"ContainerStarted","Data":"356b98851190dbaa8e8bff41d5c120f61863bf3c24ea68cf4356381a1c36a29b"} Oct 30 17:26:09.183181 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:09.183129 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cb6251-mmbcd" podStartSLOduration=2.183112557 podStartE2EDuration="2.183112557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:26:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:26:09.182468535 +0000 UTC m=+3796.409643631" watchObservedRunningTime="2026-10-30 17:26:09.183112557 +0000 UTC m=+3796.410287655" Oct 30 17:26:09.370036 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:09.369947 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d93a3edf-8s8ws"] Oct 30 17:26:09.373999 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:09.373979 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d93a3edf-8s8ws" Oct 30 17:26:09.386538 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:09.386510 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d93a3edf-8s8ws"] Oct 30 17:26:09.498627 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:09.498594 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50e6c831-46ad-4a5f-adea-4271dcfd6eb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d93a3edf-8s8ws\" (UID: \"50e6c831-46ad-4a5f-adea-4271dcfd6eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d93a3edf-8s8ws" Oct 30 17:26:09.498627 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:09.498625 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbkcv\" (UniqueName: \"kubernetes.io/projected/50e6c831-46ad-4a5f-adea-4271dcfd6eb2-kube-api-access-hbkcv\") pod \"maas-api-revoke-sub-d93a3edf-8s8ws\" (UID: \"50e6c831-46ad-4a5f-adea-4271dcfd6eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d93a3edf-8s8ws" Oct 30 17:26:09.498863 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:09.498683 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50e6c831-46ad-4a5f-adea-4271dcfd6eb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d93a3edf-8s8ws\" (UID: \"50e6c831-46ad-4a5f-adea-4271dcfd6eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d93a3edf-8s8ws" Oct 30 17:26:09.600602 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:09.600570 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50e6c831-46ad-4a5f-adea-4271dcfd6eb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d93a3edf-8s8ws\" (UID: \"50e6c831-46ad-4a5f-adea-4271dcfd6eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d93a3edf-8s8ws" Oct 30 17:26:09.600822 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:09.600771 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50e6c831-46ad-4a5f-adea-4271dcfd6eb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d93a3edf-8s8ws\" (UID: \"50e6c831-46ad-4a5f-adea-4271dcfd6eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d93a3edf-8s8ws" Oct 30 17:26:09.600925 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:09.600837 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbkcv\" (UniqueName: \"kubernetes.io/projected/50e6c831-46ad-4a5f-adea-4271dcfd6eb2-kube-api-access-hbkcv\") pod \"maas-api-revoke-sub-d93a3edf-8s8ws\" (UID: \"50e6c831-46ad-4a5f-adea-4271dcfd6eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d93a3edf-8s8ws" Oct 30 17:26:09.601465 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:09.601403 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50e6c831-46ad-4a5f-adea-4271dcfd6eb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d93a3edf-8s8ws\" (UID: \"50e6c831-46ad-4a5f-adea-4271dcfd6eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d93a3edf-8s8ws" Oct 30 17:26:09.603061 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:09.603040 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50e6c831-46ad-4a5f-adea-4271dcfd6eb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d93a3edf-8s8ws\" (UID: \"50e6c831-46ad-4a5f-adea-4271dcfd6eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d93a3edf-8s8ws" Oct 30 17:26:09.624167 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:09.624056 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbkcv\" (UniqueName: \"kubernetes.io/projected/50e6c831-46ad-4a5f-adea-4271dcfd6eb2-kube-api-access-hbkcv\") pod \"maas-api-revoke-sub-d93a3edf-8s8ws\" (UID: \"50e6c831-46ad-4a5f-adea-4271dcfd6eb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d93a3edf-8s8ws" Oct 30 17:26:09.689931 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:09.689899 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d93a3edf-8s8ws" Oct 30 17:26:10.001992 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:10.001962 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d93a3edf-8s8ws"] Oct 30 17:26:10.002659 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:26:10.002632 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50e6c831_46ad_4a5f_adea_4271dcfd6eb2.slice/crio-3bcd98ead49b8ca01b0881e9853a2211200ea85d9d2c13537019afec1c25e9ed WatchSource:0}: Error finding container 3bcd98ead49b8ca01b0881e9853a2211200ea85d9d2c13537019afec1c25e9ed: Status 404 returned error can't find the container with id 3bcd98ead49b8ca01b0881e9853a2211200ea85d9d2c13537019afec1c25e9ed Oct 30 17:26:10.116702 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:10.116663 4493 generic.go:356] "Generic (PLEG): container finished" podID="96228ff4-3b01-46dd-93da-15465c07776c" containerID="444231345d0308885a0401a12cc70c6bcec14d556060ac16ef66ebab7745a4d9" exitCode=0 Oct 30 17:26:10.116918 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:10.116742 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cb6251-mmbcd" event={"ID":"96228ff4-3b01-46dd-93da-15465c07776c","Type":"ContainerDied","Data":"444231345d0308885a0401a12cc70c6bcec14d556060ac16ef66ebab7745a4d9"} Oct 30 17:26:10.118579 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:10.118550 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d93a3edf-8s8ws" event={"ID":"50e6c831-46ad-4a5f-adea-4271dcfd6eb2","Type":"ContainerStarted","Data":"3bcd98ead49b8ca01b0881e9853a2211200ea85d9d2c13537019afec1c25e9ed"} Oct 30 17:26:10.271720 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:10.271692 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5758b7-d9wls" Oct 30 17:26:10.310990 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:10.310954 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f2f8c77-10d4-4a4f-97bf-6a1c50a4702f-kube-api-access-vqgrk\" (UniqueName: \"kubernetes.io/projected/4f2f8c77-10d4-4a4f-97bf-6a1c50a4702f-kube-api-access-vqgrk\") pod \"4f2f8c77-10d4-4a4f-97bf-6a1c50a4702f\" (UID: \"4f2f8c77-10d4-4a4f-97bf-6a1c50a4702f\") " Oct 30 17:26:10.311890 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:10.311204 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f2f8c77-10d4-4a4f-97bf-6a1c50a4702f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f2f8c77-10d4-4a4f-97bf-6a1c50a4702f-maas-api-xks-ca\") pod \"4f2f8c77-10d4-4a4f-97bf-6a1c50a4702f\" (UID: \"4f2f8c77-10d4-4a4f-97bf-6a1c50a4702f\") " Oct 30 17:26:10.311890 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:10.311342 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f2f8c77-10d4-4a4f-97bf-6a1c50a4702f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f2f8c77-10d4-4a4f-97bf-6a1c50a4702f-maas-api-openshift-ca\") pod \"4f2f8c77-10d4-4a4f-97bf-6a1c50a4702f\" (UID: \"4f2f8c77-10d4-4a4f-97bf-6a1c50a4702f\") " Oct 30 17:26:10.313561 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:10.313534 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f2f8c77-10d4-4a4f-97bf-6a1c50a4702f-kube-api-access-vqgrk" pod "4f2f8c77-10d4-4a4f-97bf-6a1c50a4702f" (UID: "4f2f8c77-10d4-4a4f-97bf-6a1c50a4702f"). InnerVolumeSpecName "kube-api-access-vqgrk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:26:10.313844 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:10.313821 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f2f8c77-10d4-4a4f-97bf-6a1c50a4702f-maas-api-xks-ca" pod "4f2f8c77-10d4-4a4f-97bf-6a1c50a4702f" (UID: "4f2f8c77-10d4-4a4f-97bf-6a1c50a4702f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:26:10.314169 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:10.314140 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f2f8c77-10d4-4a4f-97bf-6a1c50a4702f-maas-api-openshift-ca" pod "4f2f8c77-10d4-4a4f-97bf-6a1c50a4702f" (UID: "4f2f8c77-10d4-4a4f-97bf-6a1c50a4702f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:26:10.314734 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:10.314697 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f2f8c77-10d4-4a4f-97bf-6a1c50a4702f-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:26:10.314734 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:10.314725 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f2f8c77-10d4-4a4f-97bf-6a1c50a4702f-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:26:10.314988 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:10.314739 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqgrk\" (UniqueName: \"kubernetes.io/projected/4f2f8c77-10d4-4a4f-97bf-6a1c50a4702f-kube-api-access-vqgrk\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:26:10.887731 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:10.887649 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw"] Oct 30 17:26:10.888239 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:10.887971 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw" podUID="e7475582-02dd-456b-a5d5-d894755aba74" containerName="istio-proxy" containerID="cri-o://0f134a4f15ef6855db28d0d167317cb30d32595258bd3c610b5eb8266b17d493" gracePeriod=30 Oct 30 17:26:11.130309 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:11.129695 4493 generic.go:356] "Generic (PLEG): container finished" podID="50e6c831-46ad-4a5f-adea-4271dcfd6eb2" containerID="baf53902262949138e0401cf1fe0c959bf8f2a3af1f5df2325382c4c7049fb7a" exitCode=0 Oct 30 17:26:11.130309 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:11.129900 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d93a3edf-8s8ws" event={"ID":"50e6c831-46ad-4a5f-adea-4271dcfd6eb2","Type":"ContainerDied","Data":"baf53902262949138e0401cf1fe0c959bf8f2a3af1f5df2325382c4c7049fb7a"} Oct 30 17:26:11.136826 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:11.133834 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5758b7-d9wls" Oct 30 17:26:11.136826 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:11.134429 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5758b7-d9wls" event={"ID":"4f2f8c77-10d4-4a4f-97bf-6a1c50a4702f","Type":"ContainerDied","Data":"2abb2794ad3ecfe2957f6ffd65aa374868173f49e4bbe2ae5ba5bebbd33ea52f"} Oct 30 17:26:11.136826 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:11.134471 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2abb2794ad3ecfe2957f6ffd65aa374868173f49e4bbe2ae5ba5bebbd33ea52f" Oct 30 17:26:11.306885 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:11.306854 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cb6251-mmbcd" Oct 30 17:26:11.429489 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:11.429451 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96228ff4-3b01-46dd-93da-15465c07776c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96228ff4-3b01-46dd-93da-15465c07776c-maas-api-xks-ca\") pod \"96228ff4-3b01-46dd-93da-15465c07776c\" (UID: \"96228ff4-3b01-46dd-93da-15465c07776c\") " Oct 30 17:26:11.429697 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:11.429573 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96228ff4-3b01-46dd-93da-15465c07776c-kube-api-access-7tp4f\" (UniqueName: \"kubernetes.io/projected/96228ff4-3b01-46dd-93da-15465c07776c-kube-api-access-7tp4f\") pod \"96228ff4-3b01-46dd-93da-15465c07776c\" (UID: \"96228ff4-3b01-46dd-93da-15465c07776c\") " Oct 30 17:26:11.429765 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:11.429726 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96228ff4-3b01-46dd-93da-15465c07776c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96228ff4-3b01-46dd-93da-15465c07776c-maas-api-openshift-ca\") pod \"96228ff4-3b01-46dd-93da-15465c07776c\" (UID: \"96228ff4-3b01-46dd-93da-15465c07776c\") " Oct 30 17:26:11.430093 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:11.430067 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96228ff4-3b01-46dd-93da-15465c07776c-maas-api-openshift-ca" pod "96228ff4-3b01-46dd-93da-15465c07776c" (UID: "96228ff4-3b01-46dd-93da-15465c07776c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:26:11.430239 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:11.430222 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96228ff4-3b01-46dd-93da-15465c07776c-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:26:11.431956 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:11.431934 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96228ff4-3b01-46dd-93da-15465c07776c-kube-api-access-7tp4f" pod "96228ff4-3b01-46dd-93da-15465c07776c" (UID: "96228ff4-3b01-46dd-93da-15465c07776c"). InnerVolumeSpecName "kube-api-access-7tp4f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:26:11.432196 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:11.432181 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96228ff4-3b01-46dd-93da-15465c07776c-maas-api-xks-ca" pod "96228ff4-3b01-46dd-93da-15465c07776c" (UID: "96228ff4-3b01-46dd-93da-15465c07776c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:26:11.489529 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:11.489456 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-43197b-c5g9z"] Oct 30 17:26:11.490092 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:11.490074 4493 cpu_manager.go:414] "containerMap: removing container" podUID="96228ff4-3b01-46dd-93da-15465c07776c" containerName="revoke-keys" Oct 30 17:26:11.490166 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:11.490093 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96228ff4-3b01-46dd-93da-15465c07776c" containerName="revoke-keys" Oct 30 17:26:11.490166 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:11.490110 4493 cpu_manager.go:414] "containerMap: removing container" podUID="4f2f8c77-10d4-4a4f-97bf-6a1c50a4702f" containerName="revoke-keys" Oct 30 17:26:11.490166 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:11.490116 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f2f8c77-10d4-4a4f-97bf-6a1c50a4702f" containerName="revoke-keys" Oct 30 17:26:11.490275 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:11.490230 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="96228ff4-3b01-46dd-93da-15465c07776c" containerName="revoke-keys" Oct 30 17:26:11.490275 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:11.490239 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f2f8c77-10d4-4a4f-97bf-6a1c50a4702f" containerName="revoke-keys" Oct 30 17:26:11.493653 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:11.493637 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-43197b-c5g9z" Oct 30 17:26:11.517513 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:11.517482 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-43197b-c5g9z"] Oct 30 17:26:11.531341 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:11.531306 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp7t9\" (UniqueName: \"kubernetes.io/projected/4fba1cad-d578-4e53-afcb-3698d267a658-kube-api-access-hp7t9\") pod \"maas-api-revoke-keys-e2e-extmod-b-43197b-c5g9z\" (UID: \"4fba1cad-d578-4e53-afcb-3698d267a658\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-43197b-c5g9z" Oct 30 17:26:11.531539 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:11.531388 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fba1cad-d578-4e53-afcb-3698d267a658-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-43197b-c5g9z\" (UID: \"4fba1cad-d578-4e53-afcb-3698d267a658\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-43197b-c5g9z" Oct 30 17:26:11.531539 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:11.531440 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fba1cad-d578-4e53-afcb-3698d267a658-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-43197b-c5g9z\" (UID: \"4fba1cad-d578-4e53-afcb-3698d267a658\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-43197b-c5g9z" Oct 30 17:26:11.531670 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:11.531559 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tp4f\" (UniqueName: \"kubernetes.io/projected/96228ff4-3b01-46dd-93da-15465c07776c-kube-api-access-7tp4f\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:26:11.531670 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:11.531579 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96228ff4-3b01-46dd-93da-15465c07776c-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:26:11.632859 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:11.632823 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp7t9\" (UniqueName: \"kubernetes.io/projected/4fba1cad-d578-4e53-afcb-3698d267a658-kube-api-access-hp7t9\") pod \"maas-api-revoke-keys-e2e-extmod-b-43197b-c5g9z\" (UID: \"4fba1cad-d578-4e53-afcb-3698d267a658\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-43197b-c5g9z" Oct 30 17:26:11.633032 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:11.632884 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fba1cad-d578-4e53-afcb-3698d267a658-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-43197b-c5g9z\" (UID: \"4fba1cad-d578-4e53-afcb-3698d267a658\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-43197b-c5g9z" Oct 30 17:26:11.633032 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:11.632928 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fba1cad-d578-4e53-afcb-3698d267a658-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-43197b-c5g9z\" (UID: \"4fba1cad-d578-4e53-afcb-3698d267a658\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-43197b-c5g9z" Oct 30 17:26:11.633667 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:11.633642 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fba1cad-d578-4e53-afcb-3698d267a658-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-43197b-c5g9z\" (UID: \"4fba1cad-d578-4e53-afcb-3698d267a658\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-43197b-c5g9z" Oct 30 17:26:11.635746 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:11.635718 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fba1cad-d578-4e53-afcb-3698d267a658-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-43197b-c5g9z\" (UID: \"4fba1cad-d578-4e53-afcb-3698d267a658\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-43197b-c5g9z" Oct 30 17:26:11.655617 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:11.655595 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp7t9\" (UniqueName: \"kubernetes.io/projected/4fba1cad-d578-4e53-afcb-3698d267a658-kube-api-access-hp7t9\") pod \"maas-api-revoke-keys-e2e-extmod-b-43197b-c5g9z\" (UID: \"4fba1cad-d578-4e53-afcb-3698d267a658\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-43197b-c5g9z" Oct 30 17:26:11.806688 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:11.806606 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-43197b-c5g9z" Oct 30 17:26:12.123966 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:12.123939 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-43197b-c5g9z"] Oct 30 17:26:12.124848 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:26:12.124819 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fba1cad_d578_4e53_afcb_3698d267a658.slice/crio-dc4ce7721cec7011993b51e6a5ff9ed7e9ea96488f30bbf6dd11a2a323cb09d1 WatchSource:0}: Error finding container dc4ce7721cec7011993b51e6a5ff9ed7e9ea96488f30bbf6dd11a2a323cb09d1: Status 404 returned error can't find the container with id dc4ce7721cec7011993b51e6a5ff9ed7e9ea96488f30bbf6dd11a2a323cb09d1 Oct 30 17:26:12.144983 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:12.144956 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cb6251-mmbcd" event={"ID":"96228ff4-3b01-46dd-93da-15465c07776c","Type":"ContainerDied","Data":"356b98851190dbaa8e8bff41d5c120f61863bf3c24ea68cf4356381a1c36a29b"} Oct 30 17:26:12.145267 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:12.145008 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="356b98851190dbaa8e8bff41d5c120f61863bf3c24ea68cf4356381a1c36a29b" Oct 30 17:26:12.145267 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:12.145066 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cb6251-mmbcd" Oct 30 17:26:12.150940 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:12.150916 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-43197b-c5g9z" event={"ID":"4fba1cad-d578-4e53-afcb-3698d267a658","Type":"ContainerStarted","Data":"dc4ce7721cec7011993b51e6a5ff9ed7e9ea96488f30bbf6dd11a2a323cb09d1"} Oct 30 17:26:12.328923 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:12.328899 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d93a3edf-8s8ws" Oct 30 17:26:12.443919 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:12.443887 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50e6c831-46ad-4a5f-adea-4271dcfd6eb2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50e6c831-46ad-4a5f-adea-4271dcfd6eb2-maas-api-xks-ca\") pod \"50e6c831-46ad-4a5f-adea-4271dcfd6eb2\" (UID: \"50e6c831-46ad-4a5f-adea-4271dcfd6eb2\") " Oct 30 17:26:12.444074 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:12.443938 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50e6c831-46ad-4a5f-adea-4271dcfd6eb2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50e6c831-46ad-4a5f-adea-4271dcfd6eb2-maas-api-openshift-ca\") pod \"50e6c831-46ad-4a5f-adea-4271dcfd6eb2\" (UID: \"50e6c831-46ad-4a5f-adea-4271dcfd6eb2\") " Oct 30 17:26:12.444074 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:12.444023 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50e6c831-46ad-4a5f-adea-4271dcfd6eb2-kube-api-access-hbkcv\" (UniqueName: \"kubernetes.io/projected/50e6c831-46ad-4a5f-adea-4271dcfd6eb2-kube-api-access-hbkcv\") pod \"50e6c831-46ad-4a5f-adea-4271dcfd6eb2\" (UID: \"50e6c831-46ad-4a5f-adea-4271dcfd6eb2\") " Oct 30 17:26:12.444595 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:12.444564 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50e6c831-46ad-4a5f-adea-4271dcfd6eb2-maas-api-openshift-ca" pod "50e6c831-46ad-4a5f-adea-4271dcfd6eb2" (UID: "50e6c831-46ad-4a5f-adea-4271dcfd6eb2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:26:12.446711 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:12.446560 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50e6c831-46ad-4a5f-adea-4271dcfd6eb2-maas-api-xks-ca" pod "50e6c831-46ad-4a5f-adea-4271dcfd6eb2" (UID: "50e6c831-46ad-4a5f-adea-4271dcfd6eb2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:26:12.446884 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:12.446862 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50e6c831-46ad-4a5f-adea-4271dcfd6eb2-kube-api-access-hbkcv" pod "50e6c831-46ad-4a5f-adea-4271dcfd6eb2" (UID: "50e6c831-46ad-4a5f-adea-4271dcfd6eb2"). InnerVolumeSpecName "kube-api-access-hbkcv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:26:12.545657 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:12.545626 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50e6c831-46ad-4a5f-adea-4271dcfd6eb2-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:26:12.545657 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:12.545654 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50e6c831-46ad-4a5f-adea-4271dcfd6eb2-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:26:12.545986 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:12.545665 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbkcv\" (UniqueName: \"kubernetes.io/projected/50e6c831-46ad-4a5f-adea-4271dcfd6eb2-kube-api-access-hbkcv\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:26:12.599209 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:12.599186 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w" Oct 30 17:26:12.646326 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:12.646294 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e21f846-6c01-496f-998e-ccfe2a18d5cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e21f846-6c01-496f-998e-ccfe2a18d5cd-workload-certs\") pod \"8e21f846-6c01-496f-998e-ccfe2a18d5cd\" (UID: \"8e21f846-6c01-496f-998e-ccfe2a18d5cd\") " Oct 30 17:26:12.646487 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:12.646346 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e21f846-6c01-496f-998e-ccfe2a18d5cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e21f846-6c01-496f-998e-ccfe2a18d5cd-istiod-ca-cert\") pod \"8e21f846-6c01-496f-998e-ccfe2a18d5cd\" (UID: \"8e21f846-6c01-496f-998e-ccfe2a18d5cd\") " Oct 30 17:26:12.646487 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:12.646452 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e21f846-6c01-496f-998e-ccfe2a18d5cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e21f846-6c01-496f-998e-ccfe2a18d5cd-istio-podinfo\") pod \"8e21f846-6c01-496f-998e-ccfe2a18d5cd\" (UID: \"8e21f846-6c01-496f-998e-ccfe2a18d5cd\") " Oct 30 17:26:12.646622 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:12.646490 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e21f846-6c01-496f-998e-ccfe2a18d5cd-kube-api-access-qz7cb\" (UniqueName: \"kubernetes.io/projected/8e21f846-6c01-496f-998e-ccfe2a18d5cd-kube-api-access-qz7cb\") pod \"8e21f846-6c01-496f-998e-ccfe2a18d5cd\" (UID: \"8e21f846-6c01-496f-998e-ccfe2a18d5cd\") " Oct 30 17:26:12.646622 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:12.646551 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e21f846-6c01-496f-998e-ccfe2a18d5cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e21f846-6c01-496f-998e-ccfe2a18d5cd-workload-socket\") pod \"8e21f846-6c01-496f-998e-ccfe2a18d5cd\" (UID: \"8e21f846-6c01-496f-998e-ccfe2a18d5cd\") " Oct 30 17:26:12.647121 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:12.646631 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e21f846-6c01-496f-998e-ccfe2a18d5cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e21f846-6c01-496f-998e-ccfe2a18d5cd-credential-socket\") pod \"8e21f846-6c01-496f-998e-ccfe2a18d5cd\" (UID: \"8e21f846-6c01-496f-998e-ccfe2a18d5cd\") " Oct 30 17:26:12.647121 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:12.646663 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e21f846-6c01-496f-998e-ccfe2a18d5cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e21f846-6c01-496f-998e-ccfe2a18d5cd-istio-data\") pod \"8e21f846-6c01-496f-998e-ccfe2a18d5cd\" (UID: \"8e21f846-6c01-496f-998e-ccfe2a18d5cd\") " Oct 30 17:26:12.647121 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:12.646691 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e21f846-6c01-496f-998e-ccfe2a18d5cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e21f846-6c01-496f-998e-ccfe2a18d5cd-istio-envoy\") pod \"8e21f846-6c01-496f-998e-ccfe2a18d5cd\" (UID: \"8e21f846-6c01-496f-998e-ccfe2a18d5cd\") " Oct 30 17:26:12.647121 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:12.646748 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e21f846-6c01-496f-998e-ccfe2a18d5cd-istio-token\" (UniqueName: \"kubernetes.io/projected/8e21f846-6c01-496f-998e-ccfe2a18d5cd-istio-token\") pod \"8e21f846-6c01-496f-998e-ccfe2a18d5cd\" (UID: \"8e21f846-6c01-496f-998e-ccfe2a18d5cd\") " Oct 30 17:26:12.648877 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:12.648133 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e21f846-6c01-496f-998e-ccfe2a18d5cd-credential-socket" pod "8e21f846-6c01-496f-998e-ccfe2a18d5cd" (UID: "8e21f846-6c01-496f-998e-ccfe2a18d5cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:26:12.648877 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:12.648319 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e21f846-6c01-496f-998e-ccfe2a18d5cd-workload-socket" pod "8e21f846-6c01-496f-998e-ccfe2a18d5cd" (UID: "8e21f846-6c01-496f-998e-ccfe2a18d5cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:26:12.648877 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:12.648498 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e21f846-6c01-496f-998e-ccfe2a18d5cd-istio-data" pod "8e21f846-6c01-496f-998e-ccfe2a18d5cd" (UID: "8e21f846-6c01-496f-998e-ccfe2a18d5cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:26:12.648877 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:12.648807 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e21f846-6c01-496f-998e-ccfe2a18d5cd-istiod-ca-cert" pod "8e21f846-6c01-496f-998e-ccfe2a18d5cd" (UID: "8e21f846-6c01-496f-998e-ccfe2a18d5cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:26:12.649523 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:12.649037 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e21f846-6c01-496f-998e-ccfe2a18d5cd-workload-certs" pod "8e21f846-6c01-496f-998e-ccfe2a18d5cd" (UID: "8e21f846-6c01-496f-998e-ccfe2a18d5cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:26:12.652055 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:12.652027 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e21f846-6c01-496f-998e-ccfe2a18d5cd-istio-podinfo" pod "8e21f846-6c01-496f-998e-ccfe2a18d5cd" (UID: "8e21f846-6c01-496f-998e-ccfe2a18d5cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:26:12.654071 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:12.654043 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e21f846-6c01-496f-998e-ccfe2a18d5cd-istio-token" pod "8e21f846-6c01-496f-998e-ccfe2a18d5cd" (UID: "8e21f846-6c01-496f-998e-ccfe2a18d5cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:26:12.654211 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:12.654157 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e21f846-6c01-496f-998e-ccfe2a18d5cd-istio-envoy" pod "8e21f846-6c01-496f-998e-ccfe2a18d5cd" (UID: "8e21f846-6c01-496f-998e-ccfe2a18d5cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:26:12.655572 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:12.655551 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e21f846-6c01-496f-998e-ccfe2a18d5cd-kube-api-access-qz7cb" pod "8e21f846-6c01-496f-998e-ccfe2a18d5cd" (UID: "8e21f846-6c01-496f-998e-ccfe2a18d5cd"). InnerVolumeSpecName "kube-api-access-qz7cb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:26:12.749020 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:12.748938 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e21f846-6c01-496f-998e-ccfe2a18d5cd-credential-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:26:12.749020 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:12.748967 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e21f846-6c01-496f-998e-ccfe2a18d5cd-istio-data\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:26:12.749020 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:12.748978 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e21f846-6c01-496f-998e-ccfe2a18d5cd-istio-envoy\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:26:12.749020 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:12.748987 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e21f846-6c01-496f-998e-ccfe2a18d5cd-istio-token\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:26:12.749020 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:12.748995 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e21f846-6c01-496f-998e-ccfe2a18d5cd-workload-certs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:26:12.749020 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:12.749004 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e21f846-6c01-496f-998e-ccfe2a18d5cd-istiod-ca-cert\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:26:12.749020 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:12.749017 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e21f846-6c01-496f-998e-ccfe2a18d5cd-istio-podinfo\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:26:12.749020 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:12.749026 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qz7cb\" (UniqueName: \"kubernetes.io/projected/8e21f846-6c01-496f-998e-ccfe2a18d5cd-kube-api-access-qz7cb\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:26:12.749418 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:12.749035 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e21f846-6c01-496f-998e-ccfe2a18d5cd-workload-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:26:13.160701 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:13.160665 4493 generic.go:356] "Generic (PLEG): container finished" podID="8e21f846-6c01-496f-998e-ccfe2a18d5cd" containerID="b00a36923850817ca34a07fff2dc81de8bbd8b84e0459f629104b19a2bce9295" exitCode=0 Oct 30 17:26:13.161189 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:13.160732 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w" Oct 30 17:26:13.161189 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:13.160752 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w" event={"ID":"8e21f846-6c01-496f-998e-ccfe2a18d5cd","Type":"ContainerDied","Data":"b00a36923850817ca34a07fff2dc81de8bbd8b84e0459f629104b19a2bce9295"} Oct 30 17:26:13.161189 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:13.160863 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w" event={"ID":"8e21f846-6c01-496f-998e-ccfe2a18d5cd","Type":"ContainerDied","Data":"3498d6c1599541a6286792da99b9fdd6b7b48e52c2b1ecf42abbd6f038a27025"} Oct 30 17:26:13.161189 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:13.160893 4493 scope.go:122] "RemoveContainer" containerID="b00a36923850817ca34a07fff2dc81de8bbd8b84e0459f629104b19a2bce9295" Oct 30 17:26:13.162319 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:13.162297 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d93a3edf-8s8ws" Oct 30 17:26:13.162440 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:13.162338 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d93a3edf-8s8ws" event={"ID":"50e6c831-46ad-4a5f-adea-4271dcfd6eb2","Type":"ContainerDied","Data":"3bcd98ead49b8ca01b0881e9853a2211200ea85d9d2c13537019afec1c25e9ed"} Oct 30 17:26:13.162440 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:13.162385 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3bcd98ead49b8ca01b0881e9853a2211200ea85d9d2c13537019afec1c25e9ed" Oct 30 17:26:13.164449 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:13.164420 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-43197b-c5g9z" event={"ID":"4fba1cad-d578-4e53-afcb-3698d267a658","Type":"ContainerStarted","Data":"43087fcdc63a92249fbe8257eea158063ceb71669abbd96697db4e0579b1b054"} Oct 30 17:26:13.173602 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:13.173584 4493 scope.go:122] "RemoveContainer" containerID="5df355cf8894c3cd55e3c567e4a600995b29412c5e2b8a1ce4dc7165513c8b38" Oct 30 17:26:13.184022 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:13.184004 4493 scope.go:122] "RemoveContainer" containerID="b00a36923850817ca34a07fff2dc81de8bbd8b84e0459f629104b19a2bce9295" Oct 30 17:26:13.184296 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:26:13.184276 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b00a36923850817ca34a07fff2dc81de8bbd8b84e0459f629104b19a2bce9295\": container with ID starting with b00a36923850817ca34a07fff2dc81de8bbd8b84e0459f629104b19a2bce9295 not found: ID does not exist" containerID="b00a36923850817ca34a07fff2dc81de8bbd8b84e0459f629104b19a2bce9295" Oct 30 17:26:13.184376 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:13.184307 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b00a36923850817ca34a07fff2dc81de8bbd8b84e0459f629104b19a2bce9295"} err="failed to get container status \"b00a36923850817ca34a07fff2dc81de8bbd8b84e0459f629104b19a2bce9295\": rpc error: code = NotFound desc = could not find container \"b00a36923850817ca34a07fff2dc81de8bbd8b84e0459f629104b19a2bce9295\": container with ID starting with b00a36923850817ca34a07fff2dc81de8bbd8b84e0459f629104b19a2bce9295 not found: ID does not exist" Oct 30 17:26:13.184376 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:13.184332 4493 scope.go:122] "RemoveContainer" containerID="5df355cf8894c3cd55e3c567e4a600995b29412c5e2b8a1ce4dc7165513c8b38" Oct 30 17:26:13.184610 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:26:13.184585 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5df355cf8894c3cd55e3c567e4a600995b29412c5e2b8a1ce4dc7165513c8b38\": container with ID starting with 5df355cf8894c3cd55e3c567e4a600995b29412c5e2b8a1ce4dc7165513c8b38 not found: ID does not exist" containerID="5df355cf8894c3cd55e3c567e4a600995b29412c5e2b8a1ce4dc7165513c8b38" Oct 30 17:26:13.184695 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:13.184611 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5df355cf8894c3cd55e3c567e4a600995b29412c5e2b8a1ce4dc7165513c8b38"} err="failed to get container status \"5df355cf8894c3cd55e3c567e4a600995b29412c5e2b8a1ce4dc7165513c8b38\": rpc error: code = NotFound desc = could not find container \"5df355cf8894c3cd55e3c567e4a600995b29412c5e2b8a1ce4dc7165513c8b38\": container with ID starting with 5df355cf8894c3cd55e3c567e4a600995b29412c5e2b8a1ce4dc7165513c8b38 not found: ID does not exist" Oct 30 17:26:13.229772 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:13.229712 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-43197b-c5g9z" podStartSLOduration=2.229698732 podStartE2EDuration="2.229698732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:26:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:26:13.228924671 +0000 UTC m=+3800.456099768" watchObservedRunningTime="2026-10-30 17:26:13.229698732 +0000 UTC m=+3800.456873852" Oct 30 17:26:13.265982 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:13.265953 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w"] Oct 30 17:26:13.279343 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:13.279315 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-5c6b3c-openshift-default-5685c7d757-tx64w"] Oct 30 17:26:14.179450 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:14.179418 4493 generic.go:356] "Generic (PLEG): container finished" podID="4fba1cad-d578-4e53-afcb-3698d267a658" containerID="43087fcdc63a92249fbe8257eea158063ceb71669abbd96697db4e0579b1b054" exitCode=0 Oct 30 17:26:14.179904 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:14.179502 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-43197b-c5g9z" event={"ID":"4fba1cad-d578-4e53-afcb-3698d267a658","Type":"ContainerDied","Data":"43087fcdc63a92249fbe8257eea158063ceb71669abbd96697db4e0579b1b054"} Oct 30 17:26:14.934833 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:14.934776 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e21f846-6c01-496f-998e-ccfe2a18d5cd" path="/var/lib/kubelet/pods/8e21f846-6c01-496f-998e-ccfe2a18d5cd/volumes" Oct 30 17:26:15.359421 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:15.359395 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-43197b-c5g9z" Oct 30 17:26:15.487839 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:15.487717 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fba1cad-d578-4e53-afcb-3698d267a658-kube-api-access-hp7t9\" (UniqueName: \"kubernetes.io/projected/4fba1cad-d578-4e53-afcb-3698d267a658-kube-api-access-hp7t9\") pod \"4fba1cad-d578-4e53-afcb-3698d267a658\" (UID: \"4fba1cad-d578-4e53-afcb-3698d267a658\") " Oct 30 17:26:15.488014 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:15.487991 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4fba1cad-d578-4e53-afcb-3698d267a658-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fba1cad-d578-4e53-afcb-3698d267a658-maas-api-xks-ca\") pod \"4fba1cad-d578-4e53-afcb-3698d267a658\" (UID: \"4fba1cad-d578-4e53-afcb-3698d267a658\") " Oct 30 17:26:15.488127 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:15.488101 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fba1cad-d578-4e53-afcb-3698d267a658-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fba1cad-d578-4e53-afcb-3698d267a658-maas-api-openshift-ca\") pod \"4fba1cad-d578-4e53-afcb-3698d267a658\" (UID: \"4fba1cad-d578-4e53-afcb-3698d267a658\") " Oct 30 17:26:15.488414 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:15.488390 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fba1cad-d578-4e53-afcb-3698d267a658-maas-api-openshift-ca" pod "4fba1cad-d578-4e53-afcb-3698d267a658" (UID: "4fba1cad-d578-4e53-afcb-3698d267a658"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:26:15.488575 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:15.488557 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fba1cad-d578-4e53-afcb-3698d267a658-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:26:15.490060 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:15.490036 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fba1cad-d578-4e53-afcb-3698d267a658-kube-api-access-hp7t9" pod "4fba1cad-d578-4e53-afcb-3698d267a658" (UID: "4fba1cad-d578-4e53-afcb-3698d267a658"). InnerVolumeSpecName "kube-api-access-hp7t9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:26:15.490267 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:15.490240 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fba1cad-d578-4e53-afcb-3698d267a658-maas-api-xks-ca" pod "4fba1cad-d578-4e53-afcb-3698d267a658" (UID: "4fba1cad-d578-4e53-afcb-3698d267a658"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:26:15.589561 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:15.589529 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fba1cad-d578-4e53-afcb-3698d267a658-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:26:15.589561 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:15.589556 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hp7t9\" (UniqueName: \"kubernetes.io/projected/4fba1cad-d578-4e53-afcb-3698d267a658-kube-api-access-hp7t9\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:26:16.159763 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:16.159701 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw" Oct 30 17:26:16.197346 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:16.197312 4493 generic.go:356] "Generic (PLEG): container finished" podID="e7475582-02dd-456b-a5d5-d894755aba74" containerID="0f134a4f15ef6855db28d0d167317cb30d32595258bd3c610b5eb8266b17d493" exitCode=0 Oct 30 17:26:16.197500 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:16.197373 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw" Oct 30 17:26:16.197500 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:16.197405 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw" event={"ID":"e7475582-02dd-456b-a5d5-d894755aba74","Type":"ContainerDied","Data":"0f134a4f15ef6855db28d0d167317cb30d32595258bd3c610b5eb8266b17d493"} Oct 30 17:26:16.197500 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:16.197463 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw" event={"ID":"e7475582-02dd-456b-a5d5-d894755aba74","Type":"ContainerDied","Data":"c95fb708f621b3523ffbd44a28e29ea7adc82422fb32249f0e12f9789d8dd624"} Oct 30 17:26:16.197500 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:16.197488 4493 scope.go:122] "RemoveContainer" containerID="0f134a4f15ef6855db28d0d167317cb30d32595258bd3c610b5eb8266b17d493" Oct 30 17:26:16.199063 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:16.199043 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-43197b-c5g9z" event={"ID":"4fba1cad-d578-4e53-afcb-3698d267a658","Type":"ContainerDied","Data":"dc4ce7721cec7011993b51e6a5ff9ed7e9ea96488f30bbf6dd11a2a323cb09d1"} Oct 30 17:26:16.199063 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:16.199056 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-43197b-c5g9z" Oct 30 17:26:16.199247 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:16.199080 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc4ce7721cec7011993b51e6a5ff9ed7e9ea96488f30bbf6dd11a2a323cb09d1" Oct 30 17:26:16.210236 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:16.210219 4493 scope.go:122] "RemoveContainer" containerID="c723df0cc3d721fbe627362f65e2b31c4f3b447d38484c601f0b610ff0fc7e58" Oct 30 17:26:16.223152 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:16.223124 4493 scope.go:122] "RemoveContainer" containerID="0f134a4f15ef6855db28d0d167317cb30d32595258bd3c610b5eb8266b17d493" Oct 30 17:26:16.223508 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:26:16.223484 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f134a4f15ef6855db28d0d167317cb30d32595258bd3c610b5eb8266b17d493\": container with ID starting with 0f134a4f15ef6855db28d0d167317cb30d32595258bd3c610b5eb8266b17d493 not found: ID does not exist" containerID="0f134a4f15ef6855db28d0d167317cb30d32595258bd3c610b5eb8266b17d493" Oct 30 17:26:16.223601 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:16.223517 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f134a4f15ef6855db28d0d167317cb30d32595258bd3c610b5eb8266b17d493"} err="failed to get container status \"0f134a4f15ef6855db28d0d167317cb30d32595258bd3c610b5eb8266b17d493\": rpc error: code = NotFound desc = could not find container \"0f134a4f15ef6855db28d0d167317cb30d32595258bd3c610b5eb8266b17d493\": container with ID starting with 0f134a4f15ef6855db28d0d167317cb30d32595258bd3c610b5eb8266b17d493 not found: ID does not exist" Oct 30 17:26:16.223601 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:16.223538 4493 scope.go:122] "RemoveContainer" containerID="c723df0cc3d721fbe627362f65e2b31c4f3b447d38484c601f0b610ff0fc7e58" Oct 30 17:26:16.223878 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:26:16.223855 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c723df0cc3d721fbe627362f65e2b31c4f3b447d38484c601f0b610ff0fc7e58\": container with ID starting with c723df0cc3d721fbe627362f65e2b31c4f3b447d38484c601f0b610ff0fc7e58 not found: ID does not exist" containerID="c723df0cc3d721fbe627362f65e2b31c4f3b447d38484c601f0b610ff0fc7e58" Oct 30 17:26:16.223949 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:16.223885 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c723df0cc3d721fbe627362f65e2b31c4f3b447d38484c601f0b610ff0fc7e58"} err="failed to get container status \"c723df0cc3d721fbe627362f65e2b31c4f3b447d38484c601f0b610ff0fc7e58\": rpc error: code = NotFound desc = could not find container \"c723df0cc3d721fbe627362f65e2b31c4f3b447d38484c601f0b610ff0fc7e58\": container with ID starting with c723df0cc3d721fbe627362f65e2b31c4f3b447d38484c601f0b610ff0fc7e58 not found: ID does not exist" Oct 30 17:26:16.297192 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:16.297149 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7475582-02dd-456b-a5d5-d894755aba74-kube-api-access-x6v76\" (UniqueName: \"kubernetes.io/projected/e7475582-02dd-456b-a5d5-d894755aba74-kube-api-access-x6v76\") pod \"e7475582-02dd-456b-a5d5-d894755aba74\" (UID: \"e7475582-02dd-456b-a5d5-d894755aba74\") " Oct 30 17:26:16.297192 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:16.297199 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7475582-02dd-456b-a5d5-d894755aba74-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7475582-02dd-456b-a5d5-d894755aba74-istio-data\") pod \"e7475582-02dd-456b-a5d5-d894755aba74\" (UID: \"e7475582-02dd-456b-a5d5-d894755aba74\") " Oct 30 17:26:16.297472 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:16.297255 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7475582-02dd-456b-a5d5-d894755aba74-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7475582-02dd-456b-a5d5-d894755aba74-credential-socket\") pod \"e7475582-02dd-456b-a5d5-d894755aba74\" (UID: \"e7475582-02dd-456b-a5d5-d894755aba74\") " Oct 30 17:26:16.297472 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:16.297340 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7475582-02dd-456b-a5d5-d894755aba74-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7475582-02dd-456b-a5d5-d894755aba74-istio-envoy\") pod \"e7475582-02dd-456b-a5d5-d894755aba74\" (UID: \"e7475582-02dd-456b-a5d5-d894755aba74\") " Oct 30 17:26:16.297472 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:16.297370 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7475582-02dd-456b-a5d5-d894755aba74-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7475582-02dd-456b-a5d5-d894755aba74-istiod-ca-cert\") pod \"e7475582-02dd-456b-a5d5-d894755aba74\" (UID: \"e7475582-02dd-456b-a5d5-d894755aba74\") " Oct 30 17:26:16.297472 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:16.297397 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e7475582-02dd-456b-a5d5-d894755aba74-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7475582-02dd-456b-a5d5-d894755aba74-istio-podinfo\") pod \"e7475582-02dd-456b-a5d5-d894755aba74\" (UID: \"e7475582-02dd-456b-a5d5-d894755aba74\") " Oct 30 17:26:16.297472 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:16.297434 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7475582-02dd-456b-a5d5-d894755aba74-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7475582-02dd-456b-a5d5-d894755aba74-workload-certs\") pod \"e7475582-02dd-456b-a5d5-d894755aba74\" (UID: \"e7475582-02dd-456b-a5d5-d894755aba74\") " Oct 30 17:26:16.297751 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:16.297488 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7475582-02dd-456b-a5d5-d894755aba74-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7475582-02dd-456b-a5d5-d894755aba74-workload-socket\") pod \"e7475582-02dd-456b-a5d5-d894755aba74\" (UID: \"e7475582-02dd-456b-a5d5-d894755aba74\") " Oct 30 17:26:16.297751 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:16.297525 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7475582-02dd-456b-a5d5-d894755aba74-istio-token\" (UniqueName: \"kubernetes.io/projected/e7475582-02dd-456b-a5d5-d894755aba74-istio-token\") pod \"e7475582-02dd-456b-a5d5-d894755aba74\" (UID: \"e7475582-02dd-456b-a5d5-d894755aba74\") " Oct 30 17:26:16.297751 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:16.297542 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7475582-02dd-456b-a5d5-d894755aba74-istio-data" pod "e7475582-02dd-456b-a5d5-d894755aba74" (UID: "e7475582-02dd-456b-a5d5-d894755aba74"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:26:16.297751 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:16.297562 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7475582-02dd-456b-a5d5-d894755aba74-credential-socket" pod "e7475582-02dd-456b-a5d5-d894755aba74" (UID: "e7475582-02dd-456b-a5d5-d894755aba74"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:26:16.297985 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:16.297839 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7475582-02dd-456b-a5d5-d894755aba74-workload-certs" pod "e7475582-02dd-456b-a5d5-d894755aba74" (UID: "e7475582-02dd-456b-a5d5-d894755aba74"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:26:16.297985 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:16.297922 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7475582-02dd-456b-a5d5-d894755aba74-workload-socket" pod "e7475582-02dd-456b-a5d5-d894755aba74" (UID: "e7475582-02dd-456b-a5d5-d894755aba74"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:26:16.297985 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:16.297924 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7475582-02dd-456b-a5d5-d894755aba74-istiod-ca-cert" pod "e7475582-02dd-456b-a5d5-d894755aba74" (UID: "e7475582-02dd-456b-a5d5-d894755aba74"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:26:16.298146 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:16.298110 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7475582-02dd-456b-a5d5-d894755aba74-credential-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:26:16.298146 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:16.298131 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7475582-02dd-456b-a5d5-d894755aba74-istiod-ca-cert\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:26:16.298252 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:16.298147 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7475582-02dd-456b-a5d5-d894755aba74-workload-certs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:26:16.298252 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:16.298160 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7475582-02dd-456b-a5d5-d894755aba74-workload-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:26:16.298357 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:16.298254 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7475582-02dd-456b-a5d5-d894755aba74-istio-data\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:26:16.299853 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:16.299831 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e7475582-02dd-456b-a5d5-d894755aba74-istio-podinfo" pod "e7475582-02dd-456b-a5d5-d894755aba74" (UID: "e7475582-02dd-456b-a5d5-d894755aba74"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:26:16.300115 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:16.300098 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7475582-02dd-456b-a5d5-d894755aba74-istio-envoy" pod "e7475582-02dd-456b-a5d5-d894755aba74" (UID: "e7475582-02dd-456b-a5d5-d894755aba74"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:26:16.300307 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:16.300286 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7475582-02dd-456b-a5d5-d894755aba74-kube-api-access-x6v76" pod "e7475582-02dd-456b-a5d5-d894755aba74" (UID: "e7475582-02dd-456b-a5d5-d894755aba74"). InnerVolumeSpecName "kube-api-access-x6v76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:26:16.300416 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:16.300355 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7475582-02dd-456b-a5d5-d894755aba74-istio-token" pod "e7475582-02dd-456b-a5d5-d894755aba74" (UID: "e7475582-02dd-456b-a5d5-d894755aba74"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:26:16.399447 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:16.399409 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7475582-02dd-456b-a5d5-d894755aba74-istio-envoy\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:26:16.399447 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:16.399445 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7475582-02dd-456b-a5d5-d894755aba74-istio-podinfo\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:26:16.399447 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:16.399454 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7475582-02dd-456b-a5d5-d894755aba74-istio-token\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:26:16.399992 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:16.399465 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6v76\" (UniqueName: \"kubernetes.io/projected/e7475582-02dd-456b-a5d5-d894755aba74-kube-api-access-x6v76\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:26:16.548291 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:16.548260 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw"] Oct 30 17:26:16.555379 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:16.555354 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-9bb5b1-openshift-default-6bf4bc9585-c6rqw"] Oct 30 17:26:16.935343 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:16.935299 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7475582-02dd-456b-a5d5-d894755aba74" path="/var/lib/kubelet/pods/e7475582-02dd-456b-a5d5-d894755aba74/volumes" Oct 30 17:26:20.275891 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:20.274320 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-d80ebb/test-model-d80ebb-kserve-6f594b78b5-pg9wm"] Oct 30 17:26:20.804771 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:20.804741 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bea7af65-j69p2"] Oct 30 17:26:20.805304 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:20.805285 4493 cpu_manager.go:414] "containerMap: removing container" podUID="e7475582-02dd-456b-a5d5-d894755aba74" containerName="istio-proxy" Oct 30 17:26:20.805353 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:20.805307 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7475582-02dd-456b-a5d5-d894755aba74" containerName="istio-proxy" Oct 30 17:26:20.805353 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:20.805322 4493 cpu_manager.go:414] "containerMap: removing container" podUID="8e21f846-6c01-496f-998e-ccfe2a18d5cd" containerName="istio-proxy" Oct 30 17:26:20.805353 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:20.805327 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e21f846-6c01-496f-998e-ccfe2a18d5cd" containerName="istio-proxy" Oct 30 17:26:20.805353 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:20.805345 4493 cpu_manager.go:414] "containerMap: removing container" podUID="8e21f846-6c01-496f-998e-ccfe2a18d5cd" containerName="istio-proxy" Oct 30 17:26:20.805353 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:20.805353 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e21f846-6c01-496f-998e-ccfe2a18d5cd" containerName="istio-proxy" Oct 30 17:26:20.805548 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:20.805360 4493 cpu_manager.go:414] "containerMap: removing container" podUID="50e6c831-46ad-4a5f-adea-4271dcfd6eb2" containerName="revoke-keys" Oct 30 17:26:20.805548 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:20.805365 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50e6c831-46ad-4a5f-adea-4271dcfd6eb2" containerName="revoke-keys" Oct 30 17:26:20.805548 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:20.805386 4493 cpu_manager.go:414] "containerMap: removing container" podUID="4fba1cad-d578-4e53-afcb-3698d267a658" containerName="revoke-keys" Oct 30 17:26:20.805548 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:20.805390 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fba1cad-d578-4e53-afcb-3698d267a658" containerName="revoke-keys" Oct 30 17:26:20.805548 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:20.805467 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e21f846-6c01-496f-998e-ccfe2a18d5cd" containerName="istio-proxy" Oct 30 17:26:20.805548 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:20.805476 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="50e6c831-46ad-4a5f-adea-4271dcfd6eb2" containerName="revoke-keys" Oct 30 17:26:20.805548 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:20.805482 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7475582-02dd-456b-a5d5-d894755aba74" containerName="istio-proxy" Oct 30 17:26:20.805548 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:20.805489 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fba1cad-d578-4e53-afcb-3698d267a658" containerName="revoke-keys" Oct 30 17:26:20.810674 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:20.810654 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bea7af65-j69p2" Oct 30 17:26:20.827078 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:20.827054 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:26:20.831413 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:20.831387 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bea7af65-j69p2"] Oct 30 17:26:20.838595 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:20.838555 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 17:26:20.859104 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:20.859080 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04895c70-8d15-4a8a-81fa-e40f18a2563c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bea7af65-j69p2\" (UID: \"04895c70-8d15-4a8a-81fa-e40f18a2563c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bea7af65-j69p2" Oct 30 17:26:20.859256 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:20.859109 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phg5c\" (UniqueName: \"kubernetes.io/projected/04895c70-8d15-4a8a-81fa-e40f18a2563c-kube-api-access-phg5c\") pod \"maas-api-revoke-sub-bea7af65-j69p2\" (UID: \"04895c70-8d15-4a8a-81fa-e40f18a2563c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bea7af65-j69p2" Oct 30 17:26:20.859256 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:20.859142 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04895c70-8d15-4a8a-81fa-e40f18a2563c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bea7af65-j69p2\" (UID: \"04895c70-8d15-4a8a-81fa-e40f18a2563c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bea7af65-j69p2" Oct 30 17:26:20.887198 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:20.887170 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:26:20.887346 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:20.887327 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:26:20.960896 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:20.960857 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04895c70-8d15-4a8a-81fa-e40f18a2563c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bea7af65-j69p2\" (UID: \"04895c70-8d15-4a8a-81fa-e40f18a2563c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bea7af65-j69p2" Oct 30 17:26:20.961115 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:20.961098 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04895c70-8d15-4a8a-81fa-e40f18a2563c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bea7af65-j69p2\" (UID: \"04895c70-8d15-4a8a-81fa-e40f18a2563c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bea7af65-j69p2" Oct 30 17:26:20.961206 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:20.961127 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phg5c\" (UniqueName: \"kubernetes.io/projected/04895c70-8d15-4a8a-81fa-e40f18a2563c-kube-api-access-phg5c\") pod \"maas-api-revoke-sub-bea7af65-j69p2\" (UID: \"04895c70-8d15-4a8a-81fa-e40f18a2563c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bea7af65-j69p2" Oct 30 17:26:20.961606 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:20.961583 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04895c70-8d15-4a8a-81fa-e40f18a2563c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bea7af65-j69p2\" (UID: \"04895c70-8d15-4a8a-81fa-e40f18a2563c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bea7af65-j69p2" Oct 30 17:26:20.963529 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:20.963511 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04895c70-8d15-4a8a-81fa-e40f18a2563c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bea7af65-j69p2\" (UID: \"04895c70-8d15-4a8a-81fa-e40f18a2563c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bea7af65-j69p2" Oct 30 17:26:20.984389 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:20.984363 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phg5c\" (UniqueName: \"kubernetes.io/projected/04895c70-8d15-4a8a-81fa-e40f18a2563c-kube-api-access-phg5c\") pod \"maas-api-revoke-sub-bea7af65-j69p2\" (UID: \"04895c70-8d15-4a8a-81fa-e40f18a2563c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bea7af65-j69p2" Oct 30 17:26:21.125049 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:21.124961 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bea7af65-j69p2" Oct 30 17:26:21.444325 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:21.444295 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bea7af65-j69p2"] Oct 30 17:26:21.444771 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:26:21.444332 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04895c70_8d15_4a8a_81fa_e40f18a2563c.slice/crio-740d189a54de3075453d54f0c89e9491115c386ef499c20d320f316a0e3331e1 WatchSource:0}: Error finding container 740d189a54de3075453d54f0c89e9491115c386ef499c20d320f316a0e3331e1: Status 404 returned error can't find the container with id 740d189a54de3075453d54f0c89e9491115c386ef499c20d320f316a0e3331e1 Oct 30 17:26:22.269065 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:22.269024 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bea7af65-j69p2" event={"ID":"04895c70-8d15-4a8a-81fa-e40f18a2563c","Type":"ContainerStarted","Data":"921531ba95fe85c32bc053b9178660ee155ddc33c8d117dddc9e7538432e344c"} Oct 30 17:26:22.269231 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:22.269087 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bea7af65-j69p2" event={"ID":"04895c70-8d15-4a8a-81fa-e40f18a2563c","Type":"ContainerStarted","Data":"740d189a54de3075453d54f0c89e9491115c386ef499c20d320f316a0e3331e1"} Oct 30 17:26:22.310300 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:22.310239 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bea7af65-j69p2" podStartSLOduration=2.310220973 podStartE2EDuration="2.310220973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:26:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:26:22.307913726 +0000 UTC m=+3809.535088839" watchObservedRunningTime="2026-10-30 17:26:22.310220973 +0000 UTC m=+3809.537396074" Oct 30 17:26:23.278500 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:23.278445 4493 generic.go:356] "Generic (PLEG): container finished" podID="04895c70-8d15-4a8a-81fa-e40f18a2563c" containerID="921531ba95fe85c32bc053b9178660ee155ddc33c8d117dddc9e7538432e344c" exitCode=0 Oct 30 17:26:23.278935 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:23.278668 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bea7af65-j69p2" event={"ID":"04895c70-8d15-4a8a-81fa-e40f18a2563c","Type":"ContainerDied","Data":"921531ba95fe85c32bc053b9178660ee155ddc33c8d117dddc9e7538432e344c"} Oct 30 17:26:24.441193 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:24.441164 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bea7af65-j69p2" Oct 30 17:26:24.505539 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:24.505503 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04895c70-8d15-4a8a-81fa-e40f18a2563c-kube-api-access-phg5c\" (UniqueName: \"kubernetes.io/projected/04895c70-8d15-4a8a-81fa-e40f18a2563c-kube-api-access-phg5c\") pod \"04895c70-8d15-4a8a-81fa-e40f18a2563c\" (UID: \"04895c70-8d15-4a8a-81fa-e40f18a2563c\") " Oct 30 17:26:24.505752 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:24.505607 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04895c70-8d15-4a8a-81fa-e40f18a2563c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04895c70-8d15-4a8a-81fa-e40f18a2563c-maas-api-openshift-ca\") pod \"04895c70-8d15-4a8a-81fa-e40f18a2563c\" (UID: \"04895c70-8d15-4a8a-81fa-e40f18a2563c\") " Oct 30 17:26:24.505752 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:24.505665 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04895c70-8d15-4a8a-81fa-e40f18a2563c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04895c70-8d15-4a8a-81fa-e40f18a2563c-maas-api-xks-ca\") pod \"04895c70-8d15-4a8a-81fa-e40f18a2563c\" (UID: \"04895c70-8d15-4a8a-81fa-e40f18a2563c\") " Oct 30 17:26:24.506230 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:24.506201 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04895c70-8d15-4a8a-81fa-e40f18a2563c-maas-api-openshift-ca" pod "04895c70-8d15-4a8a-81fa-e40f18a2563c" (UID: "04895c70-8d15-4a8a-81fa-e40f18a2563c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:26:24.508704 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:24.508676 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04895c70-8d15-4a8a-81fa-e40f18a2563c-kube-api-access-phg5c" pod "04895c70-8d15-4a8a-81fa-e40f18a2563c" (UID: "04895c70-8d15-4a8a-81fa-e40f18a2563c"). InnerVolumeSpecName "kube-api-access-phg5c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:26:24.508886 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:24.508826 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04895c70-8d15-4a8a-81fa-e40f18a2563c-maas-api-xks-ca" pod "04895c70-8d15-4a8a-81fa-e40f18a2563c" (UID: "04895c70-8d15-4a8a-81fa-e40f18a2563c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:26:24.607933 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:24.607861 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phg5c\" (UniqueName: \"kubernetes.io/projected/04895c70-8d15-4a8a-81fa-e40f18a2563c-kube-api-access-phg5c\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:26:24.607933 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:24.607889 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04895c70-8d15-4a8a-81fa-e40f18a2563c-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:26:24.607933 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:24.607899 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04895c70-8d15-4a8a-81fa-e40f18a2563c-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:26:25.296571 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:25.296490 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bea7af65-j69p2" Oct 30 17:26:25.296571 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:25.296532 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bea7af65-j69p2" event={"ID":"04895c70-8d15-4a8a-81fa-e40f18a2563c","Type":"ContainerDied","Data":"740d189a54de3075453d54f0c89e9491115c386ef499c20d320f316a0e3331e1"} Oct 30 17:26:25.296860 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:25.296599 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="740d189a54de3075453d54f0c89e9491115c386ef499c20d320f316a0e3331e1" Oct 30 17:26:30.344348 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:26:30.344311 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 17:26:34.769093 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:34.769005 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cb6251-mmbcd"] Oct 30 17:26:34.771199 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:34.771178 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-cb6251-mmbcd"] Oct 30 17:26:34.935976 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:34.935947 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96228ff4-3b01-46dd-93da-15465c07776c" path="/var/lib/kubelet/pods/96228ff4-3b01-46dd-93da-15465c07776c/volumes" Oct 30 17:26:35.318559 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:35.318515 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-d80ebb/test-model-d80ebb-kserve-6f594b78b5-pg9wm" podUID="442ab715-fe12-4e4c-8ca3-c452a3739cbe" containerName="main" containerID="cri-o://2ca03535acc16d82933a03da51cecd8827b2874ccd997ef62583b20e76161597" gracePeriod=45 Oct 30 17:26:35.597294 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:35.597266 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d80ebb/test-model-d80ebb-kserve-6f594b78b5-pg9wm" Oct 30 17:26:35.752070 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:35.752035 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/442ab715-fe12-4e4c-8ca3-c452a3739cbe-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/442ab715-fe12-4e4c-8ca3-c452a3739cbe-model-cache\") pod \"442ab715-fe12-4e4c-8ca3-c452a3739cbe\" (UID: \"442ab715-fe12-4e4c-8ca3-c452a3739cbe\") " Oct 30 17:26:35.752239 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:35.752123 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/442ab715-fe12-4e4c-8ca3-c452a3739cbe-dshm\" (UniqueName: \"kubernetes.io/empty-dir/442ab715-fe12-4e4c-8ca3-c452a3739cbe-dshm\") pod \"442ab715-fe12-4e4c-8ca3-c452a3739cbe\" (UID: \"442ab715-fe12-4e4c-8ca3-c452a3739cbe\") " Oct 30 17:26:35.752239 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:35.752193 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/442ab715-fe12-4e4c-8ca3-c452a3739cbe-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/442ab715-fe12-4e4c-8ca3-c452a3739cbe-tmp-dir\") pod \"442ab715-fe12-4e4c-8ca3-c452a3739cbe\" (UID: \"442ab715-fe12-4e4c-8ca3-c452a3739cbe\") " Oct 30 17:26:35.752388 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:35.752245 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/442ab715-fe12-4e4c-8ca3-c452a3739cbe-home\" (UniqueName: \"kubernetes.io/empty-dir/442ab715-fe12-4e4c-8ca3-c452a3739cbe-home\") pod \"442ab715-fe12-4e4c-8ca3-c452a3739cbe\" (UID: \"442ab715-fe12-4e4c-8ca3-c452a3739cbe\") " Oct 30 17:26:35.752388 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:35.752305 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/442ab715-fe12-4e4c-8ca3-c452a3739cbe-kube-api-access-q4wj7\" (UniqueName: \"kubernetes.io/projected/442ab715-fe12-4e4c-8ca3-c452a3739cbe-kube-api-access-q4wj7\") pod \"442ab715-fe12-4e4c-8ca3-c452a3739cbe\" (UID: \"442ab715-fe12-4e4c-8ca3-c452a3739cbe\") " Oct 30 17:26:35.752388 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:35.752327 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/442ab715-fe12-4e4c-8ca3-c452a3739cbe-model-cache" pod "442ab715-fe12-4e4c-8ca3-c452a3739cbe" (UID: "442ab715-fe12-4e4c-8ca3-c452a3739cbe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:26:35.752388 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:35.752354 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/442ab715-fe12-4e4c-8ca3-c452a3739cbe-tls-certs\" (UniqueName: \"kubernetes.io/secret/442ab715-fe12-4e4c-8ca3-c452a3739cbe-tls-certs\") pod \"442ab715-fe12-4e4c-8ca3-c452a3739cbe\" (UID: \"442ab715-fe12-4e4c-8ca3-c452a3739cbe\") " Oct 30 17:26:35.752596 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:35.752535 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/442ab715-fe12-4e4c-8ca3-c452a3739cbe-tmp-dir" pod "442ab715-fe12-4e4c-8ca3-c452a3739cbe" (UID: "442ab715-fe12-4e4c-8ca3-c452a3739cbe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:26:35.752902 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:35.752876 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/442ab715-fe12-4e4c-8ca3-c452a3739cbe-home" pod "442ab715-fe12-4e4c-8ca3-c452a3739cbe" (UID: "442ab715-fe12-4e4c-8ca3-c452a3739cbe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:26:35.753227 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:35.753012 4493 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/442ab715-fe12-4e4c-8ca3-c452a3739cbe-model-cache\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:26:35.753227 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:35.753025 4493 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/442ab715-fe12-4e4c-8ca3-c452a3739cbe-tmp-dir\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:26:35.753227 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:35.753033 4493 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/442ab715-fe12-4e4c-8ca3-c452a3739cbe-home\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:26:35.754858 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:35.754830 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/442ab715-fe12-4e4c-8ca3-c452a3739cbe-tls-certs" pod "442ab715-fe12-4e4c-8ca3-c452a3739cbe" (UID: "442ab715-fe12-4e4c-8ca3-c452a3739cbe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:26:35.754858 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:35.754843 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/442ab715-fe12-4e4c-8ca3-c452a3739cbe-kube-api-access-q4wj7" pod "442ab715-fe12-4e4c-8ca3-c452a3739cbe" (UID: "442ab715-fe12-4e4c-8ca3-c452a3739cbe"). InnerVolumeSpecName "kube-api-access-q4wj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:26:35.755052 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:35.754967 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/442ab715-fe12-4e4c-8ca3-c452a3739cbe-dshm" pod "442ab715-fe12-4e4c-8ca3-c452a3739cbe" (UID: "442ab715-fe12-4e4c-8ca3-c452a3739cbe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:26:35.854347 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:35.854261 4493 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/442ab715-fe12-4e4c-8ca3-c452a3739cbe-dshm\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:26:35.854347 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:35.854297 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4wj7\" (UniqueName: \"kubernetes.io/projected/442ab715-fe12-4e4c-8ca3-c452a3739cbe-kube-api-access-q4wj7\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:26:35.854347 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:35.854312 4493 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/442ab715-fe12-4e4c-8ca3-c452a3739cbe-tls-certs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:26:36.386962 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:36.386928 4493 generic.go:356] "Generic (PLEG): container finished" podID="442ab715-fe12-4e4c-8ca3-c452a3739cbe" containerID="2ca03535acc16d82933a03da51cecd8827b2874ccd997ef62583b20e76161597" exitCode=0 Oct 30 17:26:36.387139 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:36.386989 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d80ebb/test-model-d80ebb-kserve-6f594b78b5-pg9wm" Oct 30 17:26:36.387139 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:36.387002 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d80ebb/test-model-d80ebb-kserve-6f594b78b5-pg9wm" event={"ID":"442ab715-fe12-4e4c-8ca3-c452a3739cbe","Type":"ContainerDied","Data":"2ca03535acc16d82933a03da51cecd8827b2874ccd997ef62583b20e76161597"} Oct 30 17:26:36.387139 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:36.387042 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d80ebb/test-model-d80ebb-kserve-6f594b78b5-pg9wm" event={"ID":"442ab715-fe12-4e4c-8ca3-c452a3739cbe","Type":"ContainerDied","Data":"9ac5a9d3e9b4bf48d92f58c9c06dba4b4579b10e951237f6c8ebd8f3df0345c4"} Oct 30 17:26:36.387139 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:36.387059 4493 scope.go:122] "RemoveContainer" containerID="2ca03535acc16d82933a03da51cecd8827b2874ccd997ef62583b20e76161597" Oct 30 17:26:36.398422 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:36.398401 4493 scope.go:122] "RemoveContainer" containerID="2ca03535acc16d82933a03da51cecd8827b2874ccd997ef62583b20e76161597" Oct 30 17:26:36.398733 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:26:36.398707 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ca03535acc16d82933a03da51cecd8827b2874ccd997ef62583b20e76161597\": container with ID starting with 2ca03535acc16d82933a03da51cecd8827b2874ccd997ef62583b20e76161597 not found: ID does not exist" containerID="2ca03535acc16d82933a03da51cecd8827b2874ccd997ef62583b20e76161597" Oct 30 17:26:36.398870 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:36.398743 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ca03535acc16d82933a03da51cecd8827b2874ccd997ef62583b20e76161597"} err="failed to get container status \"2ca03535acc16d82933a03da51cecd8827b2874ccd997ef62583b20e76161597\": rpc error: code = NotFound desc = could not find container \"2ca03535acc16d82933a03da51cecd8827b2874ccd997ef62583b20e76161597\": container with ID starting with 2ca03535acc16d82933a03da51cecd8827b2874ccd997ef62583b20e76161597 not found: ID does not exist" Oct 30 17:26:36.439976 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:36.439948 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-d80ebb/test-model-d80ebb-kserve-6f594b78b5-pg9wm"] Oct 30 17:26:36.450848 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:36.450795 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-d80ebb/test-model-d80ebb-kserve-6f594b78b5-pg9wm"] Oct 30 17:26:36.935455 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:36.935423 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="442ab715-fe12-4e4c-8ca3-c452a3739cbe" path="/var/lib/kubelet/pods/442ab715-fe12-4e4c-8ca3-c452a3739cbe/volumes" Oct 30 17:26:38.042225 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:38.042188 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-0b09d3/test-model-0b09d3-kserve-64fc7fb899-wsjbz"] Oct 30 17:26:38.312290 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:38.312215 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d80ebb-4rd65"] Oct 30 17:26:38.313032 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:38.313009 4493 cpu_manager.go:414] "containerMap: removing container" podUID="04895c70-8d15-4a8a-81fa-e40f18a2563c" containerName="revoke-keys" Oct 30 17:26:38.313127 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:38.313036 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04895c70-8d15-4a8a-81fa-e40f18a2563c" containerName="revoke-keys" Oct 30 17:26:38.313127 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:38.313059 4493 cpu_manager.go:414] "containerMap: removing container" podUID="442ab715-fe12-4e4c-8ca3-c452a3739cbe" containerName="main" Oct 30 17:26:38.313127 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:38.313066 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="442ab715-fe12-4e4c-8ca3-c452a3739cbe" containerName="main" Oct 30 17:26:38.313127 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:38.313100 4493 cpu_manager.go:414] "containerMap: removing container" podUID="e7475582-02dd-456b-a5d5-d894755aba74" containerName="istio-proxy" Oct 30 17:26:38.313127 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:38.313108 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7475582-02dd-456b-a5d5-d894755aba74" containerName="istio-proxy" Oct 30 17:26:38.313321 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:38.313244 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e21f846-6c01-496f-998e-ccfe2a18d5cd" containerName="istio-proxy" Oct 30 17:26:38.313321 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:38.313260 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7475582-02dd-456b-a5d5-d894755aba74" containerName="istio-proxy" Oct 30 17:26:38.313321 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:38.313269 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="04895c70-8d15-4a8a-81fa-e40f18a2563c" containerName="revoke-keys" Oct 30 17:26:38.313321 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:38.313279 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="442ab715-fe12-4e4c-8ca3-c452a3739cbe" containerName="main" Oct 30 17:26:38.318769 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:38.318746 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d80ebb-4rd65" Oct 30 17:26:38.332939 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:38.332910 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:26:38.333105 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:38.333085 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 17:26:38.333398 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:38.333372 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:26:38.333522 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:38.333480 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:26:38.338870 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:38.338842 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d80ebb-4rd65"] Oct 30 17:26:38.387503 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:38.387470 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/024df33d-2b27-4059-a0c7-0a10ff2928d1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d80ebb-4rd65\" (UID: \"024df33d-2b27-4059-a0c7-0a10ff2928d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d80ebb-4rd65" Oct 30 17:26:38.387683 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:38.387589 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/024df33d-2b27-4059-a0c7-0a10ff2928d1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d80ebb-4rd65\" (UID: \"024df33d-2b27-4059-a0c7-0a10ff2928d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d80ebb-4rd65" Oct 30 17:26:38.387683 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:38.387651 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7frmn\" (UniqueName: \"kubernetes.io/projected/024df33d-2b27-4059-a0c7-0a10ff2928d1-kube-api-access-7frmn\") pod \"maas-api-revoke-keys-e2e-inf-a-d80ebb-4rd65\" (UID: \"024df33d-2b27-4059-a0c7-0a10ff2928d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d80ebb-4rd65" Oct 30 17:26:38.488685 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:38.488641 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/024df33d-2b27-4059-a0c7-0a10ff2928d1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d80ebb-4rd65\" (UID: \"024df33d-2b27-4059-a0c7-0a10ff2928d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d80ebb-4rd65" Oct 30 17:26:38.488943 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:38.488748 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/024df33d-2b27-4059-a0c7-0a10ff2928d1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d80ebb-4rd65\" (UID: \"024df33d-2b27-4059-a0c7-0a10ff2928d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d80ebb-4rd65" Oct 30 17:26:38.488943 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:38.488856 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7frmn\" (UniqueName: \"kubernetes.io/projected/024df33d-2b27-4059-a0c7-0a10ff2928d1-kube-api-access-7frmn\") pod \"maas-api-revoke-keys-e2e-inf-a-d80ebb-4rd65\" (UID: \"024df33d-2b27-4059-a0c7-0a10ff2928d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d80ebb-4rd65" Oct 30 17:26:38.489317 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:38.489297 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/024df33d-2b27-4059-a0c7-0a10ff2928d1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d80ebb-4rd65\" (UID: \"024df33d-2b27-4059-a0c7-0a10ff2928d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d80ebb-4rd65" Oct 30 17:26:38.491383 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:38.491363 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/024df33d-2b27-4059-a0c7-0a10ff2928d1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d80ebb-4rd65\" (UID: \"024df33d-2b27-4059-a0c7-0a10ff2928d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d80ebb-4rd65" Oct 30 17:26:38.511848 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:38.511824 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7frmn\" (UniqueName: \"kubernetes.io/projected/024df33d-2b27-4059-a0c7-0a10ff2928d1-kube-api-access-7frmn\") pod \"maas-api-revoke-keys-e2e-inf-a-d80ebb-4rd65\" (UID: \"024df33d-2b27-4059-a0c7-0a10ff2928d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d80ebb-4rd65" Oct 30 17:26:38.635394 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:38.635308 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d80ebb-4rd65" Oct 30 17:26:38.943170 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:38.943143 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d80ebb-4rd65"] Oct 30 17:26:38.944999 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:26:38.944963 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod024df33d_2b27_4059_a0c7_0a10ff2928d1.slice/crio-78091ac42d099527ebf47a55bfdf9bd5fa4fae5d243534320ef438fa58b0675d WatchSource:0}: Error finding container 78091ac42d099527ebf47a55bfdf9bd5fa4fae5d243534320ef438fa58b0675d: Status 404 returned error can't find the container with id 78091ac42d099527ebf47a55bfdf9bd5fa4fae5d243534320ef438fa58b0675d Oct 30 17:26:39.413837 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:39.413802 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d80ebb-4rd65" event={"ID":"024df33d-2b27-4059-a0c7-0a10ff2928d1","Type":"ContainerStarted","Data":"3400cc35dd102ad58c84f9fbb3432b2829e2a8ab513a436795cfdeeed06ffc3b"} Oct 30 17:26:39.414334 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:39.413865 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d80ebb-4rd65" event={"ID":"024df33d-2b27-4059-a0c7-0a10ff2928d1","Type":"ContainerStarted","Data":"78091ac42d099527ebf47a55bfdf9bd5fa4fae5d243534320ef438fa58b0675d"} Oct 30 17:26:39.457184 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:39.457127 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d80ebb-4rd65" podStartSLOduration=1.457109671 podStartE2EDuration="1.457109671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:26:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:26:39.453505054 +0000 UTC m=+3826.680680146" watchObservedRunningTime="2026-10-30 17:26:39.457109671 +0000 UTC m=+3826.684284775" Oct 30 17:26:40.258812 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:40.255066 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 17:26:40.473701 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:40.473666 4493 generic.go:356] "Generic (PLEG): container finished" podID="024df33d-2b27-4059-a0c7-0a10ff2928d1" containerID="3400cc35dd102ad58c84f9fbb3432b2829e2a8ab513a436795cfdeeed06ffc3b" exitCode=0 Oct 30 17:26:40.474515 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:40.473902 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d80ebb-4rd65" event={"ID":"024df33d-2b27-4059-a0c7-0a10ff2928d1","Type":"ContainerDied","Data":"3400cc35dd102ad58c84f9fbb3432b2829e2a8ab513a436795cfdeeed06ffc3b"} Oct 30 17:26:40.756517 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:40.756462 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-43197b-c5g9z"] Oct 30 17:26:40.760091 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:40.760064 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-43197b-c5g9z"] Oct 30 17:26:40.940345 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:40.940307 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fba1cad-d578-4e53-afcb-3698d267a658" path="/var/lib/kubelet/pods/4fba1cad-d578-4e53-afcb-3698d267a658/volumes" Oct 30 17:26:41.229082 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:41.229051 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5758b7-d9wls"] Oct 30 17:26:41.235808 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:41.235730 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5758b7-d9wls"] Oct 30 17:26:41.825691 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:41.825664 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d80ebb-4rd65" Oct 30 17:26:41.874142 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:41.874107 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/024df33d-2b27-4059-a0c7-0a10ff2928d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/024df33d-2b27-4059-a0c7-0a10ff2928d1-maas-api-xks-ca\") pod \"024df33d-2b27-4059-a0c7-0a10ff2928d1\" (UID: \"024df33d-2b27-4059-a0c7-0a10ff2928d1\") " Oct 30 17:26:41.874345 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:41.874264 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/024df33d-2b27-4059-a0c7-0a10ff2928d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/024df33d-2b27-4059-a0c7-0a10ff2928d1-maas-api-openshift-ca\") pod \"024df33d-2b27-4059-a0c7-0a10ff2928d1\" (UID: \"024df33d-2b27-4059-a0c7-0a10ff2928d1\") " Oct 30 17:26:41.874345 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:41.874338 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/024df33d-2b27-4059-a0c7-0a10ff2928d1-kube-api-access-7frmn\" (UniqueName: \"kubernetes.io/projected/024df33d-2b27-4059-a0c7-0a10ff2928d1-kube-api-access-7frmn\") pod \"024df33d-2b27-4059-a0c7-0a10ff2928d1\" (UID: \"024df33d-2b27-4059-a0c7-0a10ff2928d1\") " Oct 30 17:26:41.874682 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:41.874652 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/024df33d-2b27-4059-a0c7-0a10ff2928d1-maas-api-openshift-ca" pod "024df33d-2b27-4059-a0c7-0a10ff2928d1" (UID: "024df33d-2b27-4059-a0c7-0a10ff2928d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:26:41.874860 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:41.874806 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/024df33d-2b27-4059-a0c7-0a10ff2928d1-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:26:41.887049 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:41.885945 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/024df33d-2b27-4059-a0c7-0a10ff2928d1-maas-api-xks-ca" pod "024df33d-2b27-4059-a0c7-0a10ff2928d1" (UID: "024df33d-2b27-4059-a0c7-0a10ff2928d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:26:41.887049 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:41.885958 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/024df33d-2b27-4059-a0c7-0a10ff2928d1-kube-api-access-7frmn" pod "024df33d-2b27-4059-a0c7-0a10ff2928d1" (UID: "024df33d-2b27-4059-a0c7-0a10ff2928d1"). InnerVolumeSpecName "kube-api-access-7frmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:26:41.976405 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:41.976369 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7frmn\" (UniqueName: \"kubernetes.io/projected/024df33d-2b27-4059-a0c7-0a10ff2928d1-kube-api-access-7frmn\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:26:41.976405 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:41.976401 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/024df33d-2b27-4059-a0c7-0a10ff2928d1-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:26:42.493430 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:42.493396 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d80ebb-4rd65" event={"ID":"024df33d-2b27-4059-a0c7-0a10ff2928d1","Type":"ContainerDied","Data":"78091ac42d099527ebf47a55bfdf9bd5fa4fae5d243534320ef438fa58b0675d"} Oct 30 17:26:42.493654 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:42.493455 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78091ac42d099527ebf47a55bfdf9bd5fa4fae5d243534320ef438fa58b0675d" Oct 30 17:26:42.493654 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:42.493522 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d80ebb-4rd65" Oct 30 17:26:42.934662 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:42.934623 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f2f8c77-10d4-4a4f-97bf-6a1c50a4702f" path="/var/lib/kubelet/pods/4f2f8c77-10d4-4a4f-97bf-6a1c50a4702f/volumes" Oct 30 17:26:43.733118 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:43.733086 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-deb721-2mgq9"] Oct 30 17:26:43.733686 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:43.733667 4493 cpu_manager.go:414] "containerMap: removing container" podUID="024df33d-2b27-4059-a0c7-0a10ff2928d1" containerName="revoke-keys" Oct 30 17:26:43.733777 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:43.733689 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="024df33d-2b27-4059-a0c7-0a10ff2928d1" containerName="revoke-keys" Oct 30 17:26:43.733836 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:43.733831 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="024df33d-2b27-4059-a0c7-0a10ff2928d1" containerName="revoke-keys" Oct 30 17:26:43.738564 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:43.738542 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-deb721-2mgq9" Oct 30 17:26:43.765754 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:43.765726 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:26:43.765931 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:43.765860 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 17:26:43.766008 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:43.765948 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:26:43.766070 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:43.766038 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:26:43.771890 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:43.771864 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-deb721-2mgq9"] Oct 30 17:26:43.807161 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:43.807120 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2711b00e-0aa2-4572-8bcd-41c78bddd8c2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-deb721-2mgq9\" (UID: \"2711b00e-0aa2-4572-8bcd-41c78bddd8c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-deb721-2mgq9" Oct 30 17:26:43.807317 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:43.807281 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm5nn\" (UniqueName: \"kubernetes.io/projected/2711b00e-0aa2-4572-8bcd-41c78bddd8c2-kube-api-access-sm5nn\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-deb721-2mgq9\" (UID: \"2711b00e-0aa2-4572-8bcd-41c78bddd8c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-deb721-2mgq9" Oct 30 17:26:43.807412 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:43.807396 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2711b00e-0aa2-4572-8bcd-41c78bddd8c2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-deb721-2mgq9\" (UID: \"2711b00e-0aa2-4572-8bcd-41c78bddd8c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-deb721-2mgq9" Oct 30 17:26:43.908874 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:43.908838 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm5nn\" (UniqueName: \"kubernetes.io/projected/2711b00e-0aa2-4572-8bcd-41c78bddd8c2-kube-api-access-sm5nn\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-deb721-2mgq9\" (UID: \"2711b00e-0aa2-4572-8bcd-41c78bddd8c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-deb721-2mgq9" Oct 30 17:26:43.909057 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:43.908927 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2711b00e-0aa2-4572-8bcd-41c78bddd8c2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-deb721-2mgq9\" (UID: \"2711b00e-0aa2-4572-8bcd-41c78bddd8c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-deb721-2mgq9" Oct 30 17:26:43.909057 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:43.909041 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2711b00e-0aa2-4572-8bcd-41c78bddd8c2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-deb721-2mgq9\" (UID: \"2711b00e-0aa2-4572-8bcd-41c78bddd8c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-deb721-2mgq9" Oct 30 17:26:43.909920 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:43.909894 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2711b00e-0aa2-4572-8bcd-41c78bddd8c2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-deb721-2mgq9\" (UID: \"2711b00e-0aa2-4572-8bcd-41c78bddd8c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-deb721-2mgq9" Oct 30 17:26:43.911663 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:43.911634 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2711b00e-0aa2-4572-8bcd-41c78bddd8c2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-deb721-2mgq9\" (UID: \"2711b00e-0aa2-4572-8bcd-41c78bddd8c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-deb721-2mgq9" Oct 30 17:26:43.931998 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:43.931964 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm5nn\" (UniqueName: \"kubernetes.io/projected/2711b00e-0aa2-4572-8bcd-41c78bddd8c2-kube-api-access-sm5nn\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-deb721-2mgq9\" (UID: \"2711b00e-0aa2-4572-8bcd-41c78bddd8c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-deb721-2mgq9" Oct 30 17:26:44.059538 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:44.059453 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-deb721-2mgq9" Oct 30 17:26:44.372033 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:44.372003 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-deb721-2mgq9"] Oct 30 17:26:44.373467 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:26:44.373444 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2711b00e_0aa2_4572_8bcd_41c78bddd8c2.slice/crio-5e202a02f57185bf6040917b5e93855f93c719aced229a27d9ceda34ff49db65 WatchSource:0}: Error finding container 5e202a02f57185bf6040917b5e93855f93c719aced229a27d9ceda34ff49db65: Status 404 returned error can't find the container with id 5e202a02f57185bf6040917b5e93855f93c719aced229a27d9ceda34ff49db65 Oct 30 17:26:44.513482 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:44.513437 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-deb721-2mgq9" event={"ID":"2711b00e-0aa2-4572-8bcd-41c78bddd8c2","Type":"ContainerStarted","Data":"5e202a02f57185bf6040917b5e93855f93c719aced229a27d9ceda34ff49db65"} Oct 30 17:26:45.526578 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:45.526471 4493 generic.go:356] "Generic (PLEG): container finished" podID="2711b00e-0aa2-4572-8bcd-41c78bddd8c2" containerID="a64cc02f4b332f34a2f6850216dc1ffb8cb12e777b035dbc1add1ead63d4073a" exitCode=0 Oct 30 17:26:45.527097 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:45.526609 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-deb721-2mgq9" event={"ID":"2711b00e-0aa2-4572-8bcd-41c78bddd8c2","Type":"ContainerDied","Data":"a64cc02f4b332f34a2f6850216dc1ffb8cb12e777b035dbc1add1ead63d4073a"} Oct 30 17:26:46.702306 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:46.702280 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-deb721-2mgq9" Oct 30 17:26:46.748632 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:46.746183 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2711b00e-0aa2-4572-8bcd-41c78bddd8c2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2711b00e-0aa2-4572-8bcd-41c78bddd8c2-maas-api-openshift-ca\") pod \"2711b00e-0aa2-4572-8bcd-41c78bddd8c2\" (UID: \"2711b00e-0aa2-4572-8bcd-41c78bddd8c2\") " Oct 30 17:26:46.748632 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:46.746337 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2711b00e-0aa2-4572-8bcd-41c78bddd8c2-kube-api-access-sm5nn\" (UniqueName: \"kubernetes.io/projected/2711b00e-0aa2-4572-8bcd-41c78bddd8c2-kube-api-access-sm5nn\") pod \"2711b00e-0aa2-4572-8bcd-41c78bddd8c2\" (UID: \"2711b00e-0aa2-4572-8bcd-41c78bddd8c2\") " Oct 30 17:26:46.748632 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:46.746405 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2711b00e-0aa2-4572-8bcd-41c78bddd8c2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2711b00e-0aa2-4572-8bcd-41c78bddd8c2-maas-api-xks-ca\") pod \"2711b00e-0aa2-4572-8bcd-41c78bddd8c2\" (UID: \"2711b00e-0aa2-4572-8bcd-41c78bddd8c2\") " Oct 30 17:26:46.748632 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:46.746753 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2711b00e-0aa2-4572-8bcd-41c78bddd8c2-maas-api-openshift-ca" pod "2711b00e-0aa2-4572-8bcd-41c78bddd8c2" (UID: "2711b00e-0aa2-4572-8bcd-41c78bddd8c2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:26:46.748632 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:46.746907 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2711b00e-0aa2-4572-8bcd-41c78bddd8c2-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:26:46.749109 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:46.749085 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2711b00e-0aa2-4572-8bcd-41c78bddd8c2-maas-api-xks-ca" pod "2711b00e-0aa2-4572-8bcd-41c78bddd8c2" (UID: "2711b00e-0aa2-4572-8bcd-41c78bddd8c2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:26:46.749216 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:46.749200 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2711b00e-0aa2-4572-8bcd-41c78bddd8c2-kube-api-access-sm5nn" pod "2711b00e-0aa2-4572-8bcd-41c78bddd8c2" (UID: "2711b00e-0aa2-4572-8bcd-41c78bddd8c2"). InnerVolumeSpecName "kube-api-access-sm5nn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:26:46.848029 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:46.847934 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2711b00e-0aa2-4572-8bcd-41c78bddd8c2-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:26:46.848029 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:46.847961 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sm5nn\" (UniqueName: \"kubernetes.io/projected/2711b00e-0aa2-4572-8bcd-41c78bddd8c2-kube-api-access-sm5nn\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:26:47.547907 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:47.547876 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-deb721-2mgq9" Oct 30 17:26:47.548122 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:47.547907 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-deb721-2mgq9" event={"ID":"2711b00e-0aa2-4572-8bcd-41c78bddd8c2","Type":"ContainerDied","Data":"5e202a02f57185bf6040917b5e93855f93c719aced229a27d9ceda34ff49db65"} Oct 30 17:26:47.548122 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:47.547955 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e202a02f57185bf6040917b5e93855f93c719aced229a27d9ceda34ff49db65" Oct 30 17:26:48.701041 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:48.701007 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz"] Oct 30 17:26:48.704137 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:48.704095 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz" podUID="009a69cb-8246-4711-a31d-37d9ee856749" containerName="istio-proxy" containerID="cri-o://95cd765793bd4b262c39ddb7f3beba27afb8b6751ab0631132e70151908bfa8f" gracePeriod=30 Oct 30 17:26:49.305526 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:49.305492 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-799c76-4w8kz"] Oct 30 17:26:49.306055 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:49.306041 4493 cpu_manager.go:414] "containerMap: removing container" podUID="2711b00e-0aa2-4572-8bcd-41c78bddd8c2" containerName="revoke-keys" Oct 30 17:26:49.306120 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:49.306057 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2711b00e-0aa2-4572-8bcd-41c78bddd8c2" containerName="revoke-keys" Oct 30 17:26:49.306164 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:49.306146 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="2711b00e-0aa2-4572-8bcd-41c78bddd8c2" containerName="revoke-keys" Oct 30 17:26:49.309597 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:49.309576 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-799c76-4w8kz" Oct 30 17:26:49.323819 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:49.323762 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:26:49.323920 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:49.323894 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:26:49.323968 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:49.323938 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:26:49.324175 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:49.324159 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 17:26:49.348627 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:49.348587 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-799c76-4w8kz"] Oct 30 17:26:49.426944 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:49.403233 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edb07aa2-9aca-4cc8-a70d-3a5540545497-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-799c76-4w8kz\" (UID: \"edb07aa2-9aca-4cc8-a70d-3a5540545497\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-799c76-4w8kz" Oct 30 17:26:49.426944 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:49.403327 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p7jv\" (UniqueName: \"kubernetes.io/projected/edb07aa2-9aca-4cc8-a70d-3a5540545497-kube-api-access-5p7jv\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-799c76-4w8kz\" (UID: \"edb07aa2-9aca-4cc8-a70d-3a5540545497\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-799c76-4w8kz" Oct 30 17:26:49.426944 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:49.403394 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edb07aa2-9aca-4cc8-a70d-3a5540545497-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-799c76-4w8kz\" (UID: \"edb07aa2-9aca-4cc8-a70d-3a5540545497\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-799c76-4w8kz" Oct 30 17:26:49.504975 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:49.504939 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edb07aa2-9aca-4cc8-a70d-3a5540545497-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-799c76-4w8kz\" (UID: \"edb07aa2-9aca-4cc8-a70d-3a5540545497\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-799c76-4w8kz" Oct 30 17:26:49.505167 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:49.505102 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edb07aa2-9aca-4cc8-a70d-3a5540545497-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-799c76-4w8kz\" (UID: \"edb07aa2-9aca-4cc8-a70d-3a5540545497\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-799c76-4w8kz" Oct 30 17:26:49.505167 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:49.505141 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p7jv\" (UniqueName: \"kubernetes.io/projected/edb07aa2-9aca-4cc8-a70d-3a5540545497-kube-api-access-5p7jv\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-799c76-4w8kz\" (UID: \"edb07aa2-9aca-4cc8-a70d-3a5540545497\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-799c76-4w8kz" Oct 30 17:26:49.505749 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:49.505723 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edb07aa2-9aca-4cc8-a70d-3a5540545497-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-799c76-4w8kz\" (UID: \"edb07aa2-9aca-4cc8-a70d-3a5540545497\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-799c76-4w8kz" Oct 30 17:26:49.507516 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:49.507485 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edb07aa2-9aca-4cc8-a70d-3a5540545497-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-799c76-4w8kz\" (UID: \"edb07aa2-9aca-4cc8-a70d-3a5540545497\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-799c76-4w8kz" Oct 30 17:26:49.528727 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:49.528699 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p7jv\" (UniqueName: \"kubernetes.io/projected/edb07aa2-9aca-4cc8-a70d-3a5540545497-kube-api-access-5p7jv\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-799c76-4w8kz\" (UID: \"edb07aa2-9aca-4cc8-a70d-3a5540545497\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-799c76-4w8kz" Oct 30 17:26:49.628483 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:49.626992 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-799c76-4w8kz" Oct 30 17:26:49.651921 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:49.649514 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz"] Oct 30 17:26:49.651921 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:49.649759 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz" podUID="defb21aa-d0d6-48b8-ba50-4fcb900bc547" containerName="istio-proxy" containerID="cri-o://cd0d3f94c0db2d1fe7238d36caed62993ca7f437542ca6581af4ba858a26dcf8" gracePeriod=30 Oct 30 17:26:49.993887 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:49.993844 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-799c76-4w8kz"] Oct 30 17:26:49.995725 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:26:49.995682 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedb07aa2_9aca_4cc8_a70d_3a5540545497.slice/crio-77bc640df03c153f542e76608e207d32a457bfca8cd132141ff85ab6bf14e22a WatchSource:0}: Error finding container 77bc640df03c153f542e76608e207d32a457bfca8cd132141ff85ab6bf14e22a: Status 404 returned error can't find the container with id 77bc640df03c153f542e76608e207d32a457bfca8cd132141ff85ab6bf14e22a Oct 30 17:26:50.391952 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:50.391922 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-420ff3-lk9n2"] Oct 30 17:26:50.395911 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:50.395895 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-420ff3-lk9n2" Oct 30 17:26:50.418739 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:50.418711 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-420ff3-lk9n2"] Oct 30 17:26:50.427421 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:50.427393 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af343e15-1b0d-483f-bc11-7cb765c29fe4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-420ff3-lk9n2\" (UID: \"af343e15-1b0d-483f-bc11-7cb765c29fe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-420ff3-lk9n2" Oct 30 17:26:50.427565 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:50.427518 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l9x7\" (UniqueName: \"kubernetes.io/projected/af343e15-1b0d-483f-bc11-7cb765c29fe4-kube-api-access-9l9x7\") pod \"maas-api-revoke-keys-e2e-worker-w4-420ff3-lk9n2\" (UID: \"af343e15-1b0d-483f-bc11-7cb765c29fe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-420ff3-lk9n2" Oct 30 17:26:50.427668 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:50.427651 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af343e15-1b0d-483f-bc11-7cb765c29fe4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-420ff3-lk9n2\" (UID: \"af343e15-1b0d-483f-bc11-7cb765c29fe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-420ff3-lk9n2" Oct 30 17:26:50.528905 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:50.528871 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af343e15-1b0d-483f-bc11-7cb765c29fe4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-420ff3-lk9n2\" (UID: \"af343e15-1b0d-483f-bc11-7cb765c29fe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-420ff3-lk9n2" Oct 30 17:26:50.529050 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:50.528933 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l9x7\" (UniqueName: \"kubernetes.io/projected/af343e15-1b0d-483f-bc11-7cb765c29fe4-kube-api-access-9l9x7\") pod \"maas-api-revoke-keys-e2e-worker-w4-420ff3-lk9n2\" (UID: \"af343e15-1b0d-483f-bc11-7cb765c29fe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-420ff3-lk9n2" Oct 30 17:26:50.529050 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:50.528980 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af343e15-1b0d-483f-bc11-7cb765c29fe4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-420ff3-lk9n2\" (UID: \"af343e15-1b0d-483f-bc11-7cb765c29fe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-420ff3-lk9n2" Oct 30 17:26:50.529566 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:50.529545 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af343e15-1b0d-483f-bc11-7cb765c29fe4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-420ff3-lk9n2\" (UID: \"af343e15-1b0d-483f-bc11-7cb765c29fe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-420ff3-lk9n2" Oct 30 17:26:50.531346 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:50.531322 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af343e15-1b0d-483f-bc11-7cb765c29fe4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-420ff3-lk9n2\" (UID: \"af343e15-1b0d-483f-bc11-7cb765c29fe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-420ff3-lk9n2" Oct 30 17:26:50.558466 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:50.558441 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l9x7\" (UniqueName: \"kubernetes.io/projected/af343e15-1b0d-483f-bc11-7cb765c29fe4-kube-api-access-9l9x7\") pod \"maas-api-revoke-keys-e2e-worker-w4-420ff3-lk9n2\" (UID: \"af343e15-1b0d-483f-bc11-7cb765c29fe4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-420ff3-lk9n2" Oct 30 17:26:50.577872 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:50.577801 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-799c76-4w8kz" event={"ID":"edb07aa2-9aca-4cc8-a70d-3a5540545497","Type":"ContainerStarted","Data":"ebcb1c0560c3d219e97c08253c41bdf981ed6d852740986db0abce32742d1cb4"} Oct 30 17:26:50.577872 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:50.577867 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-799c76-4w8kz" event={"ID":"edb07aa2-9aca-4cc8-a70d-3a5540545497","Type":"ContainerStarted","Data":"77bc640df03c153f542e76608e207d32a457bfca8cd132141ff85ab6bf14e22a"} Oct 30 17:26:50.625326 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:50.625272 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-799c76-4w8kz" podStartSLOduration=1.625259384 podStartE2EDuration="1.625259384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:26:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:26:50.622282525 +0000 UTC m=+3837.849457626" watchObservedRunningTime="2026-10-30 17:26:50.625259384 +0000 UTC m=+3837.852434508" Oct 30 17:26:50.710015 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:50.709988 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-420ff3-lk9n2" Oct 30 17:26:51.019178 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:51.019154 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-420ff3-lk9n2"] Oct 30 17:26:51.090815 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:26:51.090759 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf343e15_1b0d_483f_bc11_7cb765c29fe4.slice/crio-bfa5be18248fdc3583b75bb6b296bbde20675bc0808bf00502fcdb6733708f88 WatchSource:0}: Error finding container bfa5be18248fdc3583b75bb6b296bbde20675bc0808bf00502fcdb6733708f88: Status 404 returned error can't find the container with id bfa5be18248fdc3583b75bb6b296bbde20675bc0808bf00502fcdb6733708f88 Oct 30 17:26:51.587402 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:51.587321 4493 generic.go:356] "Generic (PLEG): container finished" podID="edb07aa2-9aca-4cc8-a70d-3a5540545497" containerID="ebcb1c0560c3d219e97c08253c41bdf981ed6d852740986db0abce32742d1cb4" exitCode=0 Oct 30 17:26:51.587555 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:51.587413 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-799c76-4w8kz" event={"ID":"edb07aa2-9aca-4cc8-a70d-3a5540545497","Type":"ContainerDied","Data":"ebcb1c0560c3d219e97c08253c41bdf981ed6d852740986db0abce32742d1cb4"} Oct 30 17:26:51.589293 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:51.589265 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-420ff3-lk9n2" event={"ID":"af343e15-1b0d-483f-bc11-7cb765c29fe4","Type":"ContainerStarted","Data":"37fc7ba87cce94a9b042b6f21d53762be4a75c2a833bd2b7237fbfe802f29b5c"} Oct 30 17:26:51.589443 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:51.589311 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-420ff3-lk9n2" event={"ID":"af343e15-1b0d-483f-bc11-7cb765c29fe4","Type":"ContainerStarted","Data":"bfa5be18248fdc3583b75bb6b296bbde20675bc0808bf00502fcdb6733708f88"} Oct 30 17:26:51.721221 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:51.721159 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-420ff3-lk9n2" podStartSLOduration=1.721140879 podStartE2EDuration="1.721140879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:26:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:26:51.716217165 +0000 UTC m=+3838.943392266" watchObservedRunningTime="2026-10-30 17:26:51.721140879 +0000 UTC m=+3838.948315978" Oct 30 17:26:52.598949 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:52.598912 4493 generic.go:356] "Generic (PLEG): container finished" podID="af343e15-1b0d-483f-bc11-7cb765c29fe4" containerID="37fc7ba87cce94a9b042b6f21d53762be4a75c2a833bd2b7237fbfe802f29b5c" exitCode=0 Oct 30 17:26:52.599371 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:52.599005 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-420ff3-lk9n2" event={"ID":"af343e15-1b0d-483f-bc11-7cb765c29fe4","Type":"ContainerDied","Data":"37fc7ba87cce94a9b042b6f21d53762be4a75c2a833bd2b7237fbfe802f29b5c"} Oct 30 17:26:52.781269 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:52.781206 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-799c76-4w8kz" Oct 30 17:26:52.859134 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:52.859049 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edb07aa2-9aca-4cc8-a70d-3a5540545497-kube-api-access-5p7jv\" (UniqueName: \"kubernetes.io/projected/edb07aa2-9aca-4cc8-a70d-3a5540545497-kube-api-access-5p7jv\") pod \"edb07aa2-9aca-4cc8-a70d-3a5540545497\" (UID: \"edb07aa2-9aca-4cc8-a70d-3a5540545497\") " Oct 30 17:26:52.859318 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:52.859157 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edb07aa2-9aca-4cc8-a70d-3a5540545497-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edb07aa2-9aca-4cc8-a70d-3a5540545497-maas-api-openshift-ca\") pod \"edb07aa2-9aca-4cc8-a70d-3a5540545497\" (UID: \"edb07aa2-9aca-4cc8-a70d-3a5540545497\") " Oct 30 17:26:52.859318 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:52.859238 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/edb07aa2-9aca-4cc8-a70d-3a5540545497-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edb07aa2-9aca-4cc8-a70d-3a5540545497-maas-api-xks-ca\") pod \"edb07aa2-9aca-4cc8-a70d-3a5540545497\" (UID: \"edb07aa2-9aca-4cc8-a70d-3a5540545497\") " Oct 30 17:26:52.859892 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:52.859857 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edb07aa2-9aca-4cc8-a70d-3a5540545497-maas-api-openshift-ca" pod "edb07aa2-9aca-4cc8-a70d-3a5540545497" (UID: "edb07aa2-9aca-4cc8-a70d-3a5540545497"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:26:52.861731 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:52.861705 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edb07aa2-9aca-4cc8-a70d-3a5540545497-kube-api-access-5p7jv" pod "edb07aa2-9aca-4cc8-a70d-3a5540545497" (UID: "edb07aa2-9aca-4cc8-a70d-3a5540545497"). InnerVolumeSpecName "kube-api-access-5p7jv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:26:52.862701 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:52.862672 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edb07aa2-9aca-4cc8-a70d-3a5540545497-maas-api-xks-ca" pod "edb07aa2-9aca-4cc8-a70d-3a5540545497" (UID: "edb07aa2-9aca-4cc8-a70d-3a5540545497"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:26:52.960551 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:52.960528 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5p7jv\" (UniqueName: \"kubernetes.io/projected/edb07aa2-9aca-4cc8-a70d-3a5540545497-kube-api-access-5p7jv\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:26:52.960551 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:52.960552 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edb07aa2-9aca-4cc8-a70d-3a5540545497-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:26:52.960716 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:52.960562 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edb07aa2-9aca-4cc8-a70d-3a5540545497-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:26:53.064958 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:53.064900 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-0b09d3/test-model-0b09d3-kserve-64fc7fb899-wsjbz" podUID="97c8ed56-7059-4f2e-b508-cc011621264c" containerName="main" containerID="cri-o://c4fb1b55f948c5deca548a10ec362e7738079ff7970b9006833d7252e096803a" gracePeriod=45 Oct 30 17:26:53.319355 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:53.319328 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0b09d3/test-model-0b09d3-kserve-64fc7fb899-wsjbz" Oct 30 17:26:53.364694 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:53.364662 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97c8ed56-7059-4f2e-b508-cc011621264c-tls-certs\" (UniqueName: \"kubernetes.io/secret/97c8ed56-7059-4f2e-b508-cc011621264c-tls-certs\") pod \"97c8ed56-7059-4f2e-b508-cc011621264c\" (UID: \"97c8ed56-7059-4f2e-b508-cc011621264c\") " Oct 30 17:26:53.364894 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:53.364741 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97c8ed56-7059-4f2e-b508-cc011621264c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/97c8ed56-7059-4f2e-b508-cc011621264c-dshm\") pod \"97c8ed56-7059-4f2e-b508-cc011621264c\" (UID: \"97c8ed56-7059-4f2e-b508-cc011621264c\") " Oct 30 17:26:53.364894 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:53.364768 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97c8ed56-7059-4f2e-b508-cc011621264c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97c8ed56-7059-4f2e-b508-cc011621264c-model-cache\") pod \"97c8ed56-7059-4f2e-b508-cc011621264c\" (UID: \"97c8ed56-7059-4f2e-b508-cc011621264c\") " Oct 30 17:26:53.364894 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:53.364863 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97c8ed56-7059-4f2e-b508-cc011621264c-kube-api-access-2s9fc\" (UniqueName: \"kubernetes.io/projected/97c8ed56-7059-4f2e-b508-cc011621264c-kube-api-access-2s9fc\") pod \"97c8ed56-7059-4f2e-b508-cc011621264c\" (UID: \"97c8ed56-7059-4f2e-b508-cc011621264c\") " Oct 30 17:26:53.365076 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:53.364927 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97c8ed56-7059-4f2e-b508-cc011621264c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97c8ed56-7059-4f2e-b508-cc011621264c-tmp-dir\") pod \"97c8ed56-7059-4f2e-b508-cc011621264c\" (UID: \"97c8ed56-7059-4f2e-b508-cc011621264c\") " Oct 30 17:26:53.365076 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:53.364984 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97c8ed56-7059-4f2e-b508-cc011621264c-home\" (UniqueName: \"kubernetes.io/empty-dir/97c8ed56-7059-4f2e-b508-cc011621264c-home\") pod \"97c8ed56-7059-4f2e-b508-cc011621264c\" (UID: \"97c8ed56-7059-4f2e-b508-cc011621264c\") " Oct 30 17:26:53.365709 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:53.365682 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97c8ed56-7059-4f2e-b508-cc011621264c-home" pod "97c8ed56-7059-4f2e-b508-cc011621264c" (UID: "97c8ed56-7059-4f2e-b508-cc011621264c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:26:53.365998 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:53.365947 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97c8ed56-7059-4f2e-b508-cc011621264c-model-cache" pod "97c8ed56-7059-4f2e-b508-cc011621264c" (UID: "97c8ed56-7059-4f2e-b508-cc011621264c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:26:53.366165 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:53.366088 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97c8ed56-7059-4f2e-b508-cc011621264c-tmp-dir" pod "97c8ed56-7059-4f2e-b508-cc011621264c" (UID: "97c8ed56-7059-4f2e-b508-cc011621264c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:26:53.370248 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:53.368382 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97c8ed56-7059-4f2e-b508-cc011621264c-tls-certs" pod "97c8ed56-7059-4f2e-b508-cc011621264c" (UID: "97c8ed56-7059-4f2e-b508-cc011621264c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:26:53.370248 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:53.370105 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97c8ed56-7059-4f2e-b508-cc011621264c-dshm" pod "97c8ed56-7059-4f2e-b508-cc011621264c" (UID: "97c8ed56-7059-4f2e-b508-cc011621264c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:26:53.370248 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:53.370219 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97c8ed56-7059-4f2e-b508-cc011621264c-kube-api-access-2s9fc" pod "97c8ed56-7059-4f2e-b508-cc011621264c" (UID: "97c8ed56-7059-4f2e-b508-cc011621264c"). InnerVolumeSpecName "kube-api-access-2s9fc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:26:53.466968 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:53.466894 4493 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97c8ed56-7059-4f2e-b508-cc011621264c-home\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:26:53.466968 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:53.466920 4493 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97c8ed56-7059-4f2e-b508-cc011621264c-tls-certs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:26:53.466968 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:53.466930 4493 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97c8ed56-7059-4f2e-b508-cc011621264c-dshm\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:26:53.466968 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:53.466939 4493 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97c8ed56-7059-4f2e-b508-cc011621264c-model-cache\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:26:53.466968 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:53.466947 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2s9fc\" (UniqueName: \"kubernetes.io/projected/97c8ed56-7059-4f2e-b508-cc011621264c-kube-api-access-2s9fc\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:26:53.466968 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:53.466956 4493 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97c8ed56-7059-4f2e-b508-cc011621264c-tmp-dir\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:26:53.608058 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:53.608029 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-799c76-4w8kz" Oct 30 17:26:53.608538 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:53.608031 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-799c76-4w8kz" event={"ID":"edb07aa2-9aca-4cc8-a70d-3a5540545497","Type":"ContainerDied","Data":"77bc640df03c153f542e76608e207d32a457bfca8cd132141ff85ab6bf14e22a"} Oct 30 17:26:53.608538 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:53.608183 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77bc640df03c153f542e76608e207d32a457bfca8cd132141ff85ab6bf14e22a" Oct 30 17:26:53.609587 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:53.609562 4493 generic.go:356] "Generic (PLEG): container finished" podID="97c8ed56-7059-4f2e-b508-cc011621264c" containerID="c4fb1b55f948c5deca548a10ec362e7738079ff7970b9006833d7252e096803a" exitCode=0 Oct 30 17:26:53.609716 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:53.609607 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0b09d3/test-model-0b09d3-kserve-64fc7fb899-wsjbz" Oct 30 17:26:53.609716 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:53.609652 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0b09d3/test-model-0b09d3-kserve-64fc7fb899-wsjbz" event={"ID":"97c8ed56-7059-4f2e-b508-cc011621264c","Type":"ContainerDied","Data":"c4fb1b55f948c5deca548a10ec362e7738079ff7970b9006833d7252e096803a"} Oct 30 17:26:53.609716 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:53.609686 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0b09d3/test-model-0b09d3-kserve-64fc7fb899-wsjbz" event={"ID":"97c8ed56-7059-4f2e-b508-cc011621264c","Type":"ContainerDied","Data":"59f079b9609d36969129fb7cabbf3d2b0871180236622cd03a84de30d2c7e1c2"} Oct 30 17:26:53.609716 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:53.609701 4493 scope.go:122] "RemoveContainer" containerID="c4fb1b55f948c5deca548a10ec362e7738079ff7970b9006833d7252e096803a" Oct 30 17:26:53.621667 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:53.621650 4493 scope.go:122] "RemoveContainer" containerID="c4fb1b55f948c5deca548a10ec362e7738079ff7970b9006833d7252e096803a" Oct 30 17:26:53.622029 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:26:53.622009 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4fb1b55f948c5deca548a10ec362e7738079ff7970b9006833d7252e096803a\": container with ID starting with c4fb1b55f948c5deca548a10ec362e7738079ff7970b9006833d7252e096803a not found: ID does not exist" containerID="c4fb1b55f948c5deca548a10ec362e7738079ff7970b9006833d7252e096803a" Oct 30 17:26:53.622119 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:53.622043 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4fb1b55f948c5deca548a10ec362e7738079ff7970b9006833d7252e096803a"} err="failed to get container status \"c4fb1b55f948c5deca548a10ec362e7738079ff7970b9006833d7252e096803a\": rpc error: code = NotFound desc = could not find container \"c4fb1b55f948c5deca548a10ec362e7738079ff7970b9006833d7252e096803a\": container with ID starting with c4fb1b55f948c5deca548a10ec362e7738079ff7970b9006833d7252e096803a not found: ID does not exist" Oct 30 17:26:53.663900 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:53.663875 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-0b09d3/test-model-0b09d3-kserve-64fc7fb899-wsjbz"] Oct 30 17:26:53.673416 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:53.673388 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-0b09d3/test-model-0b09d3-kserve-64fc7fb899-wsjbz"] Oct 30 17:26:53.827193 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:53.827173 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-420ff3-lk9n2" Oct 30 17:26:53.872540 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:53.872170 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af343e15-1b0d-483f-bc11-7cb765c29fe4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af343e15-1b0d-483f-bc11-7cb765c29fe4-maas-api-xks-ca\") pod \"af343e15-1b0d-483f-bc11-7cb765c29fe4\" (UID: \"af343e15-1b0d-483f-bc11-7cb765c29fe4\") " Oct 30 17:26:53.872540 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:53.872245 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af343e15-1b0d-483f-bc11-7cb765c29fe4-kube-api-access-9l9x7\" (UniqueName: \"kubernetes.io/projected/af343e15-1b0d-483f-bc11-7cb765c29fe4-kube-api-access-9l9x7\") pod \"af343e15-1b0d-483f-bc11-7cb765c29fe4\" (UID: \"af343e15-1b0d-483f-bc11-7cb765c29fe4\") " Oct 30 17:26:53.872540 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:53.872427 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af343e15-1b0d-483f-bc11-7cb765c29fe4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af343e15-1b0d-483f-bc11-7cb765c29fe4-maas-api-openshift-ca\") pod \"af343e15-1b0d-483f-bc11-7cb765c29fe4\" (UID: \"af343e15-1b0d-483f-bc11-7cb765c29fe4\") " Oct 30 17:26:53.873282 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:53.873252 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af343e15-1b0d-483f-bc11-7cb765c29fe4-maas-api-openshift-ca" pod "af343e15-1b0d-483f-bc11-7cb765c29fe4" (UID: "af343e15-1b0d-483f-bc11-7cb765c29fe4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:26:53.875457 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:53.875425 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af343e15-1b0d-483f-bc11-7cb765c29fe4-maas-api-xks-ca" pod "af343e15-1b0d-483f-bc11-7cb765c29fe4" (UID: "af343e15-1b0d-483f-bc11-7cb765c29fe4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:26:53.878370 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:53.877201 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af343e15-1b0d-483f-bc11-7cb765c29fe4-kube-api-access-9l9x7" pod "af343e15-1b0d-483f-bc11-7cb765c29fe4" (UID: "af343e15-1b0d-483f-bc11-7cb765c29fe4"). InnerVolumeSpecName "kube-api-access-9l9x7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:26:53.974700 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:53.974610 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af343e15-1b0d-483f-bc11-7cb765c29fe4-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:26:53.974700 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:53.974646 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af343e15-1b0d-483f-bc11-7cb765c29fe4-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:26:53.974700 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:53.974663 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9l9x7\" (UniqueName: \"kubernetes.io/projected/af343e15-1b0d-483f-bc11-7cb765c29fe4-kube-api-access-9l9x7\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:26:54.003451 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:54.003430 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz" Oct 30 17:26:54.075911 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:54.075876 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/009a69cb-8246-4711-a31d-37d9ee856749-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/009a69cb-8246-4711-a31d-37d9ee856749-workload-certs\") pod \"009a69cb-8246-4711-a31d-37d9ee856749\" (UID: \"009a69cb-8246-4711-a31d-37d9ee856749\") " Oct 30 17:26:54.076094 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:54.075946 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/009a69cb-8246-4711-a31d-37d9ee856749-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/009a69cb-8246-4711-a31d-37d9ee856749-istio-envoy\") pod \"009a69cb-8246-4711-a31d-37d9ee856749\" (UID: \"009a69cb-8246-4711-a31d-37d9ee856749\") " Oct 30 17:26:54.076094 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:54.076003 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/009a69cb-8246-4711-a31d-37d9ee856749-kube-api-access-vckfw\" (UniqueName: \"kubernetes.io/projected/009a69cb-8246-4711-a31d-37d9ee856749-kube-api-access-vckfw\") pod \"009a69cb-8246-4711-a31d-37d9ee856749\" (UID: \"009a69cb-8246-4711-a31d-37d9ee856749\") " Oct 30 17:26:54.076094 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:54.076079 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/009a69cb-8246-4711-a31d-37d9ee856749-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/009a69cb-8246-4711-a31d-37d9ee856749-istiod-ca-cert\") pod \"009a69cb-8246-4711-a31d-37d9ee856749\" (UID: \"009a69cb-8246-4711-a31d-37d9ee856749\") " Oct 30 17:26:54.076240 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:54.076150 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/009a69cb-8246-4711-a31d-37d9ee856749-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/009a69cb-8246-4711-a31d-37d9ee856749-istio-podinfo\") pod \"009a69cb-8246-4711-a31d-37d9ee856749\" (UID: \"009a69cb-8246-4711-a31d-37d9ee856749\") " Oct 30 17:26:54.076240 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:54.076198 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/009a69cb-8246-4711-a31d-37d9ee856749-istio-token\" (UniqueName: \"kubernetes.io/projected/009a69cb-8246-4711-a31d-37d9ee856749-istio-token\") pod \"009a69cb-8246-4711-a31d-37d9ee856749\" (UID: \"009a69cb-8246-4711-a31d-37d9ee856749\") " Oct 30 17:26:54.076240 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:54.076198 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/009a69cb-8246-4711-a31d-37d9ee856749-workload-certs" pod "009a69cb-8246-4711-a31d-37d9ee856749" (UID: "009a69cb-8246-4711-a31d-37d9ee856749"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:26:54.076358 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:54.076238 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/009a69cb-8246-4711-a31d-37d9ee856749-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/009a69cb-8246-4711-a31d-37d9ee856749-istio-data\") pod \"009a69cb-8246-4711-a31d-37d9ee856749\" (UID: \"009a69cb-8246-4711-a31d-37d9ee856749\") " Oct 30 17:26:54.076358 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:54.076303 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/009a69cb-8246-4711-a31d-37d9ee856749-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/009a69cb-8246-4711-a31d-37d9ee856749-credential-socket\") pod \"009a69cb-8246-4711-a31d-37d9ee856749\" (UID: \"009a69cb-8246-4711-a31d-37d9ee856749\") " Oct 30 17:26:54.076435 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:54.076365 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/009a69cb-8246-4711-a31d-37d9ee856749-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/009a69cb-8246-4711-a31d-37d9ee856749-workload-socket\") pod \"009a69cb-8246-4711-a31d-37d9ee856749\" (UID: \"009a69cb-8246-4711-a31d-37d9ee856749\") " Oct 30 17:26:54.076522 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:54.076502 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/009a69cb-8246-4711-a31d-37d9ee856749-istiod-ca-cert" pod "009a69cb-8246-4711-a31d-37d9ee856749" (UID: "009a69cb-8246-4711-a31d-37d9ee856749"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:26:54.076817 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:54.076761 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/009a69cb-8246-4711-a31d-37d9ee856749-workload-socket" pod "009a69cb-8246-4711-a31d-37d9ee856749" (UID: "009a69cb-8246-4711-a31d-37d9ee856749"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:26:54.076942 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:54.076894 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/009a69cb-8246-4711-a31d-37d9ee856749-istio-data" pod "009a69cb-8246-4711-a31d-37d9ee856749" (UID: "009a69cb-8246-4711-a31d-37d9ee856749"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:26:54.077033 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:54.077012 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/009a69cb-8246-4711-a31d-37d9ee856749-credential-socket" pod "009a69cb-8246-4711-a31d-37d9ee856749" (UID: "009a69cb-8246-4711-a31d-37d9ee856749"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:26:54.077255 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:54.077050 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/009a69cb-8246-4711-a31d-37d9ee856749-istio-data\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:26:54.077255 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:54.077068 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/009a69cb-8246-4711-a31d-37d9ee856749-workload-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:26:54.077255 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:54.077088 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/009a69cb-8246-4711-a31d-37d9ee856749-workload-certs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:26:54.077255 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:54.077104 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/009a69cb-8246-4711-a31d-37d9ee856749-istiod-ca-cert\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:26:54.078865 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:54.078837 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/009a69cb-8246-4711-a31d-37d9ee856749-istio-token" pod "009a69cb-8246-4711-a31d-37d9ee856749" (UID: "009a69cb-8246-4711-a31d-37d9ee856749"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:26:54.078957 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:54.078895 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/009a69cb-8246-4711-a31d-37d9ee856749-istio-podinfo" pod "009a69cb-8246-4711-a31d-37d9ee856749" (UID: "009a69cb-8246-4711-a31d-37d9ee856749"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:26:54.079377 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:54.079363 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/009a69cb-8246-4711-a31d-37d9ee856749-kube-api-access-vckfw" pod "009a69cb-8246-4711-a31d-37d9ee856749" (UID: "009a69cb-8246-4711-a31d-37d9ee856749"). InnerVolumeSpecName "kube-api-access-vckfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:26:54.079431 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:54.079412 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/009a69cb-8246-4711-a31d-37d9ee856749-istio-envoy" pod "009a69cb-8246-4711-a31d-37d9ee856749" (UID: "009a69cb-8246-4711-a31d-37d9ee856749"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:26:54.177860 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:54.177821 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/009a69cb-8246-4711-a31d-37d9ee856749-credential-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:26:54.177860 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:54.177859 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/009a69cb-8246-4711-a31d-37d9ee856749-istio-envoy\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:26:54.178079 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:54.177874 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vckfw\" (UniqueName: \"kubernetes.io/projected/009a69cb-8246-4711-a31d-37d9ee856749-kube-api-access-vckfw\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:26:54.178079 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:54.177892 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/009a69cb-8246-4711-a31d-37d9ee856749-istio-podinfo\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:26:54.178079 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:54.177901 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/009a69cb-8246-4711-a31d-37d9ee856749-istio-token\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:26:54.619463 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:54.619437 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-420ff3-lk9n2" Oct 30 17:26:54.620007 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:54.619463 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-420ff3-lk9n2" event={"ID":"af343e15-1b0d-483f-bc11-7cb765c29fe4","Type":"ContainerDied","Data":"bfa5be18248fdc3583b75bb6b296bbde20675bc0808bf00502fcdb6733708f88"} Oct 30 17:26:54.620007 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:54.619511 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bfa5be18248fdc3583b75bb6b296bbde20675bc0808bf00502fcdb6733708f88" Oct 30 17:26:54.621508 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:54.621483 4493 generic.go:356] "Generic (PLEG): container finished" podID="009a69cb-8246-4711-a31d-37d9ee856749" containerID="95cd765793bd4b262c39ddb7f3beba27afb8b6751ab0631132e70151908bfa8f" exitCode=0 Oct 30 17:26:54.621681 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:54.621536 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz" Oct 30 17:26:54.621681 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:54.621570 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz" event={"ID":"009a69cb-8246-4711-a31d-37d9ee856749","Type":"ContainerDied","Data":"95cd765793bd4b262c39ddb7f3beba27afb8b6751ab0631132e70151908bfa8f"} Oct 30 17:26:54.621681 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:54.621615 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz" event={"ID":"009a69cb-8246-4711-a31d-37d9ee856749","Type":"ContainerDied","Data":"d08654755ce83d0707c6c5d5df7ea4886eaef4654d37a3eabaa2a80d9a21a7ac"} Oct 30 17:26:54.621681 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:54.621641 4493 scope.go:122] "RemoveContainer" containerID="95cd765793bd4b262c39ddb7f3beba27afb8b6751ab0631132e70151908bfa8f" Oct 30 17:26:54.634472 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:54.634454 4493 scope.go:122] "RemoveContainer" containerID="891f9c6a6b5517919f373e475fa945315bea74d1b1e5833dd00eea5670c0dde6" Oct 30 17:26:54.645727 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:54.645707 4493 scope.go:122] "RemoveContainer" containerID="95cd765793bd4b262c39ddb7f3beba27afb8b6751ab0631132e70151908bfa8f" Oct 30 17:26:54.646119 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:26:54.646091 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95cd765793bd4b262c39ddb7f3beba27afb8b6751ab0631132e70151908bfa8f\": container with ID starting with 95cd765793bd4b262c39ddb7f3beba27afb8b6751ab0631132e70151908bfa8f not found: ID does not exist" containerID="95cd765793bd4b262c39ddb7f3beba27afb8b6751ab0631132e70151908bfa8f" Oct 30 17:26:54.646214 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:54.646128 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95cd765793bd4b262c39ddb7f3beba27afb8b6751ab0631132e70151908bfa8f"} err="failed to get container status \"95cd765793bd4b262c39ddb7f3beba27afb8b6751ab0631132e70151908bfa8f\": rpc error: code = NotFound desc = could not find container \"95cd765793bd4b262c39ddb7f3beba27afb8b6751ab0631132e70151908bfa8f\": container with ID starting with 95cd765793bd4b262c39ddb7f3beba27afb8b6751ab0631132e70151908bfa8f not found: ID does not exist" Oct 30 17:26:54.646214 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:54.646152 4493 scope.go:122] "RemoveContainer" containerID="891f9c6a6b5517919f373e475fa945315bea74d1b1e5833dd00eea5670c0dde6" Oct 30 17:26:54.646453 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:26:54.646430 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"891f9c6a6b5517919f373e475fa945315bea74d1b1e5833dd00eea5670c0dde6\": container with ID starting with 891f9c6a6b5517919f373e475fa945315bea74d1b1e5833dd00eea5670c0dde6 not found: ID does not exist" containerID="891f9c6a6b5517919f373e475fa945315bea74d1b1e5833dd00eea5670c0dde6" Oct 30 17:26:54.646506 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:54.646454 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"891f9c6a6b5517919f373e475fa945315bea74d1b1e5833dd00eea5670c0dde6"} err="failed to get container status \"891f9c6a6b5517919f373e475fa945315bea74d1b1e5833dd00eea5670c0dde6\": rpc error: code = NotFound desc = could not find container \"891f9c6a6b5517919f373e475fa945315bea74d1b1e5833dd00eea5670c0dde6\": container with ID starting with 891f9c6a6b5517919f373e475fa945315bea74d1b1e5833dd00eea5670c0dde6 not found: ID does not exist" Oct 30 17:26:54.674585 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:54.674563 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz"] Oct 30 17:26:54.688387 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:54.688362 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-43197b-openshift-default-bd446dd5c-xt2jz"] Oct 30 17:26:54.942985 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:54.941971 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="009a69cb-8246-4711-a31d-37d9ee856749" path="/var/lib/kubelet/pods/009a69cb-8246-4711-a31d-37d9ee856749/volumes" Oct 30 17:26:54.942985 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:54.942612 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97c8ed56-7059-4f2e-b508-cc011621264c" path="/var/lib/kubelet/pods/97c8ed56-7059-4f2e-b508-cc011621264c/volumes" Oct 30 17:26:55.015968 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:55.015944 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz" Oct 30 17:26:55.089251 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:55.089216 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/defb21aa-d0d6-48b8-ba50-4fcb900bc547-istio-token\" (UniqueName: \"kubernetes.io/projected/defb21aa-d0d6-48b8-ba50-4fcb900bc547-istio-token\") pod \"defb21aa-d0d6-48b8-ba50-4fcb900bc547\" (UID: \"defb21aa-d0d6-48b8-ba50-4fcb900bc547\") " Oct 30 17:26:55.089414 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:55.089294 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/defb21aa-d0d6-48b8-ba50-4fcb900bc547-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/defb21aa-d0d6-48b8-ba50-4fcb900bc547-istio-envoy\") pod \"defb21aa-d0d6-48b8-ba50-4fcb900bc547\" (UID: \"defb21aa-d0d6-48b8-ba50-4fcb900bc547\") " Oct 30 17:26:55.089414 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:55.089393 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/defb21aa-d0d6-48b8-ba50-4fcb900bc547-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/defb21aa-d0d6-48b8-ba50-4fcb900bc547-istiod-ca-cert\") pod \"defb21aa-d0d6-48b8-ba50-4fcb900bc547\" (UID: \"defb21aa-d0d6-48b8-ba50-4fcb900bc547\") " Oct 30 17:26:55.089533 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:55.089478 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/defb21aa-d0d6-48b8-ba50-4fcb900bc547-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/defb21aa-d0d6-48b8-ba50-4fcb900bc547-workload-socket\") pod \"defb21aa-d0d6-48b8-ba50-4fcb900bc547\" (UID: \"defb21aa-d0d6-48b8-ba50-4fcb900bc547\") " Oct 30 17:26:55.089533 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:55.089514 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/defb21aa-d0d6-48b8-ba50-4fcb900bc547-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/defb21aa-d0d6-48b8-ba50-4fcb900bc547-credential-socket\") pod \"defb21aa-d0d6-48b8-ba50-4fcb900bc547\" (UID: \"defb21aa-d0d6-48b8-ba50-4fcb900bc547\") " Oct 30 17:26:55.089656 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:55.089564 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/defb21aa-d0d6-48b8-ba50-4fcb900bc547-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/defb21aa-d0d6-48b8-ba50-4fcb900bc547-istio-podinfo\") pod \"defb21aa-d0d6-48b8-ba50-4fcb900bc547\" (UID: \"defb21aa-d0d6-48b8-ba50-4fcb900bc547\") " Oct 30 17:26:55.089656 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:55.089597 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/defb21aa-d0d6-48b8-ba50-4fcb900bc547-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/defb21aa-d0d6-48b8-ba50-4fcb900bc547-workload-certs\") pod \"defb21aa-d0d6-48b8-ba50-4fcb900bc547\" (UID: \"defb21aa-d0d6-48b8-ba50-4fcb900bc547\") " Oct 30 17:26:55.089766 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:55.089672 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/defb21aa-d0d6-48b8-ba50-4fcb900bc547-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/defb21aa-d0d6-48b8-ba50-4fcb900bc547-istio-data\") pod \"defb21aa-d0d6-48b8-ba50-4fcb900bc547\" (UID: \"defb21aa-d0d6-48b8-ba50-4fcb900bc547\") " Oct 30 17:26:55.089766 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:55.089722 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/defb21aa-d0d6-48b8-ba50-4fcb900bc547-kube-api-access-qvbmc\" (UniqueName: \"kubernetes.io/projected/defb21aa-d0d6-48b8-ba50-4fcb900bc547-kube-api-access-qvbmc\") pod \"defb21aa-d0d6-48b8-ba50-4fcb900bc547\" (UID: \"defb21aa-d0d6-48b8-ba50-4fcb900bc547\") " Oct 30 17:26:55.089920 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:55.089817 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/defb21aa-d0d6-48b8-ba50-4fcb900bc547-istiod-ca-cert" pod "defb21aa-d0d6-48b8-ba50-4fcb900bc547" (UID: "defb21aa-d0d6-48b8-ba50-4fcb900bc547"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:26:55.090178 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:55.090052 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/defb21aa-d0d6-48b8-ba50-4fcb900bc547-workload-socket" pod "defb21aa-d0d6-48b8-ba50-4fcb900bc547" (UID: "defb21aa-d0d6-48b8-ba50-4fcb900bc547"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:26:55.090178 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:55.090072 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/defb21aa-d0d6-48b8-ba50-4fcb900bc547-workload-certs" pod "defb21aa-d0d6-48b8-ba50-4fcb900bc547" (UID: "defb21aa-d0d6-48b8-ba50-4fcb900bc547"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:26:55.090347 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:55.090221 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/defb21aa-d0d6-48b8-ba50-4fcb900bc547-istio-data" pod "defb21aa-d0d6-48b8-ba50-4fcb900bc547" (UID: "defb21aa-d0d6-48b8-ba50-4fcb900bc547"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:26:55.090347 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:55.090264 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/defb21aa-d0d6-48b8-ba50-4fcb900bc547-credential-socket" pod "defb21aa-d0d6-48b8-ba50-4fcb900bc547" (UID: "defb21aa-d0d6-48b8-ba50-4fcb900bc547"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:26:55.090541 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:55.090524 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/defb21aa-d0d6-48b8-ba50-4fcb900bc547-workload-certs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:26:55.090615 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:55.090547 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/defb21aa-d0d6-48b8-ba50-4fcb900bc547-istio-data\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:26:55.090615 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:55.090559 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/defb21aa-d0d6-48b8-ba50-4fcb900bc547-istiod-ca-cert\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:26:55.090615 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:55.090574 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/defb21aa-d0d6-48b8-ba50-4fcb900bc547-workload-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:26:55.090615 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:55.090587 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/defb21aa-d0d6-48b8-ba50-4fcb900bc547-credential-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:26:55.092221 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:55.092193 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/defb21aa-d0d6-48b8-ba50-4fcb900bc547-istio-token" pod "defb21aa-d0d6-48b8-ba50-4fcb900bc547" (UID: "defb21aa-d0d6-48b8-ba50-4fcb900bc547"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:26:55.092373 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:55.092346 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/defb21aa-d0d6-48b8-ba50-4fcb900bc547-istio-envoy" pod "defb21aa-d0d6-48b8-ba50-4fcb900bc547" (UID: "defb21aa-d0d6-48b8-ba50-4fcb900bc547"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:26:55.092656 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:55.092638 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/defb21aa-d0d6-48b8-ba50-4fcb900bc547-istio-podinfo" pod "defb21aa-d0d6-48b8-ba50-4fcb900bc547" (UID: "defb21aa-d0d6-48b8-ba50-4fcb900bc547"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:26:55.092891 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:55.092867 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/defb21aa-d0d6-48b8-ba50-4fcb900bc547-kube-api-access-qvbmc" pod "defb21aa-d0d6-48b8-ba50-4fcb900bc547" (UID: "defb21aa-d0d6-48b8-ba50-4fcb900bc547"). InnerVolumeSpecName "kube-api-access-qvbmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:26:55.155828 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:55.153822 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4"] Oct 30 17:26:55.192123 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:55.192049 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/defb21aa-d0d6-48b8-ba50-4fcb900bc547-istio-token\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:26:55.192123 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:55.192081 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/defb21aa-d0d6-48b8-ba50-4fcb900bc547-istio-envoy\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:26:55.192123 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:55.192093 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/defb21aa-d0d6-48b8-ba50-4fcb900bc547-istio-podinfo\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:26:55.192123 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:55.192105 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvbmc\" (UniqueName: \"kubernetes.io/projected/defb21aa-d0d6-48b8-ba50-4fcb900bc547-kube-api-access-qvbmc\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:26:55.637921 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:55.636905 4493 generic.go:356] "Generic (PLEG): container finished" podID="defb21aa-d0d6-48b8-ba50-4fcb900bc547" containerID="cd0d3f94c0db2d1fe7238d36caed62993ca7f437542ca6581af4ba858a26dcf8" exitCode=0 Oct 30 17:26:55.637921 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:55.637051 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz" event={"ID":"defb21aa-d0d6-48b8-ba50-4fcb900bc547","Type":"ContainerDied","Data":"cd0d3f94c0db2d1fe7238d36caed62993ca7f437542ca6581af4ba858a26dcf8"} Oct 30 17:26:55.637921 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:55.637127 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz" event={"ID":"defb21aa-d0d6-48b8-ba50-4fcb900bc547","Type":"ContainerDied","Data":"eb3eb4494aaf96fe565eead77279e47a13663f22c79358099a6668c3b1654470"} Oct 30 17:26:55.637921 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:55.637151 4493 scope.go:122] "RemoveContainer" containerID="cd0d3f94c0db2d1fe7238d36caed62993ca7f437542ca6581af4ba858a26dcf8" Oct 30 17:26:55.637921 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:55.637274 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz" Oct 30 17:26:55.654278 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:55.654254 4493 scope.go:122] "RemoveContainer" containerID="0f67a2519bbe7709e504d80e7a989283a61519544a02432f59742bce309ac05b" Oct 30 17:26:55.665472 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:55.665450 4493 scope.go:122] "RemoveContainer" containerID="cd0d3f94c0db2d1fe7238d36caed62993ca7f437542ca6581af4ba858a26dcf8" Oct 30 17:26:55.665746 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:26:55.665723 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd0d3f94c0db2d1fe7238d36caed62993ca7f437542ca6581af4ba858a26dcf8\": container with ID starting with cd0d3f94c0db2d1fe7238d36caed62993ca7f437542ca6581af4ba858a26dcf8 not found: ID does not exist" containerID="cd0d3f94c0db2d1fe7238d36caed62993ca7f437542ca6581af4ba858a26dcf8" Oct 30 17:26:55.665861 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:55.665759 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd0d3f94c0db2d1fe7238d36caed62993ca7f437542ca6581af4ba858a26dcf8"} err="failed to get container status \"cd0d3f94c0db2d1fe7238d36caed62993ca7f437542ca6581af4ba858a26dcf8\": rpc error: code = NotFound desc = could not find container \"cd0d3f94c0db2d1fe7238d36caed62993ca7f437542ca6581af4ba858a26dcf8\": container with ID starting with cd0d3f94c0db2d1fe7238d36caed62993ca7f437542ca6581af4ba858a26dcf8 not found: ID does not exist" Oct 30 17:26:55.665861 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:55.665798 4493 scope.go:122] "RemoveContainer" containerID="0f67a2519bbe7709e504d80e7a989283a61519544a02432f59742bce309ac05b" Oct 30 17:26:55.666068 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:26:55.666049 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f67a2519bbe7709e504d80e7a989283a61519544a02432f59742bce309ac05b\": container with ID starting with 0f67a2519bbe7709e504d80e7a989283a61519544a02432f59742bce309ac05b not found: ID does not exist" containerID="0f67a2519bbe7709e504d80e7a989283a61519544a02432f59742bce309ac05b" Oct 30 17:26:55.666113 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:55.666075 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f67a2519bbe7709e504d80e7a989283a61519544a02432f59742bce309ac05b"} err="failed to get container status \"0f67a2519bbe7709e504d80e7a989283a61519544a02432f59742bce309ac05b\": rpc error: code = NotFound desc = could not find container \"0f67a2519bbe7709e504d80e7a989283a61519544a02432f59742bce309ac05b\": container with ID starting with 0f67a2519bbe7709e504d80e7a989283a61519544a02432f59742bce309ac05b not found: ID does not exist" Oct 30 17:26:55.689418 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:55.689387 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz"] Oct 30 17:26:55.701362 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:55.701340 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-5758b7-openshift-default-66f5fbf89c-fsdrz"] Oct 30 17:26:56.936084 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:56.936047 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="defb21aa-d0d6-48b8-ba50-4fcb900bc547" path="/var/lib/kubelet/pods/defb21aa-d0d6-48b8-ba50-4fcb900bc547/volumes" Oct 30 17:26:58.401622 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:26:58.401583 4493 scope.go:122] "RemoveContainer" containerID="5ea80a0432b9205a6792942a4cf028de036e131eb9095b515c22ec5a1dd58111" Oct 30 17:27:00.167396 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:00.167361 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d80ebb-4rd65"] Oct 30 17:27:00.178746 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:00.178707 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d80ebb-4rd65"] Oct 30 17:27:00.343566 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:27:00.343528 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 17:27:00.897978 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:00.897943 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-420ff3-7f7b8b4bdc-6d2x4"] Oct 30 17:27:00.934474 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:00.934432 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="024df33d-2b27-4059-a0c7-0a10ff2928d1" path="/var/lib/kubelet/pods/024df33d-2b27-4059-a0c7-0a10ff2928d1/volumes" Oct 30 17:27:05.899075 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:05.899030 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-420ff3-7f7b8b4bdc-6d2x4" podUID="a6fef0c5-c804-4953-a1be-9b49291e63d5" containerName="payload-pre-processing" containerID="cri-o://f7ecc2170b4bb59f45b7186085f7a50f1981ea051891d3303e993c09cff69d3a" gracePeriod=25 Oct 30 17:27:06.119475 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:06.119436 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-deb721-2mgq9"] Oct 30 17:27:06.134660 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:06.134621 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-deb721-2mgq9"] Oct 30 17:27:06.947650 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:06.947598 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2711b00e-0aa2-4572-8bcd-41c78bddd8c2" path="/var/lib/kubelet/pods/2711b00e-0aa2-4572-8bcd-41c78bddd8c2/volumes" Oct 30 17:27:08.295036 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:08.295004 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th"] Oct 30 17:27:08.295538 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:08.295212 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th" podUID="66f731c4-e019-4f25-83d6-5c6519009981" containerName="istio-proxy" containerID="cri-o://be3349f6bb5e573c5ef3c308300d19f26166b5d8c9f8eb3800b81343985a567d" gracePeriod=30 Oct 30 17:27:08.954140 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:08.954110 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0b09d3-jzj7g"] Oct 30 17:27:08.954668 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:08.954650 4493 cpu_manager.go:414] "containerMap: removing container" podUID="009a69cb-8246-4711-a31d-37d9ee856749" containerName="istio-proxy" Oct 30 17:27:08.954668 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:08.954667 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="009a69cb-8246-4711-a31d-37d9ee856749" containerName="istio-proxy" Oct 30 17:27:08.954869 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:08.954687 4493 cpu_manager.go:414] "containerMap: removing container" podUID="edb07aa2-9aca-4cc8-a70d-3a5540545497" containerName="revoke-keys" Oct 30 17:27:08.954869 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:08.954693 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edb07aa2-9aca-4cc8-a70d-3a5540545497" containerName="revoke-keys" Oct 30 17:27:08.954869 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:08.954734 4493 cpu_manager.go:414] "containerMap: removing container" podUID="af343e15-1b0d-483f-bc11-7cb765c29fe4" containerName="revoke-keys" Oct 30 17:27:08.954869 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:08.954739 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af343e15-1b0d-483f-bc11-7cb765c29fe4" containerName="revoke-keys" Oct 30 17:27:08.954869 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:08.954761 4493 cpu_manager.go:414] "containerMap: removing container" podUID="97c8ed56-7059-4f2e-b508-cc011621264c" containerName="main" Oct 30 17:27:08.954869 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:08.954767 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97c8ed56-7059-4f2e-b508-cc011621264c" containerName="main" Oct 30 17:27:08.954869 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:08.954794 4493 cpu_manager.go:414] "containerMap: removing container" podUID="defb21aa-d0d6-48b8-ba50-4fcb900bc547" containerName="istio-proxy" Oct 30 17:27:08.954869 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:08.954800 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="defb21aa-d0d6-48b8-ba50-4fcb900bc547" containerName="istio-proxy" Oct 30 17:27:08.954869 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:08.954807 4493 cpu_manager.go:414] "containerMap: removing container" podUID="defb21aa-d0d6-48b8-ba50-4fcb900bc547" containerName="istio-proxy" Oct 30 17:27:08.954869 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:08.954812 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="defb21aa-d0d6-48b8-ba50-4fcb900bc547" containerName="istio-proxy" Oct 30 17:27:08.955177 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:08.954905 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="edb07aa2-9aca-4cc8-a70d-3a5540545497" containerName="revoke-keys" Oct 30 17:27:08.955177 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:08.954916 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="af343e15-1b0d-483f-bc11-7cb765c29fe4" containerName="revoke-keys" Oct 30 17:27:08.955177 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:08.954923 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="97c8ed56-7059-4f2e-b508-cc011621264c" containerName="main" Oct 30 17:27:08.955177 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:08.954931 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="defb21aa-d0d6-48b8-ba50-4fcb900bc547" containerName="istio-proxy" Oct 30 17:27:08.955177 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:08.954936 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="defb21aa-d0d6-48b8-ba50-4fcb900bc547" containerName="istio-proxy" Oct 30 17:27:08.955177 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:08.954943 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="009a69cb-8246-4711-a31d-37d9ee856749" containerName="istio-proxy" Oct 30 17:27:08.955177 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:08.954950 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="009a69cb-8246-4711-a31d-37d9ee856749" containerName="istio-proxy" Oct 30 17:27:08.959622 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:08.959605 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0b09d3-jzj7g" Oct 30 17:27:08.975220 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:08.975197 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 17:27:08.981429 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:08.981409 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:27:08.984150 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:08.984121 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzh68\" (UniqueName: \"kubernetes.io/projected/611d6f10-784f-488c-b68a-382987229f03-kube-api-access-jzh68\") pod \"maas-api-revoke-keys-e2e-inf-b-0b09d3-jzj7g\" (UID: \"611d6f10-784f-488c-b68a-382987229f03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0b09d3-jzj7g" Oct 30 17:27:08.984729 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:08.984706 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/611d6f10-784f-488c-b68a-382987229f03-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0b09d3-jzj7g\" (UID: \"611d6f10-784f-488c-b68a-382987229f03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0b09d3-jzj7g" Oct 30 17:27:08.984968 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:08.984937 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/611d6f10-784f-488c-b68a-382987229f03-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0b09d3-jzj7g\" (UID: \"611d6f10-784f-488c-b68a-382987229f03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0b09d3-jzj7g" Oct 30 17:27:08.987172 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:08.987148 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:27:08.987937 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:08.987905 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:27:08.991814 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:08.991772 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0b09d3-jzj7g"] Oct 30 17:27:09.086667 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:09.086631 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/611d6f10-784f-488c-b68a-382987229f03-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0b09d3-jzj7g\" (UID: \"611d6f10-784f-488c-b68a-382987229f03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0b09d3-jzj7g" Oct 30 17:27:09.086903 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:09.086682 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/611d6f10-784f-488c-b68a-382987229f03-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0b09d3-jzj7g\" (UID: \"611d6f10-784f-488c-b68a-382987229f03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0b09d3-jzj7g" Oct 30 17:27:09.086903 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:09.086761 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzh68\" (UniqueName: \"kubernetes.io/projected/611d6f10-784f-488c-b68a-382987229f03-kube-api-access-jzh68\") pod \"maas-api-revoke-keys-e2e-inf-b-0b09d3-jzj7g\" (UID: \"611d6f10-784f-488c-b68a-382987229f03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0b09d3-jzj7g" Oct 30 17:27:09.087486 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:09.087457 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/611d6f10-784f-488c-b68a-382987229f03-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0b09d3-jzj7g\" (UID: \"611d6f10-784f-488c-b68a-382987229f03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0b09d3-jzj7g" Oct 30 17:27:09.089358 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:09.089336 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/611d6f10-784f-488c-b68a-382987229f03-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0b09d3-jzj7g\" (UID: \"611d6f10-784f-488c-b68a-382987229f03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0b09d3-jzj7g" Oct 30 17:27:09.109889 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:09.109859 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzh68\" (UniqueName: \"kubernetes.io/projected/611d6f10-784f-488c-b68a-382987229f03-kube-api-access-jzh68\") pod \"maas-api-revoke-keys-e2e-inf-b-0b09d3-jzj7g\" (UID: \"611d6f10-784f-488c-b68a-382987229f03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0b09d3-jzj7g" Oct 30 17:27:09.273230 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:09.273142 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0b09d3-jzj7g" Oct 30 17:27:09.582515 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:09.582490 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0b09d3-jzj7g"] Oct 30 17:27:09.584308 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:27:09.584276 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod611d6f10_784f_488c_b68a_382987229f03.slice/crio-3e80b58ce07afd9582a6bf21b42367046f5396148c4754808ab0c31810f77f0b WatchSource:0}: Error finding container 3e80b58ce07afd9582a6bf21b42367046f5396148c4754808ab0c31810f77f0b: Status 404 returned error can't find the container with id 3e80b58ce07afd9582a6bf21b42367046f5396148c4754808ab0c31810f77f0b Oct 30 17:27:09.787557 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:09.787521 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0b09d3-jzj7g" event={"ID":"611d6f10-784f-488c-b68a-382987229f03","Type":"ContainerStarted","Data":"3e80b58ce07afd9582a6bf21b42367046f5396148c4754808ab0c31810f77f0b"} Oct 30 17:27:10.176345 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:10.176232 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" podUID="14805045-a727-41ba-8bd0-100d41f9f27d" containerName="main" containerID="cri-o://8a24eac0d5002d9945036ecbb5b8fb68b88425ede0a7b03e14601c9f617cc27f" gracePeriod=45 Oct 30 17:27:10.346296 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:10.346151 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" podUID="14805045-a727-41ba-8bd0-100d41f9f27d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": dial tcp 10.134.0.60:8000: connect: connection refused" Oct 30 17:27:10.593159 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:10.593122 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" Oct 30 17:27:10.708187 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:10.708152 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14805045-a727-41ba-8bd0-100d41f9f27d-home\" (UniqueName: \"kubernetes.io/empty-dir/14805045-a727-41ba-8bd0-100d41f9f27d-home\") pod \"14805045-a727-41ba-8bd0-100d41f9f27d\" (UID: \"14805045-a727-41ba-8bd0-100d41f9f27d\") " Oct 30 17:27:10.708385 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:10.708267 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14805045-a727-41ba-8bd0-100d41f9f27d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14805045-a727-41ba-8bd0-100d41f9f27d-model-cache\") pod \"14805045-a727-41ba-8bd0-100d41f9f27d\" (UID: \"14805045-a727-41ba-8bd0-100d41f9f27d\") " Oct 30 17:27:10.708385 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:10.708341 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14805045-a727-41ba-8bd0-100d41f9f27d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14805045-a727-41ba-8bd0-100d41f9f27d-tmp-dir\") pod \"14805045-a727-41ba-8bd0-100d41f9f27d\" (UID: \"14805045-a727-41ba-8bd0-100d41f9f27d\") " Oct 30 17:27:10.708509 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:10.708390 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14805045-a727-41ba-8bd0-100d41f9f27d-tls-certs\" (UniqueName: \"kubernetes.io/secret/14805045-a727-41ba-8bd0-100d41f9f27d-tls-certs\") pod \"14805045-a727-41ba-8bd0-100d41f9f27d\" (UID: \"14805045-a727-41ba-8bd0-100d41f9f27d\") " Oct 30 17:27:10.708509 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:10.708450 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14805045-a727-41ba-8bd0-100d41f9f27d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/14805045-a727-41ba-8bd0-100d41f9f27d-dshm\") pod \"14805045-a727-41ba-8bd0-100d41f9f27d\" (UID: \"14805045-a727-41ba-8bd0-100d41f9f27d\") " Oct 30 17:27:10.708617 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:10.708507 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14805045-a727-41ba-8bd0-100d41f9f27d-kube-api-access-dkpzd\" (UniqueName: \"kubernetes.io/projected/14805045-a727-41ba-8bd0-100d41f9f27d-kube-api-access-dkpzd\") pod \"14805045-a727-41ba-8bd0-100d41f9f27d\" (UID: \"14805045-a727-41ba-8bd0-100d41f9f27d\") " Oct 30 17:27:10.709960 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:10.709915 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14805045-a727-41ba-8bd0-100d41f9f27d-model-cache" pod "14805045-a727-41ba-8bd0-100d41f9f27d" (UID: "14805045-a727-41ba-8bd0-100d41f9f27d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:27:10.710421 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:10.710401 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14805045-a727-41ba-8bd0-100d41f9f27d-home" pod "14805045-a727-41ba-8bd0-100d41f9f27d" (UID: "14805045-a727-41ba-8bd0-100d41f9f27d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:27:10.710569 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:10.710554 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14805045-a727-41ba-8bd0-100d41f9f27d-tmp-dir" pod "14805045-a727-41ba-8bd0-100d41f9f27d" (UID: "14805045-a727-41ba-8bd0-100d41f9f27d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:27:10.712179 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:10.712139 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14805045-a727-41ba-8bd0-100d41f9f27d-tls-certs" pod "14805045-a727-41ba-8bd0-100d41f9f27d" (UID: "14805045-a727-41ba-8bd0-100d41f9f27d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:27:10.712179 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:10.712124 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14805045-a727-41ba-8bd0-100d41f9f27d-kube-api-access-dkpzd" pod "14805045-a727-41ba-8bd0-100d41f9f27d" (UID: "14805045-a727-41ba-8bd0-100d41f9f27d"). InnerVolumeSpecName "kube-api-access-dkpzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:27:10.713432 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:10.713400 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14805045-a727-41ba-8bd0-100d41f9f27d-dshm" pod "14805045-a727-41ba-8bd0-100d41f9f27d" (UID: "14805045-a727-41ba-8bd0-100d41f9f27d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:27:10.797670 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:10.797577 4493 generic.go:356] "Generic (PLEG): container finished" podID="14805045-a727-41ba-8bd0-100d41f9f27d" containerID="8a24eac0d5002d9945036ecbb5b8fb68b88425ede0a7b03e14601c9f617cc27f" exitCode=0 Oct 30 17:27:10.797670 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:10.797621 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" Oct 30 17:27:10.797961 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:10.797662 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" event={"ID":"14805045-a727-41ba-8bd0-100d41f9f27d","Type":"ContainerDied","Data":"8a24eac0d5002d9945036ecbb5b8fb68b88425ede0a7b03e14601c9f617cc27f"} Oct 30 17:27:10.797961 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:10.797713 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4" event={"ID":"14805045-a727-41ba-8bd0-100d41f9f27d","Type":"ContainerDied","Data":"a9f4e488519cdc139ad3e08c8487c3daf9e8667d4c70b44f1c5480757148da2f"} Oct 30 17:27:10.797961 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:10.797736 4493 scope.go:122] "RemoveContainer" containerID="8a24eac0d5002d9945036ecbb5b8fb68b88425ede0a7b03e14601c9f617cc27f" Oct 30 17:27:10.800213 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:10.800188 4493 generic.go:356] "Generic (PLEG): container finished" podID="611d6f10-784f-488c-b68a-382987229f03" containerID="053f951f63b71fe6a93fd205a638396f4c742337db10eb54f8d84ac9773a6f92" exitCode=0 Oct 30 17:27:10.800386 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:10.800238 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0b09d3-jzj7g" event={"ID":"611d6f10-784f-488c-b68a-382987229f03","Type":"ContainerDied","Data":"053f951f63b71fe6a93fd205a638396f4c742337db10eb54f8d84ac9773a6f92"} Oct 30 17:27:10.809966 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:10.809941 4493 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14805045-a727-41ba-8bd0-100d41f9f27d-home\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:10.810075 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:10.809967 4493 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14805045-a727-41ba-8bd0-100d41f9f27d-model-cache\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:10.810075 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:10.809981 4493 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14805045-a727-41ba-8bd0-100d41f9f27d-tmp-dir\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:10.810075 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:10.809993 4493 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14805045-a727-41ba-8bd0-100d41f9f27d-tls-certs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:10.810075 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:10.810005 4493 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14805045-a727-41ba-8bd0-100d41f9f27d-dshm\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:10.810075 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:10.810018 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkpzd\" (UniqueName: \"kubernetes.io/projected/14805045-a727-41ba-8bd0-100d41f9f27d-kube-api-access-dkpzd\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:10.812846 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:10.812827 4493 scope.go:122] "RemoveContainer" containerID="8a24eac0d5002d9945036ecbb5b8fb68b88425ede0a7b03e14601c9f617cc27f" Oct 30 17:27:10.813180 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:27:10.813153 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a24eac0d5002d9945036ecbb5b8fb68b88425ede0a7b03e14601c9f617cc27f\": container with ID starting with 8a24eac0d5002d9945036ecbb5b8fb68b88425ede0a7b03e14601c9f617cc27f not found: ID does not exist" containerID="8a24eac0d5002d9945036ecbb5b8fb68b88425ede0a7b03e14601c9f617cc27f" Oct 30 17:27:10.813241 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:10.813192 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a24eac0d5002d9945036ecbb5b8fb68b88425ede0a7b03e14601c9f617cc27f"} err="failed to get container status \"8a24eac0d5002d9945036ecbb5b8fb68b88425ede0a7b03e14601c9f617cc27f\": rpc error: code = NotFound desc = could not find container \"8a24eac0d5002d9945036ecbb5b8fb68b88425ede0a7b03e14601c9f617cc27f\": container with ID starting with 8a24eac0d5002d9945036ecbb5b8fb68b88425ede0a7b03e14601c9f617cc27f not found: ID does not exist" Oct 30 17:27:10.893050 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:10.893004 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4"] Oct 30 17:27:10.896143 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:10.896104 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-420ff3/simulator-w4-420ff3-kserve-6d994d64b6-frdh4"] Oct 30 17:27:10.935377 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:10.935346 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14805045-a727-41ba-8bd0-100d41f9f27d" path="/var/lib/kubelet/pods/14805045-a727-41ba-8bd0-100d41f9f27d/volumes" Oct 30 17:27:11.331383 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:11.331339 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-420ff3-7f7b8b4bdc-6d2x4" podUID="a6fef0c5-c804-4953-a1be-9b49291e63d5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:27:12.116626 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:12.116595 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0b09d3-jzj7g" Oct 30 17:27:12.232930 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:12.232898 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/611d6f10-784f-488c-b68a-382987229f03-kube-api-access-jzh68\" (UniqueName: \"kubernetes.io/projected/611d6f10-784f-488c-b68a-382987229f03-kube-api-access-jzh68\") pod \"611d6f10-784f-488c-b68a-382987229f03\" (UID: \"611d6f10-784f-488c-b68a-382987229f03\") " Oct 30 17:27:12.233128 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:12.233022 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/611d6f10-784f-488c-b68a-382987229f03-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/611d6f10-784f-488c-b68a-382987229f03-maas-api-openshift-ca\") pod \"611d6f10-784f-488c-b68a-382987229f03\" (UID: \"611d6f10-784f-488c-b68a-382987229f03\") " Oct 30 17:27:12.233128 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:12.233063 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/611d6f10-784f-488c-b68a-382987229f03-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/611d6f10-784f-488c-b68a-382987229f03-maas-api-xks-ca\") pod \"611d6f10-784f-488c-b68a-382987229f03\" (UID: \"611d6f10-784f-488c-b68a-382987229f03\") " Oct 30 17:27:12.233387 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:12.233362 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/611d6f10-784f-488c-b68a-382987229f03-maas-api-openshift-ca" pod "611d6f10-784f-488c-b68a-382987229f03" (UID: "611d6f10-784f-488c-b68a-382987229f03"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:27:12.233600 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:12.233580 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/611d6f10-784f-488c-b68a-382987229f03-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:12.235292 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:12.235267 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/611d6f10-784f-488c-b68a-382987229f03-maas-api-xks-ca" pod "611d6f10-784f-488c-b68a-382987229f03" (UID: "611d6f10-784f-488c-b68a-382987229f03"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:27:12.235423 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:12.235381 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/611d6f10-784f-488c-b68a-382987229f03-kube-api-access-jzh68" pod "611d6f10-784f-488c-b68a-382987229f03" (UID: "611d6f10-784f-488c-b68a-382987229f03"). InnerVolumeSpecName "kube-api-access-jzh68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:27:12.334705 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:12.334623 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/611d6f10-784f-488c-b68a-382987229f03-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:12.334705 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:12.334650 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzh68\" (UniqueName: \"kubernetes.io/projected/611d6f10-784f-488c-b68a-382987229f03-kube-api-access-jzh68\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:12.434391 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:12.434346 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-799c76-4w8kz"] Oct 30 17:27:12.440046 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:12.440017 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-799c76-4w8kz"] Oct 30 17:27:12.823522 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:12.823484 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0b09d3-jzj7g" event={"ID":"611d6f10-784f-488c-b68a-382987229f03","Type":"ContainerDied","Data":"3e80b58ce07afd9582a6bf21b42367046f5396148c4754808ab0c31810f77f0b"} Oct 30 17:27:12.823683 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:12.823539 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e80b58ce07afd9582a6bf21b42367046f5396148c4754808ab0c31810f77f0b" Oct 30 17:27:12.823683 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:12.823601 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0b09d3-jzj7g" Oct 30 17:27:12.942960 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:12.942917 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edb07aa2-9aca-4cc8-a70d-3a5540545497" path="/var/lib/kubelet/pods/edb07aa2-9aca-4cc8-a70d-3a5540545497/volumes" Oct 30 17:27:13.583510 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:13.583485 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th" Oct 30 17:27:13.650240 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:13.650208 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/66f731c4-e019-4f25-83d6-5c6519009981-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66f731c4-e019-4f25-83d6-5c6519009981-istio-podinfo\") pod \"66f731c4-e019-4f25-83d6-5c6519009981\" (UID: \"66f731c4-e019-4f25-83d6-5c6519009981\") " Oct 30 17:27:13.650425 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:13.650273 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66f731c4-e019-4f25-83d6-5c6519009981-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66f731c4-e019-4f25-83d6-5c6519009981-istio-envoy\") pod \"66f731c4-e019-4f25-83d6-5c6519009981\" (UID: \"66f731c4-e019-4f25-83d6-5c6519009981\") " Oct 30 17:27:13.650493 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:13.650472 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66f731c4-e019-4f25-83d6-5c6519009981-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66f731c4-e019-4f25-83d6-5c6519009981-workload-socket\") pod \"66f731c4-e019-4f25-83d6-5c6519009981\" (UID: \"66f731c4-e019-4f25-83d6-5c6519009981\") " Oct 30 17:27:13.650712 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:13.650549 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66f731c4-e019-4f25-83d6-5c6519009981-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66f731c4-e019-4f25-83d6-5c6519009981-istiod-ca-cert\") pod \"66f731c4-e019-4f25-83d6-5c6519009981\" (UID: \"66f731c4-e019-4f25-83d6-5c6519009981\") " Oct 30 17:27:13.650712 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:13.650603 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66f731c4-e019-4f25-83d6-5c6519009981-istio-token\" (UniqueName: \"kubernetes.io/projected/66f731c4-e019-4f25-83d6-5c6519009981-istio-token\") pod \"66f731c4-e019-4f25-83d6-5c6519009981\" (UID: \"66f731c4-e019-4f25-83d6-5c6519009981\") " Oct 30 17:27:13.650712 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:13.650708 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66f731c4-e019-4f25-83d6-5c6519009981-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66f731c4-e019-4f25-83d6-5c6519009981-credential-socket\") pod \"66f731c4-e019-4f25-83d6-5c6519009981\" (UID: \"66f731c4-e019-4f25-83d6-5c6519009981\") " Oct 30 17:27:13.650976 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:13.650768 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66f731c4-e019-4f25-83d6-5c6519009981-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66f731c4-e019-4f25-83d6-5c6519009981-workload-certs\") pod \"66f731c4-e019-4f25-83d6-5c6519009981\" (UID: \"66f731c4-e019-4f25-83d6-5c6519009981\") " Oct 30 17:27:13.650976 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:13.650845 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66f731c4-e019-4f25-83d6-5c6519009981-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66f731c4-e019-4f25-83d6-5c6519009981-istio-data\") pod \"66f731c4-e019-4f25-83d6-5c6519009981\" (UID: \"66f731c4-e019-4f25-83d6-5c6519009981\") " Oct 30 17:27:13.651252 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:13.651019 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66f731c4-e019-4f25-83d6-5c6519009981-kube-api-access-hw76x\" (UniqueName: \"kubernetes.io/projected/66f731c4-e019-4f25-83d6-5c6519009981-kube-api-access-hw76x\") pod \"66f731c4-e019-4f25-83d6-5c6519009981\" (UID: \"66f731c4-e019-4f25-83d6-5c6519009981\") " Oct 30 17:27:13.651800 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:13.651365 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66f731c4-e019-4f25-83d6-5c6519009981-workload-socket" pod "66f731c4-e019-4f25-83d6-5c6519009981" (UID: "66f731c4-e019-4f25-83d6-5c6519009981"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:27:13.651800 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:13.651566 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66f731c4-e019-4f25-83d6-5c6519009981-workload-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:13.653376 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:13.652441 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66f731c4-e019-4f25-83d6-5c6519009981-workload-certs" pod "66f731c4-e019-4f25-83d6-5c6519009981" (UID: "66f731c4-e019-4f25-83d6-5c6519009981"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:27:13.653376 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:13.652654 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66f731c4-e019-4f25-83d6-5c6519009981-istiod-ca-cert" pod "66f731c4-e019-4f25-83d6-5c6519009981" (UID: "66f731c4-e019-4f25-83d6-5c6519009981"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:27:13.653376 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:13.652688 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66f731c4-e019-4f25-83d6-5c6519009981-istio-data" pod "66f731c4-e019-4f25-83d6-5c6519009981" (UID: "66f731c4-e019-4f25-83d6-5c6519009981"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:27:13.653675 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:13.653642 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66f731c4-e019-4f25-83d6-5c6519009981-credential-socket" pod "66f731c4-e019-4f25-83d6-5c6519009981" (UID: "66f731c4-e019-4f25-83d6-5c6519009981"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:27:13.658077 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:13.658045 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66f731c4-e019-4f25-83d6-5c6519009981-istio-envoy" pod "66f731c4-e019-4f25-83d6-5c6519009981" (UID: "66f731c4-e019-4f25-83d6-5c6519009981"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:27:13.658360 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:13.658336 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66f731c4-e019-4f25-83d6-5c6519009981-kube-api-access-hw76x" pod "66f731c4-e019-4f25-83d6-5c6519009981" (UID: "66f731c4-e019-4f25-83d6-5c6519009981"). InnerVolumeSpecName "kube-api-access-hw76x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:27:13.658438 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:13.658381 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/66f731c4-e019-4f25-83d6-5c6519009981-istio-podinfo" pod "66f731c4-e019-4f25-83d6-5c6519009981" (UID: "66f731c4-e019-4f25-83d6-5c6519009981"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:27:13.661130 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:13.661112 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66f731c4-e019-4f25-83d6-5c6519009981-istio-token" pod "66f731c4-e019-4f25-83d6-5c6519009981" (UID: "66f731c4-e019-4f25-83d6-5c6519009981"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:27:13.752825 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:13.752773 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66f731c4-e019-4f25-83d6-5c6519009981-credential-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:13.752825 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:13.752823 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66f731c4-e019-4f25-83d6-5c6519009981-workload-certs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:13.753101 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:13.752838 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66f731c4-e019-4f25-83d6-5c6519009981-istio-data\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:13.753101 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:13.752848 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hw76x\" (UniqueName: \"kubernetes.io/projected/66f731c4-e019-4f25-83d6-5c6519009981-kube-api-access-hw76x\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:13.753101 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:13.752862 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66f731c4-e019-4f25-83d6-5c6519009981-istio-podinfo\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:13.753101 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:13.752876 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66f731c4-e019-4f25-83d6-5c6519009981-istio-envoy\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:13.753101 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:13.752889 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66f731c4-e019-4f25-83d6-5c6519009981-istiod-ca-cert\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:13.753101 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:13.752897 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66f731c4-e019-4f25-83d6-5c6519009981-istio-token\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:13.834657 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:13.834561 4493 generic.go:356] "Generic (PLEG): container finished" podID="66f731c4-e019-4f25-83d6-5c6519009981" containerID="be3349f6bb5e573c5ef3c308300d19f26166b5d8c9f8eb3800b81343985a567d" exitCode=0 Oct 30 17:27:13.834855 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:13.834833 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th" event={"ID":"66f731c4-e019-4f25-83d6-5c6519009981","Type":"ContainerDied","Data":"be3349f6bb5e573c5ef3c308300d19f26166b5d8c9f8eb3800b81343985a567d"} Oct 30 17:27:13.834923 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:13.834896 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th" event={"ID":"66f731c4-e019-4f25-83d6-5c6519009981","Type":"ContainerDied","Data":"09fc9c10c117f8fdf7fe1af54ff0370cd578e509e5bed13e9078519875ef6303"} Oct 30 17:27:13.834997 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:13.834927 4493 scope.go:122] "RemoveContainer" containerID="be3349f6bb5e573c5ef3c308300d19f26166b5d8c9f8eb3800b81343985a567d" Oct 30 17:27:13.835102 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:13.835084 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th" Oct 30 17:27:13.851371 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:13.851348 4493 scope.go:122] "RemoveContainer" containerID="2d82cd31282f5633e31b54b6244e1dd4e244e1150a09557c59ea70c481e25ca0" Oct 30 17:27:13.862914 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:13.862894 4493 scope.go:122] "RemoveContainer" containerID="be3349f6bb5e573c5ef3c308300d19f26166b5d8c9f8eb3800b81343985a567d" Oct 30 17:27:13.863214 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:27:13.863197 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be3349f6bb5e573c5ef3c308300d19f26166b5d8c9f8eb3800b81343985a567d\": container with ID starting with be3349f6bb5e573c5ef3c308300d19f26166b5d8c9f8eb3800b81343985a567d not found: ID does not exist" containerID="be3349f6bb5e573c5ef3c308300d19f26166b5d8c9f8eb3800b81343985a567d" Oct 30 17:27:13.863298 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:13.863227 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be3349f6bb5e573c5ef3c308300d19f26166b5d8c9f8eb3800b81343985a567d"} err="failed to get container status \"be3349f6bb5e573c5ef3c308300d19f26166b5d8c9f8eb3800b81343985a567d\": rpc error: code = NotFound desc = could not find container \"be3349f6bb5e573c5ef3c308300d19f26166b5d8c9f8eb3800b81343985a567d\": container with ID starting with be3349f6bb5e573c5ef3c308300d19f26166b5d8c9f8eb3800b81343985a567d not found: ID does not exist" Oct 30 17:27:13.863298 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:13.863249 4493 scope.go:122] "RemoveContainer" containerID="2d82cd31282f5633e31b54b6244e1dd4e244e1150a09557c59ea70c481e25ca0" Oct 30 17:27:13.863535 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:27:13.863515 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d82cd31282f5633e31b54b6244e1dd4e244e1150a09557c59ea70c481e25ca0\": container with ID starting with 2d82cd31282f5633e31b54b6244e1dd4e244e1150a09557c59ea70c481e25ca0 not found: ID does not exist" containerID="2d82cd31282f5633e31b54b6244e1dd4e244e1150a09557c59ea70c481e25ca0" Oct 30 17:27:13.863576 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:13.863542 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d82cd31282f5633e31b54b6244e1dd4e244e1150a09557c59ea70c481e25ca0"} err="failed to get container status \"2d82cd31282f5633e31b54b6244e1dd4e244e1150a09557c59ea70c481e25ca0\": rpc error: code = NotFound desc = could not find container \"2d82cd31282f5633e31b54b6244e1dd4e244e1150a09557c59ea70c481e25ca0\": container with ID starting with 2d82cd31282f5633e31b54b6244e1dd4e244e1150a09557c59ea70c481e25ca0 not found: ID does not exist" Oct 30 17:27:13.889510 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:13.889483 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th"] Oct 30 17:27:13.895721 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:13.895697 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-d80ebb-openshift-default-75c64bdcfc-wc2th"] Oct 30 17:27:14.408063 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:14.407351 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 17:27:14.935423 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:14.935387 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66f731c4-e019-4f25-83d6-5c6519009981" path="/var/lib/kubelet/pods/66f731c4-e019-4f25-83d6-5c6519009981/volumes" Oct 30 17:27:14.974578 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:14.974542 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-18519c-bsb42"] Oct 30 17:27:14.975046 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:14.975032 4493 cpu_manager.go:414] "containerMap: removing container" podUID="009a69cb-8246-4711-a31d-37d9ee856749" containerName="istio-proxy" Oct 30 17:27:14.975090 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:14.975047 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="009a69cb-8246-4711-a31d-37d9ee856749" containerName="istio-proxy" Oct 30 17:27:14.975090 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:14.975061 4493 cpu_manager.go:414] "containerMap: removing container" podUID="611d6f10-784f-488c-b68a-382987229f03" containerName="revoke-keys" Oct 30 17:27:14.975090 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:14.975066 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="611d6f10-784f-488c-b68a-382987229f03" containerName="revoke-keys" Oct 30 17:27:14.975090 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:14.975085 4493 cpu_manager.go:414] "containerMap: removing container" podUID="66f731c4-e019-4f25-83d6-5c6519009981" containerName="istio-proxy" Oct 30 17:27:14.975090 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:14.975090 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66f731c4-e019-4f25-83d6-5c6519009981" containerName="istio-proxy" Oct 30 17:27:14.975258 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:14.975101 4493 cpu_manager.go:414] "containerMap: removing container" podUID="66f731c4-e019-4f25-83d6-5c6519009981" containerName="istio-proxy" Oct 30 17:27:14.975258 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:14.975106 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66f731c4-e019-4f25-83d6-5c6519009981" containerName="istio-proxy" Oct 30 17:27:14.975258 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:14.975116 4493 cpu_manager.go:414] "containerMap: removing container" podUID="14805045-a727-41ba-8bd0-100d41f9f27d" containerName="main" Oct 30 17:27:14.975258 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:14.975120 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14805045-a727-41ba-8bd0-100d41f9f27d" containerName="main" Oct 30 17:27:14.975258 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:14.975207 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="611d6f10-784f-488c-b68a-382987229f03" containerName="revoke-keys" Oct 30 17:27:14.975258 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:14.975215 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="66f731c4-e019-4f25-83d6-5c6519009981" containerName="istio-proxy" Oct 30 17:27:14.975258 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:14.975222 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="66f731c4-e019-4f25-83d6-5c6519009981" containerName="istio-proxy" Oct 30 17:27:14.975258 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:14.975228 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="14805045-a727-41ba-8bd0-100d41f9f27d" containerName="main" Oct 30 17:27:14.980090 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:14.980068 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-18519c-bsb42" Oct 30 17:27:14.996551 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:14.996515 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:27:14.996693 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:14.996622 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 17:27:15.007909 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:15.007883 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:27:15.008050 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:15.007911 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:27:15.012811 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:15.012769 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-18519c-bsb42"] Oct 30 17:27:15.081547 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:15.081515 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqg6q\" (UniqueName: \"kubernetes.io/projected/412a5b46-0344-4a22-ba2e-fc55c7b0102a-kube-api-access-gqg6q\") pod \"maas-api-revoke-keys-e2e-worker-w2-18519c-bsb42\" (UID: \"412a5b46-0344-4a22-ba2e-fc55c7b0102a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-18519c-bsb42" Oct 30 17:27:15.081733 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:15.081586 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/412a5b46-0344-4a22-ba2e-fc55c7b0102a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-18519c-bsb42\" (UID: \"412a5b46-0344-4a22-ba2e-fc55c7b0102a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-18519c-bsb42" Oct 30 17:27:15.081733 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:15.081695 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/412a5b46-0344-4a22-ba2e-fc55c7b0102a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-18519c-bsb42\" (UID: \"412a5b46-0344-4a22-ba2e-fc55c7b0102a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-18519c-bsb42" Oct 30 17:27:15.183810 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:15.183469 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqg6q\" (UniqueName: \"kubernetes.io/projected/412a5b46-0344-4a22-ba2e-fc55c7b0102a-kube-api-access-gqg6q\") pod \"maas-api-revoke-keys-e2e-worker-w2-18519c-bsb42\" (UID: \"412a5b46-0344-4a22-ba2e-fc55c7b0102a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-18519c-bsb42" Oct 30 17:27:15.183810 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:15.183583 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/412a5b46-0344-4a22-ba2e-fc55c7b0102a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-18519c-bsb42\" (UID: \"412a5b46-0344-4a22-ba2e-fc55c7b0102a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-18519c-bsb42" Oct 30 17:27:15.183810 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:15.183655 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/412a5b46-0344-4a22-ba2e-fc55c7b0102a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-18519c-bsb42\" (UID: \"412a5b46-0344-4a22-ba2e-fc55c7b0102a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-18519c-bsb42" Oct 30 17:27:15.190555 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:15.184612 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/412a5b46-0344-4a22-ba2e-fc55c7b0102a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-18519c-bsb42\" (UID: \"412a5b46-0344-4a22-ba2e-fc55c7b0102a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-18519c-bsb42" Oct 30 17:27:15.190555 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:15.188806 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/412a5b46-0344-4a22-ba2e-fc55c7b0102a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-18519c-bsb42\" (UID: \"412a5b46-0344-4a22-ba2e-fc55c7b0102a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-18519c-bsb42" Oct 30 17:27:15.213097 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:15.213047 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqg6q\" (UniqueName: \"kubernetes.io/projected/412a5b46-0344-4a22-ba2e-fc55c7b0102a-kube-api-access-gqg6q\") pod \"maas-api-revoke-keys-e2e-worker-w2-18519c-bsb42\" (UID: \"412a5b46-0344-4a22-ba2e-fc55c7b0102a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-18519c-bsb42" Oct 30 17:27:15.294853 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:15.294821 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-18519c-bsb42" Oct 30 17:27:15.609871 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:15.609838 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-18519c-bsb42"] Oct 30 17:27:15.611754 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:27:15.611721 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod412a5b46_0344_4a22_ba2e_fc55c7b0102a.slice/crio-4fa73856812a4159cd1fb9c2add15c3e0abedc2c88d41a64c039fa338428660c WatchSource:0}: Error finding container 4fa73856812a4159cd1fb9c2add15c3e0abedc2c88d41a64c039fa338428660c: Status 404 returned error can't find the container with id 4fa73856812a4159cd1fb9c2add15c3e0abedc2c88d41a64c039fa338428660c Oct 30 17:27:15.853866 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:15.853830 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-18519c-bsb42" event={"ID":"412a5b46-0344-4a22-ba2e-fc55c7b0102a","Type":"ContainerStarted","Data":"4fa73856812a4159cd1fb9c2add15c3e0abedc2c88d41a64c039fa338428660c"} Oct 30 17:27:16.119294 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:16.119261 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fdce1e5-2rgds"] Oct 30 17:27:16.123222 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:16.123202 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fdce1e5-2rgds" Oct 30 17:27:16.145373 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:16.145346 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fdce1e5-2rgds"] Oct 30 17:27:16.163107 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:16.163071 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21f27ac6-w8frp"] Oct 30 17:27:16.173995 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:16.173965 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21f27ac6-w8frp" Oct 30 17:27:16.180282 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:16.180217 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21f27ac6-w8frp"] Oct 30 17:27:16.196696 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:16.196671 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e4debf1-371b-4670-ac5e-a60249825753-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fdce1e5-2rgds\" (UID: \"7e4debf1-371b-4670-ac5e-a60249825753\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fdce1e5-2rgds" Oct 30 17:27:16.196860 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:16.196716 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e4debf1-371b-4670-ac5e-a60249825753-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fdce1e5-2rgds\" (UID: \"7e4debf1-371b-4670-ac5e-a60249825753\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fdce1e5-2rgds" Oct 30 17:27:16.196977 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:16.196956 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhxws\" (UniqueName: \"kubernetes.io/projected/7e4debf1-371b-4670-ac5e-a60249825753-kube-api-access-rhxws\") pod \"maas-api-revoke-sub-3fdce1e5-2rgds\" (UID: \"7e4debf1-371b-4670-ac5e-a60249825753\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fdce1e5-2rgds" Oct 30 17:27:16.298398 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:16.298369 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e4debf1-371b-4670-ac5e-a60249825753-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fdce1e5-2rgds\" (UID: \"7e4debf1-371b-4670-ac5e-a60249825753\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fdce1e5-2rgds" Oct 30 17:27:16.298584 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:16.298410 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e4debf1-371b-4670-ac5e-a60249825753-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fdce1e5-2rgds\" (UID: \"7e4debf1-371b-4670-ac5e-a60249825753\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fdce1e5-2rgds" Oct 30 17:27:16.298584 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:16.298453 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/136eca69-91b5-4b17-9d60-3cd393964f39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21f27ac6-w8frp\" (UID: \"136eca69-91b5-4b17-9d60-3cd393964f39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21f27ac6-w8frp" Oct 30 17:27:16.298584 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:16.298485 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/136eca69-91b5-4b17-9d60-3cd393964f39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21f27ac6-w8frp\" (UID: \"136eca69-91b5-4b17-9d60-3cd393964f39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21f27ac6-w8frp" Oct 30 17:27:16.298776 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:16.298613 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4q7x\" (UniqueName: \"kubernetes.io/projected/136eca69-91b5-4b17-9d60-3cd393964f39-kube-api-access-n4q7x\") pod \"maas-api-revoke-sub-21f27ac6-w8frp\" (UID: \"136eca69-91b5-4b17-9d60-3cd393964f39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21f27ac6-w8frp" Oct 30 17:27:16.298776 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:16.298660 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhxws\" (UniqueName: \"kubernetes.io/projected/7e4debf1-371b-4670-ac5e-a60249825753-kube-api-access-rhxws\") pod \"maas-api-revoke-sub-3fdce1e5-2rgds\" (UID: \"7e4debf1-371b-4670-ac5e-a60249825753\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fdce1e5-2rgds" Oct 30 17:27:16.299166 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:16.299147 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e4debf1-371b-4670-ac5e-a60249825753-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fdce1e5-2rgds\" (UID: \"7e4debf1-371b-4670-ac5e-a60249825753\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fdce1e5-2rgds" Oct 30 17:27:16.300937 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:16.300921 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e4debf1-371b-4670-ac5e-a60249825753-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fdce1e5-2rgds\" (UID: \"7e4debf1-371b-4670-ac5e-a60249825753\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fdce1e5-2rgds" Oct 30 17:27:16.321826 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:16.321802 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhxws\" (UniqueName: \"kubernetes.io/projected/7e4debf1-371b-4670-ac5e-a60249825753-kube-api-access-rhxws\") pod \"maas-api-revoke-sub-3fdce1e5-2rgds\" (UID: \"7e4debf1-371b-4670-ac5e-a60249825753\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fdce1e5-2rgds" Oct 30 17:27:16.399529 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:16.399492 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/136eca69-91b5-4b17-9d60-3cd393964f39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21f27ac6-w8frp\" (UID: \"136eca69-91b5-4b17-9d60-3cd393964f39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21f27ac6-w8frp" Oct 30 17:27:16.399732 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:16.399542 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/136eca69-91b5-4b17-9d60-3cd393964f39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21f27ac6-w8frp\" (UID: \"136eca69-91b5-4b17-9d60-3cd393964f39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21f27ac6-w8frp" Oct 30 17:27:16.399732 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:16.399583 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4q7x\" (UniqueName: \"kubernetes.io/projected/136eca69-91b5-4b17-9d60-3cd393964f39-kube-api-access-n4q7x\") pod \"maas-api-revoke-sub-21f27ac6-w8frp\" (UID: \"136eca69-91b5-4b17-9d60-3cd393964f39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21f27ac6-w8frp" Oct 30 17:27:16.400326 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:16.400299 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/136eca69-91b5-4b17-9d60-3cd393964f39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21f27ac6-w8frp\" (UID: \"136eca69-91b5-4b17-9d60-3cd393964f39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21f27ac6-w8frp" Oct 30 17:27:16.402339 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:16.402240 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/136eca69-91b5-4b17-9d60-3cd393964f39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21f27ac6-w8frp\" (UID: \"136eca69-91b5-4b17-9d60-3cd393964f39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21f27ac6-w8frp" Oct 30 17:27:16.423234 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:16.423204 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4q7x\" (UniqueName: \"kubernetes.io/projected/136eca69-91b5-4b17-9d60-3cd393964f39-kube-api-access-n4q7x\") pod \"maas-api-revoke-sub-21f27ac6-w8frp\" (UID: \"136eca69-91b5-4b17-9d60-3cd393964f39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21f27ac6-w8frp" Oct 30 17:27:16.436959 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:16.436893 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fdce1e5-2rgds" Oct 30 17:27:16.492970 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:16.492583 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21f27ac6-w8frp" Oct 30 17:27:16.744641 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:16.744611 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fdce1e5-2rgds"] Oct 30 17:27:16.782859 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:27:16.782824 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e4debf1_371b_4670_ac5e_a60249825753.slice/crio-281e9710d779b53a7c6d8dd69dadf7d4a1b906d09b65fb482183273f3ece4d94 WatchSource:0}: Error finding container 281e9710d779b53a7c6d8dd69dadf7d4a1b906d09b65fb482183273f3ece4d94: Status 404 returned error can't find the container with id 281e9710d779b53a7c6d8dd69dadf7d4a1b906d09b65fb482183273f3ece4d94 Oct 30 17:27:16.824695 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:16.824667 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21f27ac6-w8frp"] Oct 30 17:27:16.825767 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:27:16.825736 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod136eca69_91b5_4b17_9d60_3cd393964f39.slice/crio-e270d341b9a919ac0fb7fd8165af51d151cd600a5593e6996e040fa38ffb897b WatchSource:0}: Error finding container e270d341b9a919ac0fb7fd8165af51d151cd600a5593e6996e040fa38ffb897b: Status 404 returned error can't find the container with id e270d341b9a919ac0fb7fd8165af51d151cd600a5593e6996e040fa38ffb897b Oct 30 17:27:16.862819 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:16.862775 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fdce1e5-2rgds" event={"ID":"7e4debf1-371b-4670-ac5e-a60249825753","Type":"ContainerStarted","Data":"281e9710d779b53a7c6d8dd69dadf7d4a1b906d09b65fb482183273f3ece4d94"} Oct 30 17:27:16.864418 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:16.864390 4493 generic.go:356] "Generic (PLEG): container finished" podID="412a5b46-0344-4a22-ba2e-fc55c7b0102a" containerID="2cd6edc29ffb76e63ccfad90463cae981b54e9811fd468f20329ff0fa495eceb" exitCode=0 Oct 30 17:27:16.864532 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:16.864456 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-18519c-bsb42" event={"ID":"412a5b46-0344-4a22-ba2e-fc55c7b0102a","Type":"ContainerDied","Data":"2cd6edc29ffb76e63ccfad90463cae981b54e9811fd468f20329ff0fa495eceb"} Oct 30 17:27:16.865833 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:16.865804 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21f27ac6-w8frp" event={"ID":"136eca69-91b5-4b17-9d60-3cd393964f39","Type":"ContainerStarted","Data":"e270d341b9a919ac0fb7fd8165af51d151cd600a5593e6996e040fa38ffb897b"} Oct 30 17:27:17.876287 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:17.876258 4493 generic.go:356] "Generic (PLEG): container finished" podID="7e4debf1-371b-4670-ac5e-a60249825753" containerID="bae571d86b29044aa01bb094e33385fdde5a5c986e112859da1330eaef8cd890" exitCode=0 Oct 30 17:27:17.876649 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:17.876347 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fdce1e5-2rgds" event={"ID":"7e4debf1-371b-4670-ac5e-a60249825753","Type":"ContainerDied","Data":"bae571d86b29044aa01bb094e33385fdde5a5c986e112859da1330eaef8cd890"} Oct 30 17:27:17.877910 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:17.877889 4493 generic.go:356] "Generic (PLEG): container finished" podID="136eca69-91b5-4b17-9d60-3cd393964f39" containerID="0829a8cd80ad9270162a38909c5e88a7be7ea207accc676e556b2e35231997b9" exitCode=0 Oct 30 17:27:17.878002 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:17.877919 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21f27ac6-w8frp" event={"ID":"136eca69-91b5-4b17-9d60-3cd393964f39","Type":"ContainerDied","Data":"0829a8cd80ad9270162a38909c5e88a7be7ea207accc676e556b2e35231997b9"} Oct 30 17:27:18.036246 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:18.036220 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-18519c-bsb42" Oct 30 17:27:18.136454 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:18.136371 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/412a5b46-0344-4a22-ba2e-fc55c7b0102a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/412a5b46-0344-4a22-ba2e-fc55c7b0102a-maas-api-openshift-ca\") pod \"412a5b46-0344-4a22-ba2e-fc55c7b0102a\" (UID: \"412a5b46-0344-4a22-ba2e-fc55c7b0102a\") " Oct 30 17:27:18.136454 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:18.136452 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/412a5b46-0344-4a22-ba2e-fc55c7b0102a-kube-api-access-gqg6q\" (UniqueName: \"kubernetes.io/projected/412a5b46-0344-4a22-ba2e-fc55c7b0102a-kube-api-access-gqg6q\") pod \"412a5b46-0344-4a22-ba2e-fc55c7b0102a\" (UID: \"412a5b46-0344-4a22-ba2e-fc55c7b0102a\") " Oct 30 17:27:18.136709 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:18.136477 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/412a5b46-0344-4a22-ba2e-fc55c7b0102a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/412a5b46-0344-4a22-ba2e-fc55c7b0102a-maas-api-xks-ca\") pod \"412a5b46-0344-4a22-ba2e-fc55c7b0102a\" (UID: \"412a5b46-0344-4a22-ba2e-fc55c7b0102a\") " Oct 30 17:27:18.136827 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:18.136801 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/412a5b46-0344-4a22-ba2e-fc55c7b0102a-maas-api-openshift-ca" pod "412a5b46-0344-4a22-ba2e-fc55c7b0102a" (UID: "412a5b46-0344-4a22-ba2e-fc55c7b0102a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:27:18.136978 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:18.136952 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/412a5b46-0344-4a22-ba2e-fc55c7b0102a-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:18.139121 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:18.139092 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/412a5b46-0344-4a22-ba2e-fc55c7b0102a-kube-api-access-gqg6q" pod "412a5b46-0344-4a22-ba2e-fc55c7b0102a" (UID: "412a5b46-0344-4a22-ba2e-fc55c7b0102a"). InnerVolumeSpecName "kube-api-access-gqg6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:27:18.139428 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:18.139401 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/412a5b46-0344-4a22-ba2e-fc55c7b0102a-maas-api-xks-ca" pod "412a5b46-0344-4a22-ba2e-fc55c7b0102a" (UID: "412a5b46-0344-4a22-ba2e-fc55c7b0102a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:27:18.238588 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:18.238554 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqg6q\" (UniqueName: \"kubernetes.io/projected/412a5b46-0344-4a22-ba2e-fc55c7b0102a-kube-api-access-gqg6q\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:18.238588 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:18.238587 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/412a5b46-0344-4a22-ba2e-fc55c7b0102a-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:18.887082 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:18.887052 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-18519c-bsb42" Oct 30 17:27:18.887600 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:18.887080 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-18519c-bsb42" event={"ID":"412a5b46-0344-4a22-ba2e-fc55c7b0102a","Type":"ContainerDied","Data":"4fa73856812a4159cd1fb9c2add15c3e0abedc2c88d41a64c039fa338428660c"} Oct 30 17:27:18.887600 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:18.887130 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4fa73856812a4159cd1fb9c2add15c3e0abedc2c88d41a64c039fa338428660c" Oct 30 17:27:19.076124 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:19.076099 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fdce1e5-2rgds" Oct 30 17:27:19.078481 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:19.078455 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21f27ac6-w8frp" Oct 30 17:27:19.251734 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:19.251700 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e4debf1-371b-4670-ac5e-a60249825753-kube-api-access-rhxws\" (UniqueName: \"kubernetes.io/projected/7e4debf1-371b-4670-ac5e-a60249825753-kube-api-access-rhxws\") pod \"7e4debf1-371b-4670-ac5e-a60249825753\" (UID: \"7e4debf1-371b-4670-ac5e-a60249825753\") " Oct 30 17:27:19.251967 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:19.251753 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/136eca69-91b5-4b17-9d60-3cd393964f39-kube-api-access-n4q7x\" (UniqueName: \"kubernetes.io/projected/136eca69-91b5-4b17-9d60-3cd393964f39-kube-api-access-n4q7x\") pod \"136eca69-91b5-4b17-9d60-3cd393964f39\" (UID: \"136eca69-91b5-4b17-9d60-3cd393964f39\") " Oct 30 17:27:19.251967 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:19.251798 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/136eca69-91b5-4b17-9d60-3cd393964f39-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/136eca69-91b5-4b17-9d60-3cd393964f39-maas-api-openshift-ca\") pod \"136eca69-91b5-4b17-9d60-3cd393964f39\" (UID: \"136eca69-91b5-4b17-9d60-3cd393964f39\") " Oct 30 17:27:19.251967 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:19.251953 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/136eca69-91b5-4b17-9d60-3cd393964f39-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/136eca69-91b5-4b17-9d60-3cd393964f39-maas-api-xks-ca\") pod \"136eca69-91b5-4b17-9d60-3cd393964f39\" (UID: \"136eca69-91b5-4b17-9d60-3cd393964f39\") " Oct 30 17:27:19.252140 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:19.251989 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e4debf1-371b-4670-ac5e-a60249825753-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e4debf1-371b-4670-ac5e-a60249825753-maas-api-xks-ca\") pod \"7e4debf1-371b-4670-ac5e-a60249825753\" (UID: \"7e4debf1-371b-4670-ac5e-a60249825753\") " Oct 30 17:27:19.252140 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:19.252055 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e4debf1-371b-4670-ac5e-a60249825753-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e4debf1-371b-4670-ac5e-a60249825753-maas-api-openshift-ca\") pod \"7e4debf1-371b-4670-ac5e-a60249825753\" (UID: \"7e4debf1-371b-4670-ac5e-a60249825753\") " Oct 30 17:27:19.252561 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:19.252308 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/136eca69-91b5-4b17-9d60-3cd393964f39-maas-api-openshift-ca" pod "136eca69-91b5-4b17-9d60-3cd393964f39" (UID: "136eca69-91b5-4b17-9d60-3cd393964f39"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:27:19.252884 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:19.252818 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e4debf1-371b-4670-ac5e-a60249825753-maas-api-openshift-ca" pod "7e4debf1-371b-4670-ac5e-a60249825753" (UID: "7e4debf1-371b-4670-ac5e-a60249825753"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:27:19.252884 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:19.252844 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/136eca69-91b5-4b17-9d60-3cd393964f39-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:19.254480 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:19.254454 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e4debf1-371b-4670-ac5e-a60249825753-kube-api-access-rhxws" pod "7e4debf1-371b-4670-ac5e-a60249825753" (UID: "7e4debf1-371b-4670-ac5e-a60249825753"). InnerVolumeSpecName "kube-api-access-rhxws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:27:19.254598 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:19.254522 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/136eca69-91b5-4b17-9d60-3cd393964f39-kube-api-access-n4q7x" pod "136eca69-91b5-4b17-9d60-3cd393964f39" (UID: "136eca69-91b5-4b17-9d60-3cd393964f39"). InnerVolumeSpecName "kube-api-access-n4q7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:27:19.255131 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:19.255107 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e4debf1-371b-4670-ac5e-a60249825753-maas-api-xks-ca" pod "7e4debf1-371b-4670-ac5e-a60249825753" (UID: "7e4debf1-371b-4670-ac5e-a60249825753"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:27:19.256154 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:19.256134 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/136eca69-91b5-4b17-9d60-3cd393964f39-maas-api-xks-ca" pod "136eca69-91b5-4b17-9d60-3cd393964f39" (UID: "136eca69-91b5-4b17-9d60-3cd393964f39"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:27:19.353829 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:19.353777 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4q7x\" (UniqueName: \"kubernetes.io/projected/136eca69-91b5-4b17-9d60-3cd393964f39-kube-api-access-n4q7x\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:19.353829 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:19.353830 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/136eca69-91b5-4b17-9d60-3cd393964f39-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:19.354038 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:19.353847 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e4debf1-371b-4670-ac5e-a60249825753-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:19.354038 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:19.353860 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e4debf1-371b-4670-ac5e-a60249825753-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:19.354038 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:19.353889 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhxws\" (UniqueName: \"kubernetes.io/projected/7e4debf1-371b-4670-ac5e-a60249825753-kube-api-access-rhxws\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:19.683257 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:19.683227 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt"] Oct 30 17:27:19.710118 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:19.710074 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh"] Oct 30 17:27:19.898045 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:19.898011 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fdce1e5-2rgds" Oct 30 17:27:19.898547 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:19.898046 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fdce1e5-2rgds" event={"ID":"7e4debf1-371b-4670-ac5e-a60249825753","Type":"ContainerDied","Data":"281e9710d779b53a7c6d8dd69dadf7d4a1b906d09b65fb482183273f3ece4d94"} Oct 30 17:27:19.898547 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:19.898096 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="281e9710d779b53a7c6d8dd69dadf7d4a1b906d09b65fb482183273f3ece4d94" Oct 30 17:27:19.899741 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:19.899714 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21f27ac6-w8frp" event={"ID":"136eca69-91b5-4b17-9d60-3cd393964f39","Type":"ContainerDied","Data":"e270d341b9a919ac0fb7fd8165af51d151cd600a5593e6996e040fa38ffb897b"} Oct 30 17:27:19.899895 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:19.899763 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e270d341b9a919ac0fb7fd8165af51d151cd600a5593e6996e040fa38ffb897b" Oct 30 17:27:19.899895 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:19.899719 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21f27ac6-w8frp" Oct 30 17:27:20.571330 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:20.571292 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-06e7f6-tvr55"] Oct 30 17:27:20.571907 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:20.571887 4493 cpu_manager.go:414] "containerMap: removing container" podUID="7e4debf1-371b-4670-ac5e-a60249825753" containerName="revoke-keys" Oct 30 17:27:20.572013 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:20.571908 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e4debf1-371b-4670-ac5e-a60249825753" containerName="revoke-keys" Oct 30 17:27:20.572013 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:20.571940 4493 cpu_manager.go:414] "containerMap: removing container" podUID="136eca69-91b5-4b17-9d60-3cd393964f39" containerName="revoke-keys" Oct 30 17:27:20.572013 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:20.571947 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="136eca69-91b5-4b17-9d60-3cd393964f39" containerName="revoke-keys" Oct 30 17:27:20.572013 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:20.571979 4493 cpu_manager.go:414] "containerMap: removing container" podUID="412a5b46-0344-4a22-ba2e-fc55c7b0102a" containerName="revoke-keys" Oct 30 17:27:20.572013 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:20.571986 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="412a5b46-0344-4a22-ba2e-fc55c7b0102a" containerName="revoke-keys" Oct 30 17:27:20.572283 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:20.572124 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="412a5b46-0344-4a22-ba2e-fc55c7b0102a" containerName="revoke-keys" Oct 30 17:27:20.572283 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:20.572144 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e4debf1-371b-4670-ac5e-a60249825753" containerName="revoke-keys" Oct 30 17:27:20.572283 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:20.572158 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="136eca69-91b5-4b17-9d60-3cd393964f39" containerName="revoke-keys" Oct 30 17:27:20.575685 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:20.575663 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-06e7f6-tvr55" Oct 30 17:27:20.587319 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:20.587295 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24b4b342-c302-4407-984b-84c7fd53680d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-06e7f6-tvr55\" (UID: \"24b4b342-c302-4407-984b-84c7fd53680d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-06e7f6-tvr55" Oct 30 17:27:20.587412 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:20.587342 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24b4b342-c302-4407-984b-84c7fd53680d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-06e7f6-tvr55\" (UID: \"24b4b342-c302-4407-984b-84c7fd53680d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-06e7f6-tvr55" Oct 30 17:27:20.587450 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:20.587426 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bxqs\" (UniqueName: \"kubernetes.io/projected/24b4b342-c302-4407-984b-84c7fd53680d-kube-api-access-6bxqs\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-06e7f6-tvr55\" (UID: \"24b4b342-c302-4407-984b-84c7fd53680d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-06e7f6-tvr55" Oct 30 17:27:20.592982 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:20.592962 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:27:20.593092 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:20.593047 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:27:20.597642 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:20.597621 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-06e7f6-tvr55"] Oct 30 17:27:20.639133 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:20.639099 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:27:20.639533 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:20.639509 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 17:27:20.689121 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:20.689084 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24b4b342-c302-4407-984b-84c7fd53680d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-06e7f6-tvr55\" (UID: \"24b4b342-c302-4407-984b-84c7fd53680d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-06e7f6-tvr55" Oct 30 17:27:20.689312 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:20.689154 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24b4b342-c302-4407-984b-84c7fd53680d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-06e7f6-tvr55\" (UID: \"24b4b342-c302-4407-984b-84c7fd53680d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-06e7f6-tvr55" Oct 30 17:27:20.689312 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:20.689223 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bxqs\" (UniqueName: \"kubernetes.io/projected/24b4b342-c302-4407-984b-84c7fd53680d-kube-api-access-6bxqs\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-06e7f6-tvr55\" (UID: \"24b4b342-c302-4407-984b-84c7fd53680d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-06e7f6-tvr55" Oct 30 17:27:20.689776 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:20.689756 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24b4b342-c302-4407-984b-84c7fd53680d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-06e7f6-tvr55\" (UID: \"24b4b342-c302-4407-984b-84c7fd53680d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-06e7f6-tvr55" Oct 30 17:27:20.691772 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:20.691749 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24b4b342-c302-4407-984b-84c7fd53680d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-06e7f6-tvr55\" (UID: \"24b4b342-c302-4407-984b-84c7fd53680d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-06e7f6-tvr55" Oct 30 17:27:20.712625 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:20.712587 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bxqs\" (UniqueName: \"kubernetes.io/projected/24b4b342-c302-4407-984b-84c7fd53680d-kube-api-access-6bxqs\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-06e7f6-tvr55\" (UID: \"24b4b342-c302-4407-984b-84c7fd53680d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-06e7f6-tvr55" Oct 30 17:27:20.889058 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:20.888983 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-06e7f6-tvr55" Oct 30 17:27:21.209371 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:21.209339 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-06e7f6-tvr55"] Oct 30 17:27:21.209851 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:27:21.209539 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24b4b342_c302_4407_984b_84c7fd53680d.slice/crio-30f9cb6e3ce7a9369ad641370e7c295c14061f384175adc8eb7a0ff78c965a58 WatchSource:0}: Error finding container 30f9cb6e3ce7a9369ad641370e7c295c14061f384175adc8eb7a0ff78c965a58: Status 404 returned error can't find the container with id 30f9cb6e3ce7a9369ad641370e7c295c14061f384175adc8eb7a0ff78c965a58 Oct 30 17:27:21.330716 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:21.330679 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-420ff3-7f7b8b4bdc-6d2x4" podUID="a6fef0c5-c804-4953-a1be-9b49291e63d5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:27:21.919086 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:21.919041 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-06e7f6-tvr55" event={"ID":"24b4b342-c302-4407-984b-84c7fd53680d","Type":"ContainerStarted","Data":"3b7896cbc4f4ce4fc9e7ba140f7ef18aa85835bdc5ee4b094684096b3b331b1c"} Oct 30 17:27:21.919256 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:21.919107 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-06e7f6-tvr55" event={"ID":"24b4b342-c302-4407-984b-84c7fd53680d","Type":"ContainerStarted","Data":"30f9cb6e3ce7a9369ad641370e7c295c14061f384175adc8eb7a0ff78c965a58"} Oct 30 17:27:21.961624 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:21.961562 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-06e7f6-tvr55" podStartSLOduration=1.961546866 podStartE2EDuration="1.961546866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:27:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:27:21.957917617 +0000 UTC m=+3869.185092721" watchObservedRunningTime="2026-10-30 17:27:21.961546866 +0000 UTC m=+3869.188721966" Oct 30 17:27:22.928749 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:22.928715 4493 generic.go:356] "Generic (PLEG): container finished" podID="24b4b342-c302-4407-984b-84c7fd53680d" containerID="3b7896cbc4f4ce4fc9e7ba140f7ef18aa85835bdc5ee4b094684096b3b331b1c" exitCode=0 Oct 30 17:27:22.932041 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:22.932012 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-06e7f6-tvr55" event={"ID":"24b4b342-c302-4407-984b-84c7fd53680d","Type":"ContainerDied","Data":"3b7896cbc4f4ce4fc9e7ba140f7ef18aa85835bdc5ee4b094684096b3b331b1c"} Oct 30 17:27:24.110163 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:24.110132 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-06e7f6-tvr55" Oct 30 17:27:24.129003 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:24.128974 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24b4b342-c302-4407-984b-84c7fd53680d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24b4b342-c302-4407-984b-84c7fd53680d-maas-api-openshift-ca\") pod \"24b4b342-c302-4407-984b-84c7fd53680d\" (UID: \"24b4b342-c302-4407-984b-84c7fd53680d\") " Oct 30 17:27:24.129132 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:24.129015 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24b4b342-c302-4407-984b-84c7fd53680d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24b4b342-c302-4407-984b-84c7fd53680d-maas-api-xks-ca\") pod \"24b4b342-c302-4407-984b-84c7fd53680d\" (UID: \"24b4b342-c302-4407-984b-84c7fd53680d\") " Oct 30 17:27:24.129132 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:24.129094 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24b4b342-c302-4407-984b-84c7fd53680d-kube-api-access-6bxqs\" (UniqueName: \"kubernetes.io/projected/24b4b342-c302-4407-984b-84c7fd53680d-kube-api-access-6bxqs\") pod \"24b4b342-c302-4407-984b-84c7fd53680d\" (UID: \"24b4b342-c302-4407-984b-84c7fd53680d\") " Oct 30 17:27:24.129342 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:24.129315 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24b4b342-c302-4407-984b-84c7fd53680d-maas-api-openshift-ca" pod "24b4b342-c302-4407-984b-84c7fd53680d" (UID: "24b4b342-c302-4407-984b-84c7fd53680d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:27:24.129483 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:24.129466 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24b4b342-c302-4407-984b-84c7fd53680d-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:24.131189 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:24.131154 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24b4b342-c302-4407-984b-84c7fd53680d-maas-api-xks-ca" pod "24b4b342-c302-4407-984b-84c7fd53680d" (UID: "24b4b342-c302-4407-984b-84c7fd53680d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:27:24.131523 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:24.131502 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24b4b342-c302-4407-984b-84c7fd53680d-kube-api-access-6bxqs" pod "24b4b342-c302-4407-984b-84c7fd53680d" (UID: "24b4b342-c302-4407-984b-84c7fd53680d"). InnerVolumeSpecName "kube-api-access-6bxqs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:27:24.230383 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:24.230296 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bxqs\" (UniqueName: \"kubernetes.io/projected/24b4b342-c302-4407-984b-84c7fd53680d-kube-api-access-6bxqs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:24.230383 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:24.230331 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24b4b342-c302-4407-984b-84c7fd53680d-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:24.948219 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:24.948193 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-06e7f6-tvr55" Oct 30 17:27:24.948365 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:24.948218 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-06e7f6-tvr55" event={"ID":"24b4b342-c302-4407-984b-84c7fd53680d","Type":"ContainerDied","Data":"30f9cb6e3ce7a9369ad641370e7c295c14061f384175adc8eb7a0ff78c965a58"} Oct 30 17:27:24.948365 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:24.948268 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30f9cb6e3ce7a9369ad641370e7c295c14061f384175adc8eb7a0ff78c965a58" Oct 30 17:27:25.881018 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:25.880984 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16140560-b8jfg"] Oct 30 17:27:25.881725 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:25.881703 4493 cpu_manager.go:414] "containerMap: removing container" podUID="24b4b342-c302-4407-984b-84c7fd53680d" containerName="revoke-keys" Oct 30 17:27:25.881872 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:25.881727 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24b4b342-c302-4407-984b-84c7fd53680d" containerName="revoke-keys" Oct 30 17:27:25.881972 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:25.881956 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="24b4b342-c302-4407-984b-84c7fd53680d" containerName="revoke-keys" Oct 30 17:27:25.885768 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:25.885746 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16140560-b8jfg" Oct 30 17:27:25.904706 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:25.904684 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f188440c-qgjw7"] Oct 30 17:27:25.909308 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:25.909289 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f188440c-qgjw7" Oct 30 17:27:25.912027 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:25.912005 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:27:25.912211 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:25.912092 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 17:27:25.912211 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:25.912167 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:27:25.914697 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:25.914674 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:27:25.920590 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:25.920565 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16140560-b8jfg"] Oct 30 17:27:25.930860 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:25.930836 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f188440c-qgjw7"] Oct 30 17:27:25.950592 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:25.950559 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6157f678-6cf7-4cc4-a4db-20c32afeb714-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f188440c-qgjw7\" (UID: \"6157f678-6cf7-4cc4-a4db-20c32afeb714\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f188440c-qgjw7" Oct 30 17:27:25.950729 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:25.950658 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl858\" (UniqueName: \"kubernetes.io/projected/6157f678-6cf7-4cc4-a4db-20c32afeb714-kube-api-access-hl858\") pod \"maas-api-revoke-sub-f188440c-qgjw7\" (UID: \"6157f678-6cf7-4cc4-a4db-20c32afeb714\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f188440c-qgjw7" Oct 30 17:27:25.950729 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:25.950720 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36ad9481-025e-488d-aa35-8fc61f4c3034-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16140560-b8jfg\" (UID: \"36ad9481-025e-488d-aa35-8fc61f4c3034\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16140560-b8jfg" Oct 30 17:27:25.950882 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:25.950748 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36ad9481-025e-488d-aa35-8fc61f4c3034-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16140560-b8jfg\" (UID: \"36ad9481-025e-488d-aa35-8fc61f4c3034\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16140560-b8jfg" Oct 30 17:27:25.950882 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:25.950838 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hspm9\" (UniqueName: \"kubernetes.io/projected/36ad9481-025e-488d-aa35-8fc61f4c3034-kube-api-access-hspm9\") pod \"maas-api-revoke-sub-16140560-b8jfg\" (UID: \"36ad9481-025e-488d-aa35-8fc61f4c3034\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16140560-b8jfg" Oct 30 17:27:25.950958 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:25.950891 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6157f678-6cf7-4cc4-a4db-20c32afeb714-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f188440c-qgjw7\" (UID: \"6157f678-6cf7-4cc4-a4db-20c32afeb714\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f188440c-qgjw7" Oct 30 17:27:26.068576 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:26.068527 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6157f678-6cf7-4cc4-a4db-20c32afeb714-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f188440c-qgjw7\" (UID: \"6157f678-6cf7-4cc4-a4db-20c32afeb714\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f188440c-qgjw7" Oct 30 17:27:26.068866 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:26.068609 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl858\" (UniqueName: \"kubernetes.io/projected/6157f678-6cf7-4cc4-a4db-20c32afeb714-kube-api-access-hl858\") pod \"maas-api-revoke-sub-f188440c-qgjw7\" (UID: \"6157f678-6cf7-4cc4-a4db-20c32afeb714\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f188440c-qgjw7" Oct 30 17:27:26.068866 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:26.068688 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36ad9481-025e-488d-aa35-8fc61f4c3034-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16140560-b8jfg\" (UID: \"36ad9481-025e-488d-aa35-8fc61f4c3034\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16140560-b8jfg" Oct 30 17:27:26.068866 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:26.068732 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36ad9481-025e-488d-aa35-8fc61f4c3034-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16140560-b8jfg\" (UID: \"36ad9481-025e-488d-aa35-8fc61f4c3034\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16140560-b8jfg" Oct 30 17:27:26.068866 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:26.068762 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hspm9\" (UniqueName: \"kubernetes.io/projected/36ad9481-025e-488d-aa35-8fc61f4c3034-kube-api-access-hspm9\") pod \"maas-api-revoke-sub-16140560-b8jfg\" (UID: \"36ad9481-025e-488d-aa35-8fc61f4c3034\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16140560-b8jfg" Oct 30 17:27:26.068866 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:26.068817 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6157f678-6cf7-4cc4-a4db-20c32afeb714-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f188440c-qgjw7\" (UID: \"6157f678-6cf7-4cc4-a4db-20c32afeb714\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f188440c-qgjw7" Oct 30 17:27:26.072818 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:26.072041 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6157f678-6cf7-4cc4-a4db-20c32afeb714-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f188440c-qgjw7\" (UID: \"6157f678-6cf7-4cc4-a4db-20c32afeb714\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f188440c-qgjw7" Oct 30 17:27:26.073105 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:26.073070 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36ad9481-025e-488d-aa35-8fc61f4c3034-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16140560-b8jfg\" (UID: \"36ad9481-025e-488d-aa35-8fc61f4c3034\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16140560-b8jfg" Oct 30 17:27:26.099063 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:26.086651 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36ad9481-025e-488d-aa35-8fc61f4c3034-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16140560-b8jfg\" (UID: \"36ad9481-025e-488d-aa35-8fc61f4c3034\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16140560-b8jfg" Oct 30 17:27:26.099063 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:26.098393 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl858\" (UniqueName: \"kubernetes.io/projected/6157f678-6cf7-4cc4-a4db-20c32afeb714-kube-api-access-hl858\") pod \"maas-api-revoke-sub-f188440c-qgjw7\" (UID: \"6157f678-6cf7-4cc4-a4db-20c32afeb714\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f188440c-qgjw7" Oct 30 17:27:26.101433 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:26.101406 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hspm9\" (UniqueName: \"kubernetes.io/projected/36ad9481-025e-488d-aa35-8fc61f4c3034-kube-api-access-hspm9\") pod \"maas-api-revoke-sub-16140560-b8jfg\" (UID: \"36ad9481-025e-488d-aa35-8fc61f4c3034\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16140560-b8jfg" Oct 30 17:27:26.104926 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:26.103714 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6157f678-6cf7-4cc4-a4db-20c32afeb714-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f188440c-qgjw7\" (UID: \"6157f678-6cf7-4cc4-a4db-20c32afeb714\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f188440c-qgjw7" Oct 30 17:27:26.200342 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:26.200309 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16140560-b8jfg" Oct 30 17:27:26.245695 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:26.245363 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-420ff3-7f7b8b4bdc-6d2x4" Oct 30 17:27:26.296063 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:26.296023 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f188440c-qgjw7" Oct 30 17:27:26.372369 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:26.372338 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6fef0c5-c804-4953-a1be-9b49291e63d5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6fef0c5-c804-4953-a1be-9b49291e63d5-plugins-config-volume\") pod \"a6fef0c5-c804-4953-a1be-9b49291e63d5\" (UID: \"a6fef0c5-c804-4953-a1be-9b49291e63d5\") " Oct 30 17:27:26.372518 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:26.372395 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6fef0c5-c804-4953-a1be-9b49291e63d5-kube-api-access-fw2vg\" (UniqueName: \"kubernetes.io/projected/a6fef0c5-c804-4953-a1be-9b49291e63d5-kube-api-access-fw2vg\") pod \"a6fef0c5-c804-4953-a1be-9b49291e63d5\" (UID: \"a6fef0c5-c804-4953-a1be-9b49291e63d5\") " Oct 30 17:27:26.373613 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:26.373564 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6fef0c5-c804-4953-a1be-9b49291e63d5-plugins-config-volume" pod "a6fef0c5-c804-4953-a1be-9b49291e63d5" (UID: "a6fef0c5-c804-4953-a1be-9b49291e63d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:27:26.378607 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:26.378581 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6fef0c5-c804-4953-a1be-9b49291e63d5-kube-api-access-fw2vg" pod "a6fef0c5-c804-4953-a1be-9b49291e63d5" (UID: "a6fef0c5-c804-4953-a1be-9b49291e63d5"). InnerVolumeSpecName "kube-api-access-fw2vg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:27:26.474533 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:26.474455 4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6fef0c5-c804-4953-a1be-9b49291e63d5-plugins-config-volume\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:26.474533 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:26.474481 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fw2vg\" (UniqueName: \"kubernetes.io/projected/a6fef0c5-c804-4953-a1be-9b49291e63d5-kube-api-access-fw2vg\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:26.527219 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:26.527194 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16140560-b8jfg"] Oct 30 17:27:26.529245 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:27:26.529202 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36ad9481_025e_488d_aa35_8fc61f4c3034.slice/crio-955dc1b83c9bbeead7c244fe4609b98132599997ecdee2a90b1482c6dd97b53e WatchSource:0}: Error finding container 955dc1b83c9bbeead7c244fe4609b98132599997ecdee2a90b1482c6dd97b53e: Status 404 returned error can't find the container with id 955dc1b83c9bbeead7c244fe4609b98132599997ecdee2a90b1482c6dd97b53e Oct 30 17:27:26.612288 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:26.612260 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f188440c-qgjw7"] Oct 30 17:27:26.706355 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:27:26.706320 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6157f678_6cf7_4cc4_a4db_20c32afeb714.slice/crio-5eb58bcf3d346b7554404e0cd4cecf9e08388a38981b879dd8b437a8b22da9ae WatchSource:0}: Error finding container 5eb58bcf3d346b7554404e0cd4cecf9e08388a38981b879dd8b437a8b22da9ae: Status 404 returned error can't find the container with id 5eb58bcf3d346b7554404e0cd4cecf9e08388a38981b879dd8b437a8b22da9ae Oct 30 17:27:26.965181 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:26.965147 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16140560-b8jfg" event={"ID":"36ad9481-025e-488d-aa35-8fc61f4c3034","Type":"ContainerStarted","Data":"769a3371f7b30ea329e213e22b565e42a23e715b8ce7858b0c4726c40fa82e52"} Oct 30 17:27:26.965595 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:26.965201 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16140560-b8jfg" event={"ID":"36ad9481-025e-488d-aa35-8fc61f4c3034","Type":"ContainerStarted","Data":"955dc1b83c9bbeead7c244fe4609b98132599997ecdee2a90b1482c6dd97b53e"} Oct 30 17:27:26.966697 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:26.966672 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f188440c-qgjw7" event={"ID":"6157f678-6cf7-4cc4-a4db-20c32afeb714","Type":"ContainerStarted","Data":"5eb58bcf3d346b7554404e0cd4cecf9e08388a38981b879dd8b437a8b22da9ae"} Oct 30 17:27:26.968113 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:26.968090 4493 generic.go:356] "Generic (PLEG): container finished" podID="a6fef0c5-c804-4953-a1be-9b49291e63d5" containerID="f7ecc2170b4bb59f45b7186085f7a50f1981ea051891d3303e993c09cff69d3a" exitCode=0 Oct 30 17:27:26.968225 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:26.968129 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-420ff3-7f7b8b4bdc-6d2x4" Oct 30 17:27:26.968225 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:26.968185 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-420ff3-7f7b8b4bdc-6d2x4" event={"ID":"a6fef0c5-c804-4953-a1be-9b49291e63d5","Type":"ContainerDied","Data":"f7ecc2170b4bb59f45b7186085f7a50f1981ea051891d3303e993c09cff69d3a"} Oct 30 17:27:26.968311 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:26.968244 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-420ff3-7f7b8b4bdc-6d2x4" event={"ID":"a6fef0c5-c804-4953-a1be-9b49291e63d5","Type":"ContainerDied","Data":"26f54ce07c633b7a62471943ae59b69a1d6c89633a68a527edf8cab623a0a342"} Oct 30 17:27:26.968311 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:26.968274 4493 scope.go:122] "RemoveContainer" containerID="f7ecc2170b4bb59f45b7186085f7a50f1981ea051891d3303e993c09cff69d3a" Oct 30 17:27:26.979214 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:26.979189 4493 scope.go:122] "RemoveContainer" containerID="f7ecc2170b4bb59f45b7186085f7a50f1981ea051891d3303e993c09cff69d3a" Oct 30 17:27:26.979512 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:27:26.979491 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7ecc2170b4bb59f45b7186085f7a50f1981ea051891d3303e993c09cff69d3a\": container with ID starting with f7ecc2170b4bb59f45b7186085f7a50f1981ea051891d3303e993c09cff69d3a not found: ID does not exist" containerID="f7ecc2170b4bb59f45b7186085f7a50f1981ea051891d3303e993c09cff69d3a" Oct 30 17:27:26.979588 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:26.979519 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7ecc2170b4bb59f45b7186085f7a50f1981ea051891d3303e993c09cff69d3a"} err="failed to get container status \"f7ecc2170b4bb59f45b7186085f7a50f1981ea051891d3303e993c09cff69d3a\": rpc error: code = NotFound desc = could not find container \"f7ecc2170b4bb59f45b7186085f7a50f1981ea051891d3303e993c09cff69d3a\": container with ID starting with f7ecc2170b4bb59f45b7186085f7a50f1981ea051891d3303e993c09cff69d3a not found: ID does not exist" Oct 30 17:27:27.011968 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:27.011911 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16140560-b8jfg" podStartSLOduration=2.011898184 podStartE2EDuration="2.011898184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:27:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:27:27.00810926 +0000 UTC m=+3874.235284359" watchObservedRunningTime="2026-10-30 17:27:27.011898184 +0000 UTC m=+3874.239073284" Oct 30 17:27:27.046543 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:27.046512 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-420ff3-7f7b8b4bdc-6d2x4"] Oct 30 17:27:27.054317 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:27.053262 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-420ff3-7f7b8b4bdc-6d2x4"] Oct 30 17:27:27.977492 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:27.977452 4493 generic.go:356] "Generic (PLEG): container finished" podID="36ad9481-025e-488d-aa35-8fc61f4c3034" containerID="769a3371f7b30ea329e213e22b565e42a23e715b8ce7858b0c4726c40fa82e52" exitCode=0 Oct 30 17:27:27.977977 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:27.977539 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16140560-b8jfg" event={"ID":"36ad9481-025e-488d-aa35-8fc61f4c3034","Type":"ContainerDied","Data":"769a3371f7b30ea329e213e22b565e42a23e715b8ce7858b0c4726c40fa82e52"} Oct 30 17:27:27.979557 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:27.979533 4493 generic.go:356] "Generic (PLEG): container finished" podID="6157f678-6cf7-4cc4-a4db-20c32afeb714" containerID="2683dc3c475c565caa8abd507364a08c2cad81194462a17847883a23b5928425" exitCode=0 Oct 30 17:27:27.979682 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:27.979645 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f188440c-qgjw7" event={"ID":"6157f678-6cf7-4cc4-a4db-20c32afeb714","Type":"ContainerDied","Data":"2683dc3c475c565caa8abd507364a08c2cad81194462a17847883a23b5928425"} Oct 30 17:27:28.933363 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:28.933320 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6fef0c5-c804-4953-a1be-9b49291e63d5" path="/var/lib/kubelet/pods/a6fef0c5-c804-4953-a1be-9b49291e63d5/volumes" Oct 30 17:27:29.184102 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:29.184041 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f188440c-qgjw7" Oct 30 17:27:29.186125 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:29.186107 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16140560-b8jfg" Oct 30 17:27:29.309347 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:29.309308 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36ad9481-025e-488d-aa35-8fc61f4c3034-kube-api-access-hspm9\" (UniqueName: \"kubernetes.io/projected/36ad9481-025e-488d-aa35-8fc61f4c3034-kube-api-access-hspm9\") pod \"36ad9481-025e-488d-aa35-8fc61f4c3034\" (UID: \"36ad9481-025e-488d-aa35-8fc61f4c3034\") " Oct 30 17:27:29.309347 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:29.309359 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6157f678-6cf7-4cc4-a4db-20c32afeb714-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6157f678-6cf7-4cc4-a4db-20c32afeb714-maas-api-xks-ca\") pod \"6157f678-6cf7-4cc4-a4db-20c32afeb714\" (UID: \"6157f678-6cf7-4cc4-a4db-20c32afeb714\") " Oct 30 17:27:29.309615 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:29.309461 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6157f678-6cf7-4cc4-a4db-20c32afeb714-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6157f678-6cf7-4cc4-a4db-20c32afeb714-maas-api-openshift-ca\") pod \"6157f678-6cf7-4cc4-a4db-20c32afeb714\" (UID: \"6157f678-6cf7-4cc4-a4db-20c32afeb714\") " Oct 30 17:27:29.309615 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:29.309553 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6157f678-6cf7-4cc4-a4db-20c32afeb714-kube-api-access-hl858\" (UniqueName: \"kubernetes.io/projected/6157f678-6cf7-4cc4-a4db-20c32afeb714-kube-api-access-hl858\") pod \"6157f678-6cf7-4cc4-a4db-20c32afeb714\" (UID: \"6157f678-6cf7-4cc4-a4db-20c32afeb714\") " Oct 30 17:27:29.309615 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:29.309590 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36ad9481-025e-488d-aa35-8fc61f4c3034-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36ad9481-025e-488d-aa35-8fc61f4c3034-maas-api-openshift-ca\") pod \"36ad9481-025e-488d-aa35-8fc61f4c3034\" (UID: \"36ad9481-025e-488d-aa35-8fc61f4c3034\") " Oct 30 17:27:29.310264 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:29.309717 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36ad9481-025e-488d-aa35-8fc61f4c3034-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36ad9481-025e-488d-aa35-8fc61f4c3034-maas-api-xks-ca\") pod \"36ad9481-025e-488d-aa35-8fc61f4c3034\" (UID: \"36ad9481-025e-488d-aa35-8fc61f4c3034\") " Oct 30 17:27:29.310940 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:29.310896 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36ad9481-025e-488d-aa35-8fc61f4c3034-maas-api-openshift-ca" pod "36ad9481-025e-488d-aa35-8fc61f4c3034" (UID: "36ad9481-025e-488d-aa35-8fc61f4c3034"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:27:29.311402 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:29.311382 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6157f678-6cf7-4cc4-a4db-20c32afeb714-maas-api-openshift-ca" pod "6157f678-6cf7-4cc4-a4db-20c32afeb714" (UID: "6157f678-6cf7-4cc4-a4db-20c32afeb714"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:27:29.312951 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:29.312818 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36ad9481-025e-488d-aa35-8fc61f4c3034-maas-api-xks-ca" pod "36ad9481-025e-488d-aa35-8fc61f4c3034" (UID: "36ad9481-025e-488d-aa35-8fc61f4c3034"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:27:29.313060 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:29.312999 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6157f678-6cf7-4cc4-a4db-20c32afeb714-maas-api-xks-ca" pod "6157f678-6cf7-4cc4-a4db-20c32afeb714" (UID: "6157f678-6cf7-4cc4-a4db-20c32afeb714"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:27:29.314851 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:29.314723 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6157f678-6cf7-4cc4-a4db-20c32afeb714-kube-api-access-hl858" pod "6157f678-6cf7-4cc4-a4db-20c32afeb714" (UID: "6157f678-6cf7-4cc4-a4db-20c32afeb714"). InnerVolumeSpecName "kube-api-access-hl858". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:27:29.315182 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:29.315158 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36ad9481-025e-488d-aa35-8fc61f4c3034-kube-api-access-hspm9" pod "36ad9481-025e-488d-aa35-8fc61f4c3034" (UID: "36ad9481-025e-488d-aa35-8fc61f4c3034"). InnerVolumeSpecName "kube-api-access-hspm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:27:29.411819 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:29.411749 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6157f678-6cf7-4cc4-a4db-20c32afeb714-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:29.411819 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:29.411814 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hl858\" (UniqueName: \"kubernetes.io/projected/6157f678-6cf7-4cc4-a4db-20c32afeb714-kube-api-access-hl858\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:29.412085 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:29.411831 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36ad9481-025e-488d-aa35-8fc61f4c3034-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:29.412085 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:29.411846 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36ad9481-025e-488d-aa35-8fc61f4c3034-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:29.412085 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:29.411859 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hspm9\" (UniqueName: \"kubernetes.io/projected/36ad9481-025e-488d-aa35-8fc61f4c3034-kube-api-access-hspm9\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:29.412085 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:29.411870 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6157f678-6cf7-4cc4-a4db-20c32afeb714-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:30.002501 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:29.999446 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16140560-b8jfg" event={"ID":"36ad9481-025e-488d-aa35-8fc61f4c3034","Type":"ContainerDied","Data":"955dc1b83c9bbeead7c244fe4609b98132599997ecdee2a90b1482c6dd97b53e"} Oct 30 17:27:30.002501 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:29.999506 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="955dc1b83c9bbeead7c244fe4609b98132599997ecdee2a90b1482c6dd97b53e" Oct 30 17:27:30.002501 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:29.999578 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16140560-b8jfg" Oct 30 17:27:30.008996 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:30.004946 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f188440c-qgjw7" event={"ID":"6157f678-6cf7-4cc4-a4db-20c32afeb714","Type":"ContainerDied","Data":"5eb58bcf3d346b7554404e0cd4cecf9e08388a38981b879dd8b437a8b22da9ae"} Oct 30 17:27:30.008996 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:30.005022 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5eb58bcf3d346b7554404e0cd4cecf9e08388a38981b879dd8b437a8b22da9ae" Oct 30 17:27:30.008996 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:30.005106 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f188440c-qgjw7" Oct 30 17:27:30.343328 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:27:30.343205 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 17:27:32.681470 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:32.678834 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-06e7f6-tvr55"] Oct 30 17:27:32.684697 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:32.684573 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-06e7f6-tvr55"] Oct 30 17:27:32.933726 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:32.933648 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24b4b342-c302-4407-984b-84c7fd53680d" path="/var/lib/kubelet/pods/24b4b342-c302-4407-984b-84c7fd53680d/volumes" Oct 30 17:27:34.541898 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:34.541858 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0b09d3-jzj7g"] Oct 30 17:27:34.553916 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:34.553877 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0b09d3-jzj7g"] Oct 30 17:27:34.697808 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:34.688440 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 17:27:34.713432 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:34.713368 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" podUID="f3e1c8ff-fcc3-4a1f-9730-c617d57b5193" containerName="main" containerID="cri-o://d22765a80c71546b77c331be945ade3d585939b7efc74b5b2adb47c9815bf2d4" gracePeriod=45 Oct 30 17:27:34.746320 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:34.740002 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" podUID="80b9bd9a-30d6-45ff-b318-26fcdc608e5b" containerName="main" containerID="cri-o://e68e01ba784a12f2a8823555b0a1d6d474c9c4857fe5bfbed1d5c7d4b68e06fd" gracePeriod=45 Oct 30 17:27:34.951833 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:34.950526 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="611d6f10-784f-488c-b68a-382987229f03" path="/var/lib/kubelet/pods/611d6f10-784f-488c-b68a-382987229f03/volumes" Oct 30 17:27:35.078863 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:35.076020 4493 generic.go:356] "Generic (PLEG): container finished" podID="f3e1c8ff-fcc3-4a1f-9730-c617d57b5193" containerID="d22765a80c71546b77c331be945ade3d585939b7efc74b5b2adb47c9815bf2d4" exitCode=0 Oct 30 17:27:35.078863 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:35.076135 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" event={"ID":"f3e1c8ff-fcc3-4a1f-9730-c617d57b5193","Type":"ContainerDied","Data":"d22765a80c71546b77c331be945ade3d585939b7efc74b5b2adb47c9815bf2d4"} Oct 30 17:27:35.078863 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:35.077717 4493 generic.go:356] "Generic (PLEG): container finished" podID="80b9bd9a-30d6-45ff-b318-26fcdc608e5b" containerID="e68e01ba784a12f2a8823555b0a1d6d474c9c4857fe5bfbed1d5c7d4b68e06fd" exitCode=0 Oct 30 17:27:35.078863 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:35.077959 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" event={"ID":"80b9bd9a-30d6-45ff-b318-26fcdc608e5b","Type":"ContainerDied","Data":"e68e01ba784a12f2a8823555b0a1d6d474c9c4857fe5bfbed1d5c7d4b68e06fd"} Oct 30 17:27:35.078863 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:35.077984 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" event={"ID":"80b9bd9a-30d6-45ff-b318-26fcdc608e5b","Type":"ContainerDied","Data":"136fc8afab04aebcdb2163ec3bc7873ff6e2904d643b100b0b4ead51c4b4decd"} Oct 30 17:27:35.078863 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:35.077997 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="136fc8afab04aebcdb2163ec3bc7873ff6e2904d643b100b0b4ead51c4b4decd" Oct 30 17:27:35.084003 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:35.083977 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" Oct 30 17:27:35.088566 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:35.088542 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" Oct 30 17:27:35.159037 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:35.158997 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3e1c8ff-fcc3-4a1f-9730-c617d57b5193-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3e1c8ff-fcc3-4a1f-9730-c617d57b5193-model-cache\") pod \"f3e1c8ff-fcc3-4a1f-9730-c617d57b5193\" (UID: \"f3e1c8ff-fcc3-4a1f-9730-c617d57b5193\") " Oct 30 17:27:35.159218 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:35.159079 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3e1c8ff-fcc3-4a1f-9730-c617d57b5193-kube-api-access-hf9dq\" (UniqueName: \"kubernetes.io/projected/f3e1c8ff-fcc3-4a1f-9730-c617d57b5193-kube-api-access-hf9dq\") pod \"f3e1c8ff-fcc3-4a1f-9730-c617d57b5193\" (UID: \"f3e1c8ff-fcc3-4a1f-9730-c617d57b5193\") " Oct 30 17:27:35.159218 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:35.159131 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80b9bd9a-30d6-45ff-b318-26fcdc608e5b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80b9bd9a-30d6-45ff-b318-26fcdc608e5b-model-cache\") pod \"80b9bd9a-30d6-45ff-b318-26fcdc608e5b\" (UID: \"80b9bd9a-30d6-45ff-b318-26fcdc608e5b\") " Oct 30 17:27:35.159218 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:35.159189 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3e1c8ff-fcc3-4a1f-9730-c617d57b5193-tls-certs\" (UniqueName: \"kubernetes.io/secret/f3e1c8ff-fcc3-4a1f-9730-c617d57b5193-tls-certs\") pod \"f3e1c8ff-fcc3-4a1f-9730-c617d57b5193\" (UID: \"f3e1c8ff-fcc3-4a1f-9730-c617d57b5193\") " Oct 30 17:27:35.159415 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:35.159248 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80b9bd9a-30d6-45ff-b318-26fcdc608e5b-kube-api-access-cc4wt\" (UniqueName: \"kubernetes.io/projected/80b9bd9a-30d6-45ff-b318-26fcdc608e5b-kube-api-access-cc4wt\") pod \"80b9bd9a-30d6-45ff-b318-26fcdc608e5b\" (UID: \"80b9bd9a-30d6-45ff-b318-26fcdc608e5b\") " Oct 30 17:27:35.159415 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:35.159295 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3e1c8ff-fcc3-4a1f-9730-c617d57b5193-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3e1c8ff-fcc3-4a1f-9730-c617d57b5193-dshm\") pod \"f3e1c8ff-fcc3-4a1f-9730-c617d57b5193\" (UID: \"f3e1c8ff-fcc3-4a1f-9730-c617d57b5193\") " Oct 30 17:27:35.159415 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:35.159323 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80b9bd9a-30d6-45ff-b318-26fcdc608e5b-home\" (UniqueName: \"kubernetes.io/empty-dir/80b9bd9a-30d6-45ff-b318-26fcdc608e5b-home\") pod \"80b9bd9a-30d6-45ff-b318-26fcdc608e5b\" (UID: \"80b9bd9a-30d6-45ff-b318-26fcdc608e5b\") " Oct 30 17:27:35.159415 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:35.159331 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3e1c8ff-fcc3-4a1f-9730-c617d57b5193-model-cache" pod "f3e1c8ff-fcc3-4a1f-9730-c617d57b5193" (UID: "f3e1c8ff-fcc3-4a1f-9730-c617d57b5193"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:27:35.159415 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:35.159361 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80b9bd9a-30d6-45ff-b318-26fcdc608e5b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80b9bd9a-30d6-45ff-b318-26fcdc608e5b-tmp-dir\") pod \"80b9bd9a-30d6-45ff-b318-26fcdc608e5b\" (UID: \"80b9bd9a-30d6-45ff-b318-26fcdc608e5b\") " Oct 30 17:27:35.159683 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:35.159418 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3e1c8ff-fcc3-4a1f-9730-c617d57b5193-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3e1c8ff-fcc3-4a1f-9730-c617d57b5193-tmp-dir\") pod \"f3e1c8ff-fcc3-4a1f-9730-c617d57b5193\" (UID: \"f3e1c8ff-fcc3-4a1f-9730-c617d57b5193\") " Oct 30 17:27:35.159683 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:35.159467 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80b9bd9a-30d6-45ff-b318-26fcdc608e5b-tls-certs\" (UniqueName: \"kubernetes.io/secret/80b9bd9a-30d6-45ff-b318-26fcdc608e5b-tls-certs\") pod \"80b9bd9a-30d6-45ff-b318-26fcdc608e5b\" (UID: \"80b9bd9a-30d6-45ff-b318-26fcdc608e5b\") " Oct 30 17:27:35.159683 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:35.159518 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80b9bd9a-30d6-45ff-b318-26fcdc608e5b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/80b9bd9a-30d6-45ff-b318-26fcdc608e5b-dshm\") pod \"80b9bd9a-30d6-45ff-b318-26fcdc608e5b\" (UID: \"80b9bd9a-30d6-45ff-b318-26fcdc608e5b\") " Oct 30 17:27:35.159683 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:35.159592 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3e1c8ff-fcc3-4a1f-9730-c617d57b5193-home\" (UniqueName: \"kubernetes.io/empty-dir/f3e1c8ff-fcc3-4a1f-9730-c617d57b5193-home\") pod \"f3e1c8ff-fcc3-4a1f-9730-c617d57b5193\" (UID: \"f3e1c8ff-fcc3-4a1f-9730-c617d57b5193\") " Oct 30 17:27:35.159935 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:35.159914 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80b9bd9a-30d6-45ff-b318-26fcdc608e5b-tmp-dir" pod "80b9bd9a-30d6-45ff-b318-26fcdc608e5b" (UID: "80b9bd9a-30d6-45ff-b318-26fcdc608e5b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:27:35.160369 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:35.160016 4493 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3e1c8ff-fcc3-4a1f-9730-c617d57b5193-model-cache\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:35.160369 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:35.160042 4493 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80b9bd9a-30d6-45ff-b318-26fcdc608e5b-tmp-dir\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:35.160369 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:35.160286 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3e1c8ff-fcc3-4a1f-9730-c617d57b5193-home" pod "f3e1c8ff-fcc3-4a1f-9730-c617d57b5193" (UID: "f3e1c8ff-fcc3-4a1f-9730-c617d57b5193"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:27:35.160588 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:35.160485 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80b9bd9a-30d6-45ff-b318-26fcdc608e5b-model-cache" pod "80b9bd9a-30d6-45ff-b318-26fcdc608e5b" (UID: "80b9bd9a-30d6-45ff-b318-26fcdc608e5b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:27:35.160656 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:35.160630 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80b9bd9a-30d6-45ff-b318-26fcdc608e5b-home" pod "80b9bd9a-30d6-45ff-b318-26fcdc608e5b" (UID: "80b9bd9a-30d6-45ff-b318-26fcdc608e5b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:27:35.160716 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:35.160672 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3e1c8ff-fcc3-4a1f-9730-c617d57b5193-tmp-dir" pod "f3e1c8ff-fcc3-4a1f-9730-c617d57b5193" (UID: "f3e1c8ff-fcc3-4a1f-9730-c617d57b5193"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:27:35.163778 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:35.163753 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80b9bd9a-30d6-45ff-b318-26fcdc608e5b-tls-certs" pod "80b9bd9a-30d6-45ff-b318-26fcdc608e5b" (UID: "80b9bd9a-30d6-45ff-b318-26fcdc608e5b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:27:35.164845 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:35.164811 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80b9bd9a-30d6-45ff-b318-26fcdc608e5b-dshm" pod "80b9bd9a-30d6-45ff-b318-26fcdc608e5b" (UID: "80b9bd9a-30d6-45ff-b318-26fcdc608e5b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:27:35.165268 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:35.165246 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80b9bd9a-30d6-45ff-b318-26fcdc608e5b-kube-api-access-cc4wt" pod "80b9bd9a-30d6-45ff-b318-26fcdc608e5b" (UID: "80b9bd9a-30d6-45ff-b318-26fcdc608e5b"). InnerVolumeSpecName "kube-api-access-cc4wt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:27:35.165651 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:35.165630 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3e1c8ff-fcc3-4a1f-9730-c617d57b5193-tls-certs" pod "f3e1c8ff-fcc3-4a1f-9730-c617d57b5193" (UID: "f3e1c8ff-fcc3-4a1f-9730-c617d57b5193"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:27:35.165874 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:35.165844 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3e1c8ff-fcc3-4a1f-9730-c617d57b5193-kube-api-access-hf9dq" pod "f3e1c8ff-fcc3-4a1f-9730-c617d57b5193" (UID: "f3e1c8ff-fcc3-4a1f-9730-c617d57b5193"). InnerVolumeSpecName "kube-api-access-hf9dq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:27:35.166486 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:35.166460 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3e1c8ff-fcc3-4a1f-9730-c617d57b5193-dshm" pod "f3e1c8ff-fcc3-4a1f-9730-c617d57b5193" (UID: "f3e1c8ff-fcc3-4a1f-9730-c617d57b5193"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:27:35.261314 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:35.261277 4493 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3e1c8ff-fcc3-4a1f-9730-c617d57b5193-tls-certs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:35.261566 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:35.261546 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cc4wt\" (UniqueName: \"kubernetes.io/projected/80b9bd9a-30d6-45ff-b318-26fcdc608e5b-kube-api-access-cc4wt\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:35.261708 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:35.261692 4493 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3e1c8ff-fcc3-4a1f-9730-c617d57b5193-dshm\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:35.261844 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:35.261830 4493 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80b9bd9a-30d6-45ff-b318-26fcdc608e5b-home\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:35.261970 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:35.261956 4493 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3e1c8ff-fcc3-4a1f-9730-c617d57b5193-tmp-dir\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:35.262066 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:35.262054 4493 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80b9bd9a-30d6-45ff-b318-26fcdc608e5b-tls-certs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:35.262165 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:35.262153 4493 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80b9bd9a-30d6-45ff-b318-26fcdc608e5b-dshm\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:35.262315 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:35.262261 4493 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3e1c8ff-fcc3-4a1f-9730-c617d57b5193-home\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:35.262315 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:35.262280 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hf9dq\" (UniqueName: \"kubernetes.io/projected/f3e1c8ff-fcc3-4a1f-9730-c617d57b5193-kube-api-access-hf9dq\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:35.262315 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:35.262295 4493 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80b9bd9a-30d6-45ff-b318-26fcdc608e5b-model-cache\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:36.087197 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:36.087112 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" event={"ID":"f3e1c8ff-fcc3-4a1f-9730-c617d57b5193","Type":"ContainerDied","Data":"9d2459c2b65702848197ff48d8f63bbb0b3ba48eea70de5fb5dc8a1eef208d5a"} Oct 30 17:27:36.087197 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:36.087132 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh" Oct 30 17:27:36.087197 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:36.087143 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt" Oct 30 17:27:36.087197 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:36.087178 4493 scope.go:122] "RemoveContainer" containerID="d22765a80c71546b77c331be945ade3d585939b7efc74b5b2adb47c9815bf2d4" Oct 30 17:27:36.147985 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:36.147945 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh"] Oct 30 17:27:36.155665 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:36.155624 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-18519c/simulator-w2-18519c-kserve-7994695cdd-rlwgh"] Oct 30 17:27:36.185168 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:36.185129 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt"] Oct 30 17:27:36.191080 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:36.191054 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-18519c/distinct-2-w2-18519c-kserve-85f6799df5-xjjxt"] Oct 30 17:27:36.934988 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:36.934951 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80b9bd9a-30d6-45ff-b318-26fcdc608e5b" path="/var/lib/kubelet/pods/80b9bd9a-30d6-45ff-b318-26fcdc608e5b/volumes" Oct 30 17:27:36.935400 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:36.935386 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3e1c8ff-fcc3-4a1f-9730-c617d57b5193" path="/var/lib/kubelet/pods/f3e1c8ff-fcc3-4a1f-9730-c617d57b5193/volumes" Oct 30 17:27:37.302748 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:37.302645 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-420ff3-lk9n2"] Oct 30 17:27:37.306508 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:37.306474 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-420ff3-lk9n2"] Oct 30 17:27:38.933915 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:38.933859 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af343e15-1b0d-483f-bc11-7cb765c29fe4" path="/var/lib/kubelet/pods/af343e15-1b0d-483f-bc11-7cb765c29fe4/volumes" Oct 30 17:27:41.208871 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:41.208830 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f3503b-mqcjn"] Oct 30 17:27:41.209564 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:41.209545 4493 cpu_manager.go:414] "containerMap: removing container" podUID="6157f678-6cf7-4cc4-a4db-20c32afeb714" containerName="revoke-keys" Oct 30 17:27:41.209647 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:41.209566 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6157f678-6cf7-4cc4-a4db-20c32afeb714" containerName="revoke-keys" Oct 30 17:27:41.209647 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:41.209598 4493 cpu_manager.go:414] "containerMap: removing container" podUID="f3e1c8ff-fcc3-4a1f-9730-c617d57b5193" containerName="main" Oct 30 17:27:41.209647 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:41.209606 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3e1c8ff-fcc3-4a1f-9730-c617d57b5193" containerName="main" Oct 30 17:27:41.209865 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:41.209653 4493 cpu_manager.go:414] "containerMap: removing container" podUID="80b9bd9a-30d6-45ff-b318-26fcdc608e5b" containerName="main" Oct 30 17:27:41.209865 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:41.209662 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80b9bd9a-30d6-45ff-b318-26fcdc608e5b" containerName="main" Oct 30 17:27:41.209865 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:41.209673 4493 cpu_manager.go:414] "containerMap: removing container" podUID="36ad9481-025e-488d-aa35-8fc61f4c3034" containerName="revoke-keys" Oct 30 17:27:41.209865 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:41.209681 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36ad9481-025e-488d-aa35-8fc61f4c3034" containerName="revoke-keys" Oct 30 17:27:41.209865 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:41.209695 4493 cpu_manager.go:414] "containerMap: removing container" podUID="a6fef0c5-c804-4953-a1be-9b49291e63d5" containerName="payload-pre-processing" Oct 30 17:27:41.209865 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:41.209703 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6fef0c5-c804-4953-a1be-9b49291e63d5" containerName="payload-pre-processing" Oct 30 17:27:41.209865 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:41.209818 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="6157f678-6cf7-4cc4-a4db-20c32afeb714" containerName="revoke-keys" Oct 30 17:27:41.209865 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:41.209832 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3e1c8ff-fcc3-4a1f-9730-c617d57b5193" containerName="main" Oct 30 17:27:41.209865 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:41.209848 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="80b9bd9a-30d6-45ff-b318-26fcdc608e5b" containerName="main" Oct 30 17:27:41.209865 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:41.209857 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="36ad9481-025e-488d-aa35-8fc61f4c3034" containerName="revoke-keys" Oct 30 17:27:41.209865 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:41.209868 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6fef0c5-c804-4953-a1be-9b49291e63d5" containerName="payload-pre-processing" Oct 30 17:27:41.215268 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:41.215247 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f3503b-mqcjn" Oct 30 17:27:41.233952 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:41.233924 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f3503b-mqcjn"] Oct 30 17:27:41.241250 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:41.241225 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:27:41.241400 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:41.241289 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:27:41.247816 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:41.246215 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:27:41.247816 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:41.246350 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 17:27:41.348054 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:41.348010 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a5999d8-6e17-496d-b28c-e3d6ac908b07-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-f3503b-mqcjn\" (UID: \"2a5999d8-6e17-496d-b28c-e3d6ac908b07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f3503b-mqcjn" Oct 30 17:27:41.348240 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:41.348201 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a5999d8-6e17-496d-b28c-e3d6ac908b07-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-f3503b-mqcjn\" (UID: \"2a5999d8-6e17-496d-b28c-e3d6ac908b07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f3503b-mqcjn" Oct 30 17:27:41.348325 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:41.348307 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh48s\" (UniqueName: \"kubernetes.io/projected/2a5999d8-6e17-496d-b28c-e3d6ac908b07-kube-api-access-nh48s\") pod \"maas-api-revoke-keys-e2e-worker-w0-f3503b-mqcjn\" (UID: \"2a5999d8-6e17-496d-b28c-e3d6ac908b07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f3503b-mqcjn" Oct 30 17:27:41.450141 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:41.450097 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a5999d8-6e17-496d-b28c-e3d6ac908b07-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-f3503b-mqcjn\" (UID: \"2a5999d8-6e17-496d-b28c-e3d6ac908b07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f3503b-mqcjn" Oct 30 17:27:41.450303 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:41.450235 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a5999d8-6e17-496d-b28c-e3d6ac908b07-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-f3503b-mqcjn\" (UID: \"2a5999d8-6e17-496d-b28c-e3d6ac908b07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f3503b-mqcjn" Oct 30 17:27:41.450382 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:41.450309 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh48s\" (UniqueName: \"kubernetes.io/projected/2a5999d8-6e17-496d-b28c-e3d6ac908b07-kube-api-access-nh48s\") pod \"maas-api-revoke-keys-e2e-worker-w0-f3503b-mqcjn\" (UID: \"2a5999d8-6e17-496d-b28c-e3d6ac908b07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f3503b-mqcjn" Oct 30 17:27:41.450984 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:41.450957 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a5999d8-6e17-496d-b28c-e3d6ac908b07-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-f3503b-mqcjn\" (UID: \"2a5999d8-6e17-496d-b28c-e3d6ac908b07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f3503b-mqcjn" Oct 30 17:27:41.452885 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:41.452856 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a5999d8-6e17-496d-b28c-e3d6ac908b07-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-f3503b-mqcjn\" (UID: \"2a5999d8-6e17-496d-b28c-e3d6ac908b07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f3503b-mqcjn" Oct 30 17:27:41.473340 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:41.473281 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh48s\" (UniqueName: \"kubernetes.io/projected/2a5999d8-6e17-496d-b28c-e3d6ac908b07-kube-api-access-nh48s\") pod \"maas-api-revoke-keys-e2e-worker-w0-f3503b-mqcjn\" (UID: \"2a5999d8-6e17-496d-b28c-e3d6ac908b07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f3503b-mqcjn" Oct 30 17:27:41.530510 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:41.530469 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f3503b-mqcjn" Oct 30 17:27:41.836222 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:41.836187 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f3503b-mqcjn"] Oct 30 17:27:41.837695 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:27:41.837663 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a5999d8_6e17_496d_b28c_e3d6ac908b07.slice/crio-cc36b860db24ae6799744c834a9a0cb37eead702f2ba622cbd56435ecacbbd08 WatchSource:0}: Error finding container cc36b860db24ae6799744c834a9a0cb37eead702f2ba622cbd56435ecacbbd08: Status 404 returned error can't find the container with id cc36b860db24ae6799744c834a9a0cb37eead702f2ba622cbd56435ecacbbd08 Oct 30 17:27:42.149315 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:42.149278 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f3503b-mqcjn" event={"ID":"2a5999d8-6e17-496d-b28c-e3d6ac908b07","Type":"ContainerStarted","Data":"0be9f3c507fedcee6e09dc62ce53f00d22cbe0bcb9b394e1a6c85c925e1d58f2"} Oct 30 17:27:42.149474 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:42.149371 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f3503b-mqcjn" event={"ID":"2a5999d8-6e17-496d-b28c-e3d6ac908b07","Type":"ContainerStarted","Data":"cc36b860db24ae6799744c834a9a0cb37eead702f2ba622cbd56435ecacbbd08"} Oct 30 17:27:42.192014 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:42.191967 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f3503b-mqcjn" podStartSLOduration=1.1919536370000001 podStartE2EDuration="1.191953637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:27:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:27:42.190541796 +0000 UTC m=+3889.417716899" watchObservedRunningTime="2026-10-30 17:27:42.191953637 +0000 UTC m=+3889.419128756" Oct 30 17:27:42.647343 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:42.647307 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4"] Oct 30 17:27:42.648153 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:42.648119 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4" podUID="a18c02c8-1a51-4dcf-aeef-696acb2d2280" containerName="istio-proxy" containerID="cri-o://0a51574a70b4dc00ccd73e8dccdb4d1ad094ccbedaae44605fb8ad39482e97e1" gracePeriod=30 Oct 30 17:27:43.160846 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:43.160810 4493 generic.go:356] "Generic (PLEG): container finished" podID="2a5999d8-6e17-496d-b28c-e3d6ac908b07" containerID="0be9f3c507fedcee6e09dc62ce53f00d22cbe0bcb9b394e1a6c85c925e1d58f2" exitCode=0 Oct 30 17:27:43.161049 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:43.160909 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f3503b-mqcjn" event={"ID":"2a5999d8-6e17-496d-b28c-e3d6ac908b07","Type":"ContainerDied","Data":"0be9f3c507fedcee6e09dc62ce53f00d22cbe0bcb9b394e1a6c85c925e1d58f2"} Oct 30 17:27:43.228088 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:43.228049 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f94fb5-j22h6"] Oct 30 17:27:43.232069 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:43.232045 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f94fb5-j22h6" Oct 30 17:27:43.260223 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:43.260160 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f94fb5-j22h6"] Oct 30 17:27:43.270448 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:43.270414 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjjvm\" (UniqueName: \"kubernetes.io/projected/38ebec4d-5fbf-4555-a697-4f3084de0d26-kube-api-access-xjjvm\") pod \"maas-api-revoke-keys-e2e-extmod-a-f94fb5-j22h6\" (UID: \"38ebec4d-5fbf-4555-a697-4f3084de0d26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f94fb5-j22h6" Oct 30 17:27:43.270604 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:43.270472 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38ebec4d-5fbf-4555-a697-4f3084de0d26-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-f94fb5-j22h6\" (UID: \"38ebec4d-5fbf-4555-a697-4f3084de0d26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f94fb5-j22h6" Oct 30 17:27:43.270604 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:43.270591 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38ebec4d-5fbf-4555-a697-4f3084de0d26-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-f94fb5-j22h6\" (UID: \"38ebec4d-5fbf-4555-a697-4f3084de0d26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f94fb5-j22h6" Oct 30 17:27:43.371280 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:43.371241 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38ebec4d-5fbf-4555-a697-4f3084de0d26-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-f94fb5-j22h6\" (UID: \"38ebec4d-5fbf-4555-a697-4f3084de0d26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f94fb5-j22h6" Oct 30 17:27:43.371491 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:43.371378 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjjvm\" (UniqueName: \"kubernetes.io/projected/38ebec4d-5fbf-4555-a697-4f3084de0d26-kube-api-access-xjjvm\") pod \"maas-api-revoke-keys-e2e-extmod-a-f94fb5-j22h6\" (UID: \"38ebec4d-5fbf-4555-a697-4f3084de0d26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f94fb5-j22h6" Oct 30 17:27:43.371491 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:43.371407 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38ebec4d-5fbf-4555-a697-4f3084de0d26-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-f94fb5-j22h6\" (UID: \"38ebec4d-5fbf-4555-a697-4f3084de0d26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f94fb5-j22h6" Oct 30 17:27:43.372133 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:43.372106 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38ebec4d-5fbf-4555-a697-4f3084de0d26-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-f94fb5-j22h6\" (UID: \"38ebec4d-5fbf-4555-a697-4f3084de0d26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f94fb5-j22h6" Oct 30 17:27:43.373941 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:43.373914 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38ebec4d-5fbf-4555-a697-4f3084de0d26-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-f94fb5-j22h6\" (UID: \"38ebec4d-5fbf-4555-a697-4f3084de0d26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f94fb5-j22h6" Oct 30 17:27:43.393915 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:43.393891 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjjvm\" (UniqueName: \"kubernetes.io/projected/38ebec4d-5fbf-4555-a697-4f3084de0d26-kube-api-access-xjjvm\") pod \"maas-api-revoke-keys-e2e-extmod-a-f94fb5-j22h6\" (UID: \"38ebec4d-5fbf-4555-a697-4f3084de0d26\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f94fb5-j22h6" Oct 30 17:27:43.553694 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:43.553608 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f94fb5-j22h6" Oct 30 17:27:43.867219 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:43.867195 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f94fb5-j22h6"] Oct 30 17:27:43.869014 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:27:43.868975 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38ebec4d_5fbf_4555_a697_4f3084de0d26.slice/crio-b20b364715c49516b7c1f85fc7e2ee37fb4d8e46ce69a8f3df573ad3ade8160a WatchSource:0}: Error finding container b20b364715c49516b7c1f85fc7e2ee37fb4d8e46ce69a8f3df573ad3ade8160a: Status 404 returned error can't find the container with id b20b364715c49516b7c1f85fc7e2ee37fb4d8e46ce69a8f3df573ad3ade8160a Oct 30 17:27:44.169733 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:44.169696 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f94fb5-j22h6" event={"ID":"38ebec4d-5fbf-4555-a697-4f3084de0d26","Type":"ContainerStarted","Data":"b20b364715c49516b7c1f85fc7e2ee37fb4d8e46ce69a8f3df573ad3ade8160a"} Oct 30 17:27:44.313600 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:44.313574 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f3503b-mqcjn" Oct 30 17:27:44.382174 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:44.382089 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a5999d8-6e17-496d-b28c-e3d6ac908b07-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a5999d8-6e17-496d-b28c-e3d6ac908b07-maas-api-openshift-ca\") pod \"2a5999d8-6e17-496d-b28c-e3d6ac908b07\" (UID: \"2a5999d8-6e17-496d-b28c-e3d6ac908b07\") " Oct 30 17:27:44.382361 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:44.382207 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a5999d8-6e17-496d-b28c-e3d6ac908b07-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a5999d8-6e17-496d-b28c-e3d6ac908b07-maas-api-xks-ca\") pod \"2a5999d8-6e17-496d-b28c-e3d6ac908b07\" (UID: \"2a5999d8-6e17-496d-b28c-e3d6ac908b07\") " Oct 30 17:27:44.382361 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:44.382251 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a5999d8-6e17-496d-b28c-e3d6ac908b07-kube-api-access-nh48s\" (UniqueName: \"kubernetes.io/projected/2a5999d8-6e17-496d-b28c-e3d6ac908b07-kube-api-access-nh48s\") pod \"2a5999d8-6e17-496d-b28c-e3d6ac908b07\" (UID: \"2a5999d8-6e17-496d-b28c-e3d6ac908b07\") " Oct 30 17:27:44.382558 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:44.382533 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a5999d8-6e17-496d-b28c-e3d6ac908b07-maas-api-openshift-ca" pod "2a5999d8-6e17-496d-b28c-e3d6ac908b07" (UID: "2a5999d8-6e17-496d-b28c-e3d6ac908b07"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:27:44.382666 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:44.382643 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a5999d8-6e17-496d-b28c-e3d6ac908b07-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:44.384378 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:44.384354 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a5999d8-6e17-496d-b28c-e3d6ac908b07-maas-api-xks-ca" pod "2a5999d8-6e17-496d-b28c-e3d6ac908b07" (UID: "2a5999d8-6e17-496d-b28c-e3d6ac908b07"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:27:44.384517 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:44.384469 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a5999d8-6e17-496d-b28c-e3d6ac908b07-kube-api-access-nh48s" pod "2a5999d8-6e17-496d-b28c-e3d6ac908b07" (UID: "2a5999d8-6e17-496d-b28c-e3d6ac908b07"). InnerVolumeSpecName "kube-api-access-nh48s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:27:44.483329 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:44.483293 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a5999d8-6e17-496d-b28c-e3d6ac908b07-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:44.483329 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:44.483328 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nh48s\" (UniqueName: \"kubernetes.io/projected/2a5999d8-6e17-496d-b28c-e3d6ac908b07-kube-api-access-nh48s\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:44.923545 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:44.923491 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7"] Oct 30 17:27:44.924270 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:44.924240 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7" podUID="efbed66f-95b0-4fd7-af06-388840a2a364" containerName="istio-proxy" containerID="cri-o://321c60a2853e7c03d368707a76747e28e8b5576ae64eb10fab27174df047b587" gracePeriod=30 Oct 30 17:27:45.179217 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:45.179121 4493 generic.go:356] "Generic (PLEG): container finished" podID="38ebec4d-5fbf-4555-a697-4f3084de0d26" containerID="8e75860c516916c51a4543c1fe29529960d29506dbde382296c4ed7ae291e3d8" exitCode=0 Oct 30 17:27:45.179386 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:45.179213 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f94fb5-j22h6" event={"ID":"38ebec4d-5fbf-4555-a697-4f3084de0d26","Type":"ContainerDied","Data":"8e75860c516916c51a4543c1fe29529960d29506dbde382296c4ed7ae291e3d8"} Oct 30 17:27:45.180744 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:45.180716 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f3503b-mqcjn" event={"ID":"2a5999d8-6e17-496d-b28c-e3d6ac908b07","Type":"ContainerDied","Data":"cc36b860db24ae6799744c834a9a0cb37eead702f2ba622cbd56435ecacbbd08"} Oct 30 17:27:45.180925 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:45.180746 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f3503b-mqcjn" Oct 30 17:27:45.180925 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:45.180754 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc36b860db24ae6799744c834a9a0cb37eead702f2ba622cbd56435ecacbbd08" Oct 30 17:27:45.948389 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:45.948355 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642"] Oct 30 17:27:45.960459 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:45.960426 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq"] Oct 30 17:27:46.345467 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:46.345439 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f94fb5-j22h6" Oct 30 17:27:46.400728 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:46.400696 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38ebec4d-5fbf-4555-a697-4f3084de0d26-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38ebec4d-5fbf-4555-a697-4f3084de0d26-maas-api-xks-ca\") pod \"38ebec4d-5fbf-4555-a697-4f3084de0d26\" (UID: \"38ebec4d-5fbf-4555-a697-4f3084de0d26\") " Oct 30 17:27:46.400908 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:46.400768 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38ebec4d-5fbf-4555-a697-4f3084de0d26-kube-api-access-xjjvm\" (UniqueName: \"kubernetes.io/projected/38ebec4d-5fbf-4555-a697-4f3084de0d26-kube-api-access-xjjvm\") pod \"38ebec4d-5fbf-4555-a697-4f3084de0d26\" (UID: \"38ebec4d-5fbf-4555-a697-4f3084de0d26\") " Oct 30 17:27:46.400908 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:46.400841 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38ebec4d-5fbf-4555-a697-4f3084de0d26-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38ebec4d-5fbf-4555-a697-4f3084de0d26-maas-api-openshift-ca\") pod \"38ebec4d-5fbf-4555-a697-4f3084de0d26\" (UID: \"38ebec4d-5fbf-4555-a697-4f3084de0d26\") " Oct 30 17:27:46.401258 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:46.401233 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38ebec4d-5fbf-4555-a697-4f3084de0d26-maas-api-openshift-ca" pod "38ebec4d-5fbf-4555-a697-4f3084de0d26" (UID: "38ebec4d-5fbf-4555-a697-4f3084de0d26"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:27:46.403321 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:46.403286 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38ebec4d-5fbf-4555-a697-4f3084de0d26-kube-api-access-xjjvm" pod "38ebec4d-5fbf-4555-a697-4f3084de0d26" (UID: "38ebec4d-5fbf-4555-a697-4f3084de0d26"). InnerVolumeSpecName "kube-api-access-xjjvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:27:46.403321 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:46.403309 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38ebec4d-5fbf-4555-a697-4f3084de0d26-maas-api-xks-ca" pod "38ebec4d-5fbf-4555-a697-4f3084de0d26" (UID: "38ebec4d-5fbf-4555-a697-4f3084de0d26"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:27:46.501937 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:46.501854 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjjvm\" (UniqueName: \"kubernetes.io/projected/38ebec4d-5fbf-4555-a697-4f3084de0d26-kube-api-access-xjjvm\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:46.501937 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:46.501892 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38ebec4d-5fbf-4555-a697-4f3084de0d26-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:46.501937 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:46.501904 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38ebec4d-5fbf-4555-a697-4f3084de0d26-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:47.207945 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:47.207855 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f94fb5-j22h6" Oct 30 17:27:47.207945 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:47.207885 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f94fb5-j22h6" event={"ID":"38ebec4d-5fbf-4555-a697-4f3084de0d26","Type":"ContainerDied","Data":"b20b364715c49516b7c1f85fc7e2ee37fb4d8e46ce69a8f3df573ad3ade8160a"} Oct 30 17:27:47.207945 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:47.207933 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b20b364715c49516b7c1f85fc7e2ee37fb4d8e46ce69a8f3df573ad3ade8160a" Oct 30 17:27:47.367604 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:47.367566 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-18519c-bsb42"] Oct 30 17:27:47.370940 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:47.370912 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-18519c-bsb42"] Oct 30 17:27:48.220467 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:48.220432 4493 generic.go:356] "Generic (PLEG): container finished" podID="a18c02c8-1a51-4dcf-aeef-696acb2d2280" containerID="0a51574a70b4dc00ccd73e8dccdb4d1ad094ccbedaae44605fb8ad39482e97e1" exitCode=0 Oct 30 17:27:48.220975 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:48.220490 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4" event={"ID":"a18c02c8-1a51-4dcf-aeef-696acb2d2280","Type":"ContainerDied","Data":"0a51574a70b4dc00ccd73e8dccdb4d1ad094ccbedaae44605fb8ad39482e97e1"} Oct 30 17:27:48.220975 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:48.220554 4493 scope.go:122] "RemoveContainer" containerID="49cc2b2335dd4d05149323f545e7f7c0c490009af8f91c804dfc19053918d7fb" Oct 30 17:27:48.325066 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:48.325043 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4" Oct 30 17:27:48.423128 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:48.423037 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a18c02c8-1a51-4dcf-aeef-696acb2d2280-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a18c02c8-1a51-4dcf-aeef-696acb2d2280-istio-envoy\") pod \"a18c02c8-1a51-4dcf-aeef-696acb2d2280\" (UID: \"a18c02c8-1a51-4dcf-aeef-696acb2d2280\") " Oct 30 17:27:48.423307 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:48.423129 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a18c02c8-1a51-4dcf-aeef-696acb2d2280-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a18c02c8-1a51-4dcf-aeef-696acb2d2280-credential-socket\") pod \"a18c02c8-1a51-4dcf-aeef-696acb2d2280\" (UID: \"a18c02c8-1a51-4dcf-aeef-696acb2d2280\") " Oct 30 17:27:48.423307 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:48.423179 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a18c02c8-1a51-4dcf-aeef-696acb2d2280-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a18c02c8-1a51-4dcf-aeef-696acb2d2280-istiod-ca-cert\") pod \"a18c02c8-1a51-4dcf-aeef-696acb2d2280\" (UID: \"a18c02c8-1a51-4dcf-aeef-696acb2d2280\") " Oct 30 17:27:48.423307 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:48.423235 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a18c02c8-1a51-4dcf-aeef-696acb2d2280-kube-api-access-rp9nt\" (UniqueName: \"kubernetes.io/projected/a18c02c8-1a51-4dcf-aeef-696acb2d2280-kube-api-access-rp9nt\") pod \"a18c02c8-1a51-4dcf-aeef-696acb2d2280\" (UID: \"a18c02c8-1a51-4dcf-aeef-696acb2d2280\") " Oct 30 17:27:48.423307 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:48.423264 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a18c02c8-1a51-4dcf-aeef-696acb2d2280-istio-token\" (UniqueName: \"kubernetes.io/projected/a18c02c8-1a51-4dcf-aeef-696acb2d2280-istio-token\") pod \"a18c02c8-1a51-4dcf-aeef-696acb2d2280\" (UID: \"a18c02c8-1a51-4dcf-aeef-696acb2d2280\") " Oct 30 17:27:48.423525 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:48.423483 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a18c02c8-1a51-4dcf-aeef-696acb2d2280-credential-socket" pod "a18c02c8-1a51-4dcf-aeef-696acb2d2280" (UID: "a18c02c8-1a51-4dcf-aeef-696acb2d2280"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:27:48.423583 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:48.423529 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a18c02c8-1a51-4dcf-aeef-696acb2d2280-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a18c02c8-1a51-4dcf-aeef-696acb2d2280-istio-podinfo\") pod \"a18c02c8-1a51-4dcf-aeef-696acb2d2280\" (UID: \"a18c02c8-1a51-4dcf-aeef-696acb2d2280\") " Oct 30 17:27:48.423583 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:48.423573 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a18c02c8-1a51-4dcf-aeef-696acb2d2280-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a18c02c8-1a51-4dcf-aeef-696acb2d2280-workload-certs\") pod \"a18c02c8-1a51-4dcf-aeef-696acb2d2280\" (UID: \"a18c02c8-1a51-4dcf-aeef-696acb2d2280\") " Oct 30 17:27:48.423687 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:48.423607 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a18c02c8-1a51-4dcf-aeef-696acb2d2280-istiod-ca-cert" pod "a18c02c8-1a51-4dcf-aeef-696acb2d2280" (UID: "a18c02c8-1a51-4dcf-aeef-696acb2d2280"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:27:48.423738 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:48.423705 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a18c02c8-1a51-4dcf-aeef-696acb2d2280-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a18c02c8-1a51-4dcf-aeef-696acb2d2280-workload-socket\") pod \"a18c02c8-1a51-4dcf-aeef-696acb2d2280\" (UID: \"a18c02c8-1a51-4dcf-aeef-696acb2d2280\") " Oct 30 17:27:48.423953 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:48.423850 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a18c02c8-1a51-4dcf-aeef-696acb2d2280-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a18c02c8-1a51-4dcf-aeef-696acb2d2280-istio-data\") pod \"a18c02c8-1a51-4dcf-aeef-696acb2d2280\" (UID: \"a18c02c8-1a51-4dcf-aeef-696acb2d2280\") " Oct 30 17:27:48.424060 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:48.423953 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a18c02c8-1a51-4dcf-aeef-696acb2d2280-workload-certs" pod "a18c02c8-1a51-4dcf-aeef-696acb2d2280" (UID: "a18c02c8-1a51-4dcf-aeef-696acb2d2280"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:27:48.424116 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:48.424084 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a18c02c8-1a51-4dcf-aeef-696acb2d2280-workload-socket" pod "a18c02c8-1a51-4dcf-aeef-696acb2d2280" (UID: "a18c02c8-1a51-4dcf-aeef-696acb2d2280"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:27:48.424235 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:48.424214 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a18c02c8-1a51-4dcf-aeef-696acb2d2280-istio-data" pod "a18c02c8-1a51-4dcf-aeef-696acb2d2280" (UID: "a18c02c8-1a51-4dcf-aeef-696acb2d2280"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:27:48.424408 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:48.424388 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a18c02c8-1a51-4dcf-aeef-696acb2d2280-workload-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:48.424408 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:48.424409 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a18c02c8-1a51-4dcf-aeef-696acb2d2280-istio-data\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:48.424592 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:48.424421 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a18c02c8-1a51-4dcf-aeef-696acb2d2280-credential-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:48.424592 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:48.424434 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a18c02c8-1a51-4dcf-aeef-696acb2d2280-istiod-ca-cert\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:48.424592 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:48.424446 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a18c02c8-1a51-4dcf-aeef-696acb2d2280-workload-certs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:48.426043 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:48.425979 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a18c02c8-1a51-4dcf-aeef-696acb2d2280-istio-token" pod "a18c02c8-1a51-4dcf-aeef-696acb2d2280" (UID: "a18c02c8-1a51-4dcf-aeef-696acb2d2280"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:27:48.426043 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:48.426011 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a18c02c8-1a51-4dcf-aeef-696acb2d2280-istio-envoy" pod "a18c02c8-1a51-4dcf-aeef-696acb2d2280" (UID: "a18c02c8-1a51-4dcf-aeef-696acb2d2280"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:27:48.426302 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:48.426277 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a18c02c8-1a51-4dcf-aeef-696acb2d2280-istio-podinfo" pod "a18c02c8-1a51-4dcf-aeef-696acb2d2280" (UID: "a18c02c8-1a51-4dcf-aeef-696acb2d2280"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:27:48.426404 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:48.426379 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a18c02c8-1a51-4dcf-aeef-696acb2d2280-kube-api-access-rp9nt" pod "a18c02c8-1a51-4dcf-aeef-696acb2d2280" (UID: "a18c02c8-1a51-4dcf-aeef-696acb2d2280"). InnerVolumeSpecName "kube-api-access-rp9nt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:27:48.525606 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:48.525570 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rp9nt\" (UniqueName: \"kubernetes.io/projected/a18c02c8-1a51-4dcf-aeef-696acb2d2280-kube-api-access-rp9nt\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:48.525606 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:48.525608 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a18c02c8-1a51-4dcf-aeef-696acb2d2280-istio-token\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:48.525843 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:48.525622 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a18c02c8-1a51-4dcf-aeef-696acb2d2280-istio-podinfo\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:48.525843 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:48.525637 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a18c02c8-1a51-4dcf-aeef-696acb2d2280-istio-envoy\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:48.932716 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:48.932683 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="412a5b46-0344-4a22-ba2e-fc55c7b0102a" path="/var/lib/kubelet/pods/412a5b46-0344-4a22-ba2e-fc55c7b0102a/volumes" Oct 30 17:27:49.230329 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:49.230243 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4" Oct 30 17:27:49.230329 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:49.230259 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4" event={"ID":"a18c02c8-1a51-4dcf-aeef-696acb2d2280","Type":"ContainerDied","Data":"6c54b1050471c4abe67cefee17f9e473dcbb1d8856d7ddc9e9671ba419f5606d"} Oct 30 17:27:49.230329 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:49.230319 4493 scope.go:122] "RemoveContainer" containerID="0a51574a70b4dc00ccd73e8dccdb4d1ad094ccbedaae44605fb8ad39482e97e1" Oct 30 17:27:49.296303 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:49.296265 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4"] Oct 30 17:27:49.306425 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:49.306392 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-0b09d3-openshift-default-79bfb47f85-jvsw4"] Oct 30 17:27:50.211566 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:50.211509 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7" Oct 30 17:27:50.239158 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:50.239128 4493 generic.go:356] "Generic (PLEG): container finished" podID="efbed66f-95b0-4fd7-af06-388840a2a364" containerID="321c60a2853e7c03d368707a76747e28e8b5576ae64eb10fab27174df047b587" exitCode=0 Oct 30 17:27:50.239592 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:50.239165 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7" Oct 30 17:27:50.239592 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:50.239219 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7" event={"ID":"efbed66f-95b0-4fd7-af06-388840a2a364","Type":"ContainerDied","Data":"321c60a2853e7c03d368707a76747e28e8b5576ae64eb10fab27174df047b587"} Oct 30 17:27:50.239592 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:50.239275 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7" event={"ID":"efbed66f-95b0-4fd7-af06-388840a2a364","Type":"ContainerDied","Data":"4e3ae3d764c3ca01821bfda6d3aba9854fce4456ff97b0e246fad34816bd1188"} Oct 30 17:27:50.239592 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:50.239311 4493 scope.go:122] "RemoveContainer" containerID="321c60a2853e7c03d368707a76747e28e8b5576ae64eb10fab27174df047b587" Oct 30 17:27:50.246840 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:50.246817 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efbed66f-95b0-4fd7-af06-388840a2a364-kube-api-access-gwshr\" (UniqueName: \"kubernetes.io/projected/efbed66f-95b0-4fd7-af06-388840a2a364-kube-api-access-gwshr\") pod \"efbed66f-95b0-4fd7-af06-388840a2a364\" (UID: \"efbed66f-95b0-4fd7-af06-388840a2a364\") " Oct 30 17:27:50.247010 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:50.246860 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efbed66f-95b0-4fd7-af06-388840a2a364-istio-token\" (UniqueName: \"kubernetes.io/projected/efbed66f-95b0-4fd7-af06-388840a2a364-istio-token\") pod \"efbed66f-95b0-4fd7-af06-388840a2a364\" (UID: \"efbed66f-95b0-4fd7-af06-388840a2a364\") " Oct 30 17:27:50.247010 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:50.246914 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efbed66f-95b0-4fd7-af06-388840a2a364-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efbed66f-95b0-4fd7-af06-388840a2a364-istio-data\") pod \"efbed66f-95b0-4fd7-af06-388840a2a364\" (UID: \"efbed66f-95b0-4fd7-af06-388840a2a364\") " Oct 30 17:27:50.247010 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:50.246956 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/efbed66f-95b0-4fd7-af06-388840a2a364-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efbed66f-95b0-4fd7-af06-388840a2a364-istio-podinfo\") pod \"efbed66f-95b0-4fd7-af06-388840a2a364\" (UID: \"efbed66f-95b0-4fd7-af06-388840a2a364\") " Oct 30 17:27:50.247010 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:50.246985 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efbed66f-95b0-4fd7-af06-388840a2a364-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efbed66f-95b0-4fd7-af06-388840a2a364-istiod-ca-cert\") pod \"efbed66f-95b0-4fd7-af06-388840a2a364\" (UID: \"efbed66f-95b0-4fd7-af06-388840a2a364\") " Oct 30 17:27:50.247199 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:50.247015 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efbed66f-95b0-4fd7-af06-388840a2a364-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efbed66f-95b0-4fd7-af06-388840a2a364-credential-socket\") pod \"efbed66f-95b0-4fd7-af06-388840a2a364\" (UID: \"efbed66f-95b0-4fd7-af06-388840a2a364\") " Oct 30 17:27:50.247199 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:50.247078 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efbed66f-95b0-4fd7-af06-388840a2a364-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efbed66f-95b0-4fd7-af06-388840a2a364-workload-certs\") pod \"efbed66f-95b0-4fd7-af06-388840a2a364\" (UID: \"efbed66f-95b0-4fd7-af06-388840a2a364\") " Oct 30 17:27:50.247199 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:50.247146 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efbed66f-95b0-4fd7-af06-388840a2a364-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efbed66f-95b0-4fd7-af06-388840a2a364-istio-envoy\") pod \"efbed66f-95b0-4fd7-af06-388840a2a364\" (UID: \"efbed66f-95b0-4fd7-af06-388840a2a364\") " Oct 30 17:27:50.248634 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:50.247201 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efbed66f-95b0-4fd7-af06-388840a2a364-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efbed66f-95b0-4fd7-af06-388840a2a364-workload-socket\") pod \"efbed66f-95b0-4fd7-af06-388840a2a364\" (UID: \"efbed66f-95b0-4fd7-af06-388840a2a364\") " Oct 30 17:27:50.248634 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:50.247507 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efbed66f-95b0-4fd7-af06-388840a2a364-istiod-ca-cert" pod "efbed66f-95b0-4fd7-af06-388840a2a364" (UID: "efbed66f-95b0-4fd7-af06-388840a2a364"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:27:50.248634 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:50.247682 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efbed66f-95b0-4fd7-af06-388840a2a364-credential-socket" pod "efbed66f-95b0-4fd7-af06-388840a2a364" (UID: "efbed66f-95b0-4fd7-af06-388840a2a364"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:27:50.248634 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:50.247508 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efbed66f-95b0-4fd7-af06-388840a2a364-workload-certs" pod "efbed66f-95b0-4fd7-af06-388840a2a364" (UID: "efbed66f-95b0-4fd7-af06-388840a2a364"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:27:50.248634 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:50.247844 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efbed66f-95b0-4fd7-af06-388840a2a364-istiod-ca-cert\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:50.248634 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:50.248103 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efbed66f-95b0-4fd7-af06-388840a2a364-istio-data" pod "efbed66f-95b0-4fd7-af06-388840a2a364" (UID: "efbed66f-95b0-4fd7-af06-388840a2a364"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:27:50.248634 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:50.248464 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efbed66f-95b0-4fd7-af06-388840a2a364-workload-socket" pod "efbed66f-95b0-4fd7-af06-388840a2a364" (UID: "efbed66f-95b0-4fd7-af06-388840a2a364"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:27:50.249626 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:50.249597 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efbed66f-95b0-4fd7-af06-388840a2a364-kube-api-access-gwshr" pod "efbed66f-95b0-4fd7-af06-388840a2a364" (UID: "efbed66f-95b0-4fd7-af06-388840a2a364"). InnerVolumeSpecName "kube-api-access-gwshr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:27:50.249990 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:50.249969 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efbed66f-95b0-4fd7-af06-388840a2a364-istio-token" pod "efbed66f-95b0-4fd7-af06-388840a2a364" (UID: "efbed66f-95b0-4fd7-af06-388840a2a364"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:27:50.250137 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:50.250120 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/efbed66f-95b0-4fd7-af06-388840a2a364-istio-podinfo" pod "efbed66f-95b0-4fd7-af06-388840a2a364" (UID: "efbed66f-95b0-4fd7-af06-388840a2a364"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:27:50.251137 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:50.251111 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efbed66f-95b0-4fd7-af06-388840a2a364-istio-envoy" pod "efbed66f-95b0-4fd7-af06-388840a2a364" (UID: "efbed66f-95b0-4fd7-af06-388840a2a364"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:27:50.252632 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:50.252551 4493 scope.go:122] "RemoveContainer" containerID="321c60a2853e7c03d368707a76747e28e8b5576ae64eb10fab27174df047b587" Oct 30 17:27:50.252935 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:27:50.252914 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"321c60a2853e7c03d368707a76747e28e8b5576ae64eb10fab27174df047b587\": container with ID starting with 321c60a2853e7c03d368707a76747e28e8b5576ae64eb10fab27174df047b587 not found: ID does not exist" containerID="321c60a2853e7c03d368707a76747e28e8b5576ae64eb10fab27174df047b587" Oct 30 17:27:50.253013 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:50.252944 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"321c60a2853e7c03d368707a76747e28e8b5576ae64eb10fab27174df047b587"} err="failed to get container status \"321c60a2853e7c03d368707a76747e28e8b5576ae64eb10fab27174df047b587\": rpc error: code = NotFound desc = could not find container \"321c60a2853e7c03d368707a76747e28e8b5576ae64eb10fab27174df047b587\": container with ID starting with 321c60a2853e7c03d368707a76747e28e8b5576ae64eb10fab27174df047b587 not found: ID does not exist" Oct 30 17:27:50.348408 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:50.348374 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwshr\" (UniqueName: \"kubernetes.io/projected/efbed66f-95b0-4fd7-af06-388840a2a364-kube-api-access-gwshr\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:50.348408 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:50.348403 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efbed66f-95b0-4fd7-af06-388840a2a364-istio-token\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:50.348408 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:50.348412 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efbed66f-95b0-4fd7-af06-388840a2a364-istio-data\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:50.348659 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:50.348420 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efbed66f-95b0-4fd7-af06-388840a2a364-istio-podinfo\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:50.348659 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:50.348428 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efbed66f-95b0-4fd7-af06-388840a2a364-credential-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:50.348659 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:50.348437 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efbed66f-95b0-4fd7-af06-388840a2a364-workload-certs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:50.348659 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:50.348445 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efbed66f-95b0-4fd7-af06-388840a2a364-istio-envoy\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:50.348659 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:50.348453 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efbed66f-95b0-4fd7-af06-388840a2a364-workload-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:50.585814 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:50.585767 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7"] Oct 30 17:27:50.597023 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:50.596995 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-v9sv7"] Oct 30 17:27:50.933921 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:50.933872 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a18c02c8-1a51-4dcf-aeef-696acb2d2280" path="/var/lib/kubelet/pods/a18c02c8-1a51-4dcf-aeef-696acb2d2280/volumes" Oct 30 17:27:50.934476 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:50.934454 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efbed66f-95b0-4fd7-af06-388840a2a364" path="/var/lib/kubelet/pods/efbed66f-95b0-4fd7-af06-388840a2a364/volumes" Oct 30 17:27:54.412531 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:54.412485 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b548074-mmcx6"] Oct 30 17:27:54.413229 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:54.412968 4493 cpu_manager.go:414] "containerMap: removing container" podUID="efbed66f-95b0-4fd7-af06-388840a2a364" containerName="istio-proxy" Oct 30 17:27:54.413229 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:54.412981 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efbed66f-95b0-4fd7-af06-388840a2a364" containerName="istio-proxy" Oct 30 17:27:54.413229 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:54.412995 4493 cpu_manager.go:414] "containerMap: removing container" podUID="2a5999d8-6e17-496d-b28c-e3d6ac908b07" containerName="revoke-keys" Oct 30 17:27:54.413229 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:54.413000 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a5999d8-6e17-496d-b28c-e3d6ac908b07" containerName="revoke-keys" Oct 30 17:27:54.413229 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:54.413015 4493 cpu_manager.go:414] "containerMap: removing container" podUID="a18c02c8-1a51-4dcf-aeef-696acb2d2280" containerName="istio-proxy" Oct 30 17:27:54.413229 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:54.413022 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a18c02c8-1a51-4dcf-aeef-696acb2d2280" containerName="istio-proxy" Oct 30 17:27:54.413229 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:54.413031 4493 cpu_manager.go:414] "containerMap: removing container" podUID="38ebec4d-5fbf-4555-a697-4f3084de0d26" containerName="revoke-keys" Oct 30 17:27:54.413229 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:54.413036 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38ebec4d-5fbf-4555-a697-4f3084de0d26" containerName="revoke-keys" Oct 30 17:27:54.413229 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:54.413109 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="efbed66f-95b0-4fd7-af06-388840a2a364" containerName="istio-proxy" Oct 30 17:27:54.413229 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:54.413116 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a5999d8-6e17-496d-b28c-e3d6ac908b07" containerName="revoke-keys" Oct 30 17:27:54.413229 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:54.413125 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="a18c02c8-1a51-4dcf-aeef-696acb2d2280" containerName="istio-proxy" Oct 30 17:27:54.413229 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:54.413131 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="38ebec4d-5fbf-4555-a697-4f3084de0d26" containerName="revoke-keys" Oct 30 17:27:54.413229 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:54.413139 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="a18c02c8-1a51-4dcf-aeef-696acb2d2280" containerName="istio-proxy" Oct 30 17:27:54.417355 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:54.417333 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b548074-mmcx6" Oct 30 17:27:54.437507 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:54.437176 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba183451-tgmlj"] Oct 30 17:27:54.437979 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:54.437951 4493 cpu_manager.go:414] "containerMap: removing container" podUID="a18c02c8-1a51-4dcf-aeef-696acb2d2280" containerName="istio-proxy" Oct 30 17:27:54.437979 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:54.437977 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a18c02c8-1a51-4dcf-aeef-696acb2d2280" containerName="istio-proxy" Oct 30 17:27:54.442059 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:54.442037 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba183451-tgmlj" Oct 30 17:27:54.443515 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:54.443049 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:27:54.443515 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:54.443158 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:27:54.443515 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:54.443228 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:27:54.446309 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:54.446287 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 17:27:54.451822 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:54.451771 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b548074-mmcx6"] Oct 30 17:27:54.463634 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:54.463600 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba183451-tgmlj"] Oct 30 17:27:54.524525 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:54.524475 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cc934bb-7ead-421d-88c1-13e1bc5e93d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b548074-mmcx6\" (UID: \"9cc934bb-7ead-421d-88c1-13e1bc5e93d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b548074-mmcx6" Oct 30 17:27:54.524759 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:54.524570 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hmlg\" (UniqueName: \"kubernetes.io/projected/9cc934bb-7ead-421d-88c1-13e1bc5e93d5-kube-api-access-5hmlg\") pod \"maas-api-revoke-sub-5b548074-mmcx6\" (UID: \"9cc934bb-7ead-421d-88c1-13e1bc5e93d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b548074-mmcx6" Oct 30 17:27:54.524759 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:54.524607 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58c27031-67a6-4647-9be5-6f8a0e2adb33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba183451-tgmlj\" (UID: \"58c27031-67a6-4647-9be5-6f8a0e2adb33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba183451-tgmlj" Oct 30 17:27:54.524759 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:54.524631 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7xrp\" (UniqueName: \"kubernetes.io/projected/58c27031-67a6-4647-9be5-6f8a0e2adb33-kube-api-access-l7xrp\") pod \"maas-api-revoke-sub-ba183451-tgmlj\" (UID: \"58c27031-67a6-4647-9be5-6f8a0e2adb33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba183451-tgmlj" Oct 30 17:27:54.524759 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:54.524716 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58c27031-67a6-4647-9be5-6f8a0e2adb33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba183451-tgmlj\" (UID: \"58c27031-67a6-4647-9be5-6f8a0e2adb33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba183451-tgmlj" Oct 30 17:27:54.524759 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:54.524755 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cc934bb-7ead-421d-88c1-13e1bc5e93d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b548074-mmcx6\" (UID: \"9cc934bb-7ead-421d-88c1-13e1bc5e93d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b548074-mmcx6" Oct 30 17:27:54.625745 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:54.625698 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cc934bb-7ead-421d-88c1-13e1bc5e93d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b548074-mmcx6\" (UID: \"9cc934bb-7ead-421d-88c1-13e1bc5e93d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b548074-mmcx6" Oct 30 17:27:54.625964 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:54.625777 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hmlg\" (UniqueName: \"kubernetes.io/projected/9cc934bb-7ead-421d-88c1-13e1bc5e93d5-kube-api-access-5hmlg\") pod \"maas-api-revoke-sub-5b548074-mmcx6\" (UID: \"9cc934bb-7ead-421d-88c1-13e1bc5e93d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b548074-mmcx6" Oct 30 17:27:54.625964 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:54.625848 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58c27031-67a6-4647-9be5-6f8a0e2adb33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba183451-tgmlj\" (UID: \"58c27031-67a6-4647-9be5-6f8a0e2adb33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba183451-tgmlj" Oct 30 17:27:54.625964 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:54.625871 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7xrp\" (UniqueName: \"kubernetes.io/projected/58c27031-67a6-4647-9be5-6f8a0e2adb33-kube-api-access-l7xrp\") pod \"maas-api-revoke-sub-ba183451-tgmlj\" (UID: \"58c27031-67a6-4647-9be5-6f8a0e2adb33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba183451-tgmlj" Oct 30 17:27:54.626130 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:54.625975 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58c27031-67a6-4647-9be5-6f8a0e2adb33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba183451-tgmlj\" (UID: \"58c27031-67a6-4647-9be5-6f8a0e2adb33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba183451-tgmlj" Oct 30 17:27:54.626130 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:54.626018 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cc934bb-7ead-421d-88c1-13e1bc5e93d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b548074-mmcx6\" (UID: \"9cc934bb-7ead-421d-88c1-13e1bc5e93d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b548074-mmcx6" Oct 30 17:27:54.626835 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:54.626810 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cc934bb-7ead-421d-88c1-13e1bc5e93d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b548074-mmcx6\" (UID: \"9cc934bb-7ead-421d-88c1-13e1bc5e93d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b548074-mmcx6" Oct 30 17:27:54.627286 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:54.627233 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58c27031-67a6-4647-9be5-6f8a0e2adb33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba183451-tgmlj\" (UID: \"58c27031-67a6-4647-9be5-6f8a0e2adb33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba183451-tgmlj" Oct 30 17:27:54.629412 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:54.629387 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58c27031-67a6-4647-9be5-6f8a0e2adb33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba183451-tgmlj\" (UID: \"58c27031-67a6-4647-9be5-6f8a0e2adb33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba183451-tgmlj" Oct 30 17:27:54.629690 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:54.629669 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cc934bb-7ead-421d-88c1-13e1bc5e93d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b548074-mmcx6\" (UID: \"9cc934bb-7ead-421d-88c1-13e1bc5e93d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b548074-mmcx6" Oct 30 17:27:54.651166 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:54.651136 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hmlg\" (UniqueName: \"kubernetes.io/projected/9cc934bb-7ead-421d-88c1-13e1bc5e93d5-kube-api-access-5hmlg\") pod \"maas-api-revoke-sub-5b548074-mmcx6\" (UID: \"9cc934bb-7ead-421d-88c1-13e1bc5e93d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b548074-mmcx6" Oct 30 17:27:54.651315 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:54.651243 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7xrp\" (UniqueName: \"kubernetes.io/projected/58c27031-67a6-4647-9be5-6f8a0e2adb33-kube-api-access-l7xrp\") pod \"maas-api-revoke-sub-ba183451-tgmlj\" (UID: \"58c27031-67a6-4647-9be5-6f8a0e2adb33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba183451-tgmlj" Oct 30 17:27:54.730492 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:54.730411 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b548074-mmcx6" Oct 30 17:27:54.757978 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:54.757940 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba183451-tgmlj" Oct 30 17:27:55.042808 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:55.042706 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-56v9s_d3a1c417-2698-4462-b7ff-1d2e66d7a10f/ovn-acl-logging/0.log" Oct 30 17:27:55.045611 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:55.045591 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-56v9s_d3a1c417-2698-4462-b7ff-1d2e66d7a10f/ovn-acl-logging/0.log" Oct 30 17:27:55.056868 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:55.056834 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b548074-mmcx6"] Oct 30 17:27:55.074810 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:27:55.072305 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cc934bb_7ead_421d_88c1_13e1bc5e93d5.slice/crio-250eea3ab4d31a331a7bc5f2608e97e3cf7c34c8adbcb043c3ee824a9c6e5382 WatchSource:0}: Error finding container 250eea3ab4d31a331a7bc5f2608e97e3cf7c34c8adbcb043c3ee824a9c6e5382: Status 404 returned error can't find the container with id 250eea3ab4d31a331a7bc5f2608e97e3cf7c34c8adbcb043c3ee824a9c6e5382 Oct 30 17:27:55.129134 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:55.129095 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba183451-tgmlj"] Oct 30 17:27:55.140890 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:55.140848 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8"] Oct 30 17:27:55.141113 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:55.141076 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8" podUID="e2c02109-c263-42aa-bdad-a995f3577d99" containerName="istio-proxy" containerID="cri-o://e2dbe35a076a50a71f243b174960bfd91b291ca96001f7ddf20c71fd56fa4ccf" gracePeriod=30 Oct 30 17:27:55.306249 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:55.306160 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b548074-mmcx6" event={"ID":"9cc934bb-7ead-421d-88c1-13e1bc5e93d5","Type":"ContainerStarted","Data":"250eea3ab4d31a331a7bc5f2608e97e3cf7c34c8adbcb043c3ee824a9c6e5382"} Oct 30 17:27:55.307799 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:55.307756 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba183451-tgmlj" event={"ID":"58c27031-67a6-4647-9be5-6f8a0e2adb33","Type":"ContainerStarted","Data":"ad68d701a059d34cb6dbf256d80d778fbc329c9edc8ea05ca9b9f8387bf64066"} Oct 30 17:27:56.317299 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:56.317209 4493 generic.go:356] "Generic (PLEG): container finished" podID="9cc934bb-7ead-421d-88c1-13e1bc5e93d5" containerID="9f8729294a1fefa103549f200b92eb7a1fbbc90b86edeefe4a30e5ea32e55596" exitCode=0 Oct 30 17:27:56.318092 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:56.317298 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b548074-mmcx6" event={"ID":"9cc934bb-7ead-421d-88c1-13e1bc5e93d5","Type":"ContainerDied","Data":"9f8729294a1fefa103549f200b92eb7a1fbbc90b86edeefe4a30e5ea32e55596"} Oct 30 17:27:56.319133 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:56.319099 4493 generic.go:356] "Generic (PLEG): container finished" podID="58c27031-67a6-4647-9be5-6f8a0e2adb33" containerID="610dd0f113ed0c4b5bf49e436aa9c604cafb5b461c2e4c3f52818426f82f80cc" exitCode=0 Oct 30 17:27:56.319302 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:56.319274 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba183451-tgmlj" event={"ID":"58c27031-67a6-4647-9be5-6f8a0e2adb33","Type":"ContainerDied","Data":"610dd0f113ed0c4b5bf49e436aa9c604cafb5b461c2e4c3f52818426f82f80cc"} Oct 30 17:27:57.050040 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:57.050004 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95cca0e3-dlzp4"] Oct 30 17:27:57.061552 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:57.061519 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95cca0e3-dlzp4"] Oct 30 17:27:57.481766 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:57.481741 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b548074-mmcx6" Oct 30 17:27:57.519705 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:57.519676 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba183451-tgmlj" Oct 30 17:27:57.570884 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:57.570850 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cc934bb-7ead-421d-88c1-13e1bc5e93d5-kube-api-access-5hmlg\" (UniqueName: \"kubernetes.io/projected/9cc934bb-7ead-421d-88c1-13e1bc5e93d5-kube-api-access-5hmlg\") pod \"9cc934bb-7ead-421d-88c1-13e1bc5e93d5\" (UID: \"9cc934bb-7ead-421d-88c1-13e1bc5e93d5\") " Oct 30 17:27:57.571089 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:57.570918 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58c27031-67a6-4647-9be5-6f8a0e2adb33-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58c27031-67a6-4647-9be5-6f8a0e2adb33-maas-api-xks-ca\") pod \"58c27031-67a6-4647-9be5-6f8a0e2adb33\" (UID: \"58c27031-67a6-4647-9be5-6f8a0e2adb33\") " Oct 30 17:27:57.571089 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:57.570976 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58c27031-67a6-4647-9be5-6f8a0e2adb33-kube-api-access-l7xrp\" (UniqueName: \"kubernetes.io/projected/58c27031-67a6-4647-9be5-6f8a0e2adb33-kube-api-access-l7xrp\") pod \"58c27031-67a6-4647-9be5-6f8a0e2adb33\" (UID: \"58c27031-67a6-4647-9be5-6f8a0e2adb33\") " Oct 30 17:27:57.571089 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:57.571051 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58c27031-67a6-4647-9be5-6f8a0e2adb33-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58c27031-67a6-4647-9be5-6f8a0e2adb33-maas-api-openshift-ca\") pod \"58c27031-67a6-4647-9be5-6f8a0e2adb33\" (UID: \"58c27031-67a6-4647-9be5-6f8a0e2adb33\") " Oct 30 17:27:57.571285 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:57.571118 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9cc934bb-7ead-421d-88c1-13e1bc5e93d5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cc934bb-7ead-421d-88c1-13e1bc5e93d5-maas-api-xks-ca\") pod \"9cc934bb-7ead-421d-88c1-13e1bc5e93d5\" (UID: \"9cc934bb-7ead-421d-88c1-13e1bc5e93d5\") " Oct 30 17:27:57.571285 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:57.571225 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cc934bb-7ead-421d-88c1-13e1bc5e93d5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cc934bb-7ead-421d-88c1-13e1bc5e93d5-maas-api-openshift-ca\") pod \"9cc934bb-7ead-421d-88c1-13e1bc5e93d5\" (UID: \"9cc934bb-7ead-421d-88c1-13e1bc5e93d5\") " Oct 30 17:27:57.571588 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:57.571546 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58c27031-67a6-4647-9be5-6f8a0e2adb33-maas-api-openshift-ca" pod "58c27031-67a6-4647-9be5-6f8a0e2adb33" (UID: "58c27031-67a6-4647-9be5-6f8a0e2adb33"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:27:57.572185 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:57.572147 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cc934bb-7ead-421d-88c1-13e1bc5e93d5-maas-api-openshift-ca" pod "9cc934bb-7ead-421d-88c1-13e1bc5e93d5" (UID: "9cc934bb-7ead-421d-88c1-13e1bc5e93d5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:27:57.574085 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:57.574064 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cc934bb-7ead-421d-88c1-13e1bc5e93d5-kube-api-access-5hmlg" pod "9cc934bb-7ead-421d-88c1-13e1bc5e93d5" (UID: "9cc934bb-7ead-421d-88c1-13e1bc5e93d5"). InnerVolumeSpecName "kube-api-access-5hmlg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:27:57.574200 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:57.574138 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58c27031-67a6-4647-9be5-6f8a0e2adb33-maas-api-xks-ca" pod "58c27031-67a6-4647-9be5-6f8a0e2adb33" (UID: "58c27031-67a6-4647-9be5-6f8a0e2adb33"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:27:57.574433 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:57.574409 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cc934bb-7ead-421d-88c1-13e1bc5e93d5-maas-api-xks-ca" pod "9cc934bb-7ead-421d-88c1-13e1bc5e93d5" (UID: "9cc934bb-7ead-421d-88c1-13e1bc5e93d5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:27:57.574490 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:57.574430 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58c27031-67a6-4647-9be5-6f8a0e2adb33-kube-api-access-l7xrp" pod "58c27031-67a6-4647-9be5-6f8a0e2adb33" (UID: "58c27031-67a6-4647-9be5-6f8a0e2adb33"). InnerVolumeSpecName "kube-api-access-l7xrp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:27:57.673065 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:57.673031 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cc934bb-7ead-421d-88c1-13e1bc5e93d5-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:57.673065 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:57.673059 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hmlg\" (UniqueName: \"kubernetes.io/projected/9cc934bb-7ead-421d-88c1-13e1bc5e93d5-kube-api-access-5hmlg\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:57.673065 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:57.673072 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58c27031-67a6-4647-9be5-6f8a0e2adb33-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:57.673318 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:57.673085 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7xrp\" (UniqueName: \"kubernetes.io/projected/58c27031-67a6-4647-9be5-6f8a0e2adb33-kube-api-access-l7xrp\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:57.673318 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:57.673093 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58c27031-67a6-4647-9be5-6f8a0e2adb33-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:57.673318 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:57.673101 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cc934bb-7ead-421d-88c1-13e1bc5e93d5-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:27:58.034484 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:58.034448 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6e52c50-9qz44"] Oct 30 17:27:58.046286 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:58.046259 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6e52c50-9qz44"] Oct 30 17:27:58.362111 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:58.362020 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b548074-mmcx6" Oct 30 17:27:58.362347 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:58.362314 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b548074-mmcx6" event={"ID":"9cc934bb-7ead-421d-88c1-13e1bc5e93d5","Type":"ContainerDied","Data":"250eea3ab4d31a331a7bc5f2608e97e3cf7c34c8adbcb043c3ee824a9c6e5382"} Oct 30 17:27:58.362534 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:58.362515 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="250eea3ab4d31a331a7bc5f2608e97e3cf7c34c8adbcb043c3ee824a9c6e5382" Oct 30 17:27:58.364220 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:58.364197 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba183451-tgmlj" Oct 30 17:27:58.364803 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:58.364265 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba183451-tgmlj" event={"ID":"58c27031-67a6-4647-9be5-6f8a0e2adb33","Type":"ContainerDied","Data":"ad68d701a059d34cb6dbf256d80d778fbc329c9edc8ea05ca9b9f8387bf64066"} Oct 30 17:27:58.364942 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:58.364841 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad68d701a059d34cb6dbf256d80d778fbc329c9edc8ea05ca9b9f8387bf64066" Oct 30 17:27:58.474399 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:58.474362 4493 scope.go:122] "RemoveContainer" containerID="60a32d3a3f764df7376ec313753e3141af20e73724baea11cb6a4e6f215178f5" Oct 30 17:27:58.486498 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:58.486473 4493 scope.go:122] "RemoveContainer" containerID="e68e01ba784a12f2a8823555b0a1d6d474c9c4857fe5bfbed1d5c7d4b68e06fd" Oct 30 17:27:58.497590 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:58.497564 4493 scope.go:122] "RemoveContainer" containerID="93f19a6d71688b2a68119cdcc4d935dc27976fbbfc73b7a63f448f64a002d167" Oct 30 17:27:58.932060 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:58.932016 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1dabcf42-41ef-4696-94d4-363543a18267" path="/var/lib/kubelet/pods/1dabcf42-41ef-4696-94d4-363543a18267/volumes" Oct 30 17:27:58.932497 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:58.932475 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5285914-ecf4-4ae4-b2bc-da1ae7583379" path="/var/lib/kubelet/pods/b5285914-ecf4-4ae4-b2bc-da1ae7583379/volumes" Oct 30 17:27:59.032283 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:59.032255 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb312a11-6d8r8"] Oct 30 17:27:59.042959 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:59.042934 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb312a11-6d8r8"] Oct 30 17:27:59.118173 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:59.118141 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f94fb5-j22h6"] Oct 30 17:27:59.122813 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:27:59.122762 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-f94fb5-j22h6"] Oct 30 17:28:00.345935 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:28:00.345897 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 17:28:00.388079 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:00.388051 4493 generic.go:356] "Generic (PLEG): container finished" podID="e2c02109-c263-42aa-bdad-a995f3577d99" containerID="e2dbe35a076a50a71f243b174960bfd91b291ca96001f7ddf20c71fd56fa4ccf" exitCode=0 Oct 30 17:28:00.388079 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:00.388086 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8" event={"ID":"e2c02109-c263-42aa-bdad-a995f3577d99","Type":"ContainerDied","Data":"e2dbe35a076a50a71f243b174960bfd91b291ca96001f7ddf20c71fd56fa4ccf"} Oct 30 17:28:00.420077 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:00.420027 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8" Oct 30 17:28:00.503086 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:00.503052 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2c02109-c263-42aa-bdad-a995f3577d99-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2c02109-c263-42aa-bdad-a995f3577d99-istiod-ca-cert\") pod \"e2c02109-c263-42aa-bdad-a995f3577d99\" (UID: \"e2c02109-c263-42aa-bdad-a995f3577d99\") " Oct 30 17:28:00.503288 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:00.503111 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2c02109-c263-42aa-bdad-a995f3577d99-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2c02109-c263-42aa-bdad-a995f3577d99-credential-socket\") pod \"e2c02109-c263-42aa-bdad-a995f3577d99\" (UID: \"e2c02109-c263-42aa-bdad-a995f3577d99\") " Oct 30 17:28:00.503288 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:00.503148 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e2c02109-c263-42aa-bdad-a995f3577d99-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2c02109-c263-42aa-bdad-a995f3577d99-istio-podinfo\") pod \"e2c02109-c263-42aa-bdad-a995f3577d99\" (UID: \"e2c02109-c263-42aa-bdad-a995f3577d99\") " Oct 30 17:28:00.503288 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:00.503172 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2c02109-c263-42aa-bdad-a995f3577d99-kube-api-access-dmnfj\" (UniqueName: \"kubernetes.io/projected/e2c02109-c263-42aa-bdad-a995f3577d99-kube-api-access-dmnfj\") pod \"e2c02109-c263-42aa-bdad-a995f3577d99\" (UID: \"e2c02109-c263-42aa-bdad-a995f3577d99\") " Oct 30 17:28:00.503288 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:00.503237 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2c02109-c263-42aa-bdad-a995f3577d99-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2c02109-c263-42aa-bdad-a995f3577d99-workload-certs\") pod \"e2c02109-c263-42aa-bdad-a995f3577d99\" (UID: \"e2c02109-c263-42aa-bdad-a995f3577d99\") " Oct 30 17:28:00.503288 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:00.503272 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2c02109-c263-42aa-bdad-a995f3577d99-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2c02109-c263-42aa-bdad-a995f3577d99-workload-socket\") pod \"e2c02109-c263-42aa-bdad-a995f3577d99\" (UID: \"e2c02109-c263-42aa-bdad-a995f3577d99\") " Oct 30 17:28:00.503600 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:00.503315 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2c02109-c263-42aa-bdad-a995f3577d99-istio-token\" (UniqueName: \"kubernetes.io/projected/e2c02109-c263-42aa-bdad-a995f3577d99-istio-token\") pod \"e2c02109-c263-42aa-bdad-a995f3577d99\" (UID: \"e2c02109-c263-42aa-bdad-a995f3577d99\") " Oct 30 17:28:00.503600 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:00.503361 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2c02109-c263-42aa-bdad-a995f3577d99-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2c02109-c263-42aa-bdad-a995f3577d99-istio-envoy\") pod \"e2c02109-c263-42aa-bdad-a995f3577d99\" (UID: \"e2c02109-c263-42aa-bdad-a995f3577d99\") " Oct 30 17:28:00.503600 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:00.503394 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2c02109-c263-42aa-bdad-a995f3577d99-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2c02109-c263-42aa-bdad-a995f3577d99-istio-data\") pod \"e2c02109-c263-42aa-bdad-a995f3577d99\" (UID: \"e2c02109-c263-42aa-bdad-a995f3577d99\") " Oct 30 17:28:00.503600 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:00.503505 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2c02109-c263-42aa-bdad-a995f3577d99-credential-socket" pod "e2c02109-c263-42aa-bdad-a995f3577d99" (UID: "e2c02109-c263-42aa-bdad-a995f3577d99"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:28:00.503600 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:00.503532 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2c02109-c263-42aa-bdad-a995f3577d99-istiod-ca-cert" pod "e2c02109-c263-42aa-bdad-a995f3577d99" (UID: "e2c02109-c263-42aa-bdad-a995f3577d99"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:28:00.503948 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:00.503724 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2c02109-c263-42aa-bdad-a995f3577d99-workload-socket" pod "e2c02109-c263-42aa-bdad-a995f3577d99" (UID: "e2c02109-c263-42aa-bdad-a995f3577d99"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:28:00.503948 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:00.503762 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2c02109-c263-42aa-bdad-a995f3577d99-workload-certs" pod "e2c02109-c263-42aa-bdad-a995f3577d99" (UID: "e2c02109-c263-42aa-bdad-a995f3577d99"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:28:00.503948 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:00.503888 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2c02109-c263-42aa-bdad-a995f3577d99-istiod-ca-cert\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:28:00.503948 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:00.503908 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2c02109-c263-42aa-bdad-a995f3577d99-credential-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:28:00.503948 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:00.503922 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2c02109-c263-42aa-bdad-a995f3577d99-workload-certs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:28:00.504223 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:00.503935 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2c02109-c263-42aa-bdad-a995f3577d99-workload-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:28:00.504223 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:00.504166 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2c02109-c263-42aa-bdad-a995f3577d99-istio-data" pod "e2c02109-c263-42aa-bdad-a995f3577d99" (UID: "e2c02109-c263-42aa-bdad-a995f3577d99"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:28:00.505703 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:00.505682 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e2c02109-c263-42aa-bdad-a995f3577d99-istio-podinfo" pod "e2c02109-c263-42aa-bdad-a995f3577d99" (UID: "e2c02109-c263-42aa-bdad-a995f3577d99"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:28:00.506147 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:00.506122 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2c02109-c263-42aa-bdad-a995f3577d99-istio-token" pod "e2c02109-c263-42aa-bdad-a995f3577d99" (UID: "e2c02109-c263-42aa-bdad-a995f3577d99"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:28:00.506516 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:00.506490 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2c02109-c263-42aa-bdad-a995f3577d99-kube-api-access-dmnfj" pod "e2c02109-c263-42aa-bdad-a995f3577d99" (UID: "e2c02109-c263-42aa-bdad-a995f3577d99"). InnerVolumeSpecName "kube-api-access-dmnfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:28:00.506611 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:00.506498 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2c02109-c263-42aa-bdad-a995f3577d99-istio-envoy" pod "e2c02109-c263-42aa-bdad-a995f3577d99" (UID: "e2c02109-c263-42aa-bdad-a995f3577d99"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:28:00.605460 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:00.605428 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2c02109-c263-42aa-bdad-a995f3577d99-istio-token\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:28:00.605460 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:00.605455 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2c02109-c263-42aa-bdad-a995f3577d99-istio-envoy\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:28:00.605460 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:00.605464 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2c02109-c263-42aa-bdad-a995f3577d99-istio-data\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:28:00.605677 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:00.605472 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2c02109-c263-42aa-bdad-a995f3577d99-istio-podinfo\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:28:00.605677 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:00.605481 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmnfj\" (UniqueName: \"kubernetes.io/projected/e2c02109-c263-42aa-bdad-a995f3577d99-kube-api-access-dmnfj\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:28:00.932932 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:00.932889 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38ebec4d-5fbf-4555-a697-4f3084de0d26" path="/var/lib/kubelet/pods/38ebec4d-5fbf-4555-a697-4f3084de0d26/volumes" Oct 30 17:28:00.933420 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:00.933399 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9581d15c-a121-4c2a-9d55-a2cf3afbe0b0" path="/var/lib/kubelet/pods/9581d15c-a121-4c2a-9d55-a2cf3afbe0b0/volumes" Oct 30 17:28:00.972815 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:00.972746 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" podUID="18b72648-f523-405e-b5cd-1266ff13f8f2" containerName="main" containerID="cri-o://29896e9cf036eebbca0f710d59c4b8f028c45b9a194c56e479f32a5625dd9fec" gracePeriod=45 Oct 30 17:28:00.987288 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:00.987162 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" podUID="08baad08-cd72-4305-b92d-c11a84ef276f" containerName="main" containerID="cri-o://a0f7895786747598a42513c8e971f1f7741fa9b45b156dbab73bb4052578a447" gracePeriod=45 Oct 30 17:28:01.111950 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:01.111907 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" podUID="18b72648-f523-405e-b5cd-1266ff13f8f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": dial tcp 10.134.0.62:8000: connect: connection refused" Oct 30 17:28:01.298536 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:01.298508 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" Oct 30 17:28:01.302988 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:01.302959 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" Oct 30 17:28:01.401573 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:01.401534 4493 generic.go:356] "Generic (PLEG): container finished" podID="18b72648-f523-405e-b5cd-1266ff13f8f2" containerID="29896e9cf036eebbca0f710d59c4b8f028c45b9a194c56e479f32a5625dd9fec" exitCode=0 Oct 30 17:28:01.401573 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:01.401583 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" Oct 30 17:28:01.402094 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:01.401616 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" event={"ID":"18b72648-f523-405e-b5cd-1266ff13f8f2","Type":"ContainerDied","Data":"29896e9cf036eebbca0f710d59c4b8f028c45b9a194c56e479f32a5625dd9fec"} Oct 30 17:28:01.402094 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:01.401674 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642" event={"ID":"18b72648-f523-405e-b5cd-1266ff13f8f2","Type":"ContainerDied","Data":"78379a38696f47352d213ee97a9c0232e0a811be5cafd9248036df3681c8c79d"} Oct 30 17:28:01.402094 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:01.401701 4493 scope.go:122] "RemoveContainer" containerID="29896e9cf036eebbca0f710d59c4b8f028c45b9a194c56e479f32a5625dd9fec" Oct 30 17:28:01.403514 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:01.403489 4493 generic.go:356] "Generic (PLEG): container finished" podID="08baad08-cd72-4305-b92d-c11a84ef276f" containerID="a0f7895786747598a42513c8e971f1f7741fa9b45b156dbab73bb4052578a447" exitCode=0 Oct 30 17:28:01.403639 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:01.403535 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" Oct 30 17:28:01.403639 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:01.403571 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" event={"ID":"08baad08-cd72-4305-b92d-c11a84ef276f","Type":"ContainerDied","Data":"a0f7895786747598a42513c8e971f1f7741fa9b45b156dbab73bb4052578a447"} Oct 30 17:28:01.403639 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:01.403614 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" event={"ID":"08baad08-cd72-4305-b92d-c11a84ef276f","Type":"ContainerDied","Data":"df1164b5832f0fd74a25422669164ac01c3e6ebad8ec25d5cc12654a8de3e4b9"} Oct 30 17:28:01.405839 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:01.405814 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8" event={"ID":"e2c02109-c263-42aa-bdad-a995f3577d99","Type":"ContainerDied","Data":"4746ca4171044ae934f02bd8c484ea4e5f8f8869820e04ef9e8f75e556d9be64"} Oct 30 17:28:01.406003 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:01.405922 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8" Oct 30 17:28:01.414117 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:01.414094 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18b72648-f523-405e-b5cd-1266ff13f8f2-tls-certs\" (UniqueName: \"kubernetes.io/secret/18b72648-f523-405e-b5cd-1266ff13f8f2-tls-certs\") pod \"18b72648-f523-405e-b5cd-1266ff13f8f2\" (UID: \"18b72648-f523-405e-b5cd-1266ff13f8f2\") " Oct 30 17:28:01.414275 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:01.414259 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08baad08-cd72-4305-b92d-c11a84ef276f-home\" (UniqueName: \"kubernetes.io/empty-dir/08baad08-cd72-4305-b92d-c11a84ef276f-home\") pod \"08baad08-cd72-4305-b92d-c11a84ef276f\" (UID: \"08baad08-cd72-4305-b92d-c11a84ef276f\") " Oct 30 17:28:01.414403 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:01.414390 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08baad08-cd72-4305-b92d-c11a84ef276f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/08baad08-cd72-4305-b92d-c11a84ef276f-dshm\") pod \"08baad08-cd72-4305-b92d-c11a84ef276f\" (UID: \"08baad08-cd72-4305-b92d-c11a84ef276f\") " Oct 30 17:28:01.414537 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:01.414512 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08baad08-cd72-4305-b92d-c11a84ef276f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08baad08-cd72-4305-b92d-c11a84ef276f-model-cache\") pod \"08baad08-cd72-4305-b92d-c11a84ef276f\" (UID: \"08baad08-cd72-4305-b92d-c11a84ef276f\") " Oct 30 17:28:01.414663 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:01.414552 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18b72648-f523-405e-b5cd-1266ff13f8f2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18b72648-f523-405e-b5cd-1266ff13f8f2-model-cache\") pod \"18b72648-f523-405e-b5cd-1266ff13f8f2\" (UID: \"18b72648-f523-405e-b5cd-1266ff13f8f2\") " Oct 30 17:28:01.414663 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:01.414609 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08baad08-cd72-4305-b92d-c11a84ef276f-tls-certs\" (UniqueName: \"kubernetes.io/secret/08baad08-cd72-4305-b92d-c11a84ef276f-tls-certs\") pod \"08baad08-cd72-4305-b92d-c11a84ef276f\" (UID: \"08baad08-cd72-4305-b92d-c11a84ef276f\") " Oct 30 17:28:01.414663 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:01.414639 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18b72648-f523-405e-b5cd-1266ff13f8f2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/18b72648-f523-405e-b5cd-1266ff13f8f2-dshm\") pod \"18b72648-f523-405e-b5cd-1266ff13f8f2\" (UID: \"18b72648-f523-405e-b5cd-1266ff13f8f2\") " Oct 30 17:28:01.414906 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:01.414667 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18b72648-f523-405e-b5cd-1266ff13f8f2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18b72648-f523-405e-b5cd-1266ff13f8f2-tmp-dir\") pod \"18b72648-f523-405e-b5cd-1266ff13f8f2\" (UID: \"18b72648-f523-405e-b5cd-1266ff13f8f2\") " Oct 30 17:28:01.414906 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:01.414728 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08baad08-cd72-4305-b92d-c11a84ef276f-kube-api-access-vj7hs\" (UniqueName: \"kubernetes.io/projected/08baad08-cd72-4305-b92d-c11a84ef276f-kube-api-access-vj7hs\") pod \"08baad08-cd72-4305-b92d-c11a84ef276f\" (UID: \"08baad08-cd72-4305-b92d-c11a84ef276f\") " Oct 30 17:28:01.414906 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:01.414873 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18b72648-f523-405e-b5cd-1266ff13f8f2-kube-api-access-vbzwm\" (UniqueName: \"kubernetes.io/projected/18b72648-f523-405e-b5cd-1266ff13f8f2-kube-api-access-vbzwm\") pod \"18b72648-f523-405e-b5cd-1266ff13f8f2\" (UID: \"18b72648-f523-405e-b5cd-1266ff13f8f2\") " Oct 30 17:28:01.415100 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:01.414925 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08baad08-cd72-4305-b92d-c11a84ef276f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08baad08-cd72-4305-b92d-c11a84ef276f-tmp-dir\") pod \"08baad08-cd72-4305-b92d-c11a84ef276f\" (UID: \"08baad08-cd72-4305-b92d-c11a84ef276f\") " Oct 30 17:28:01.415100 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:01.414999 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18b72648-f523-405e-b5cd-1266ff13f8f2-home\" (UniqueName: \"kubernetes.io/empty-dir/18b72648-f523-405e-b5cd-1266ff13f8f2-home\") pod \"18b72648-f523-405e-b5cd-1266ff13f8f2\" (UID: \"18b72648-f523-405e-b5cd-1266ff13f8f2\") " Oct 30 17:28:01.415610 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:01.415586 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18b72648-f523-405e-b5cd-1266ff13f8f2-home" pod "18b72648-f523-405e-b5cd-1266ff13f8f2" (UID: "18b72648-f523-405e-b5cd-1266ff13f8f2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:28:01.419088 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:01.419058 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18b72648-f523-405e-b5cd-1266ff13f8f2-tls-certs" pod "18b72648-f523-405e-b5cd-1266ff13f8f2" (UID: "18b72648-f523-405e-b5cd-1266ff13f8f2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:28:01.419195 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:01.419101 4493 scope.go:122] "RemoveContainer" containerID="29896e9cf036eebbca0f710d59c4b8f028c45b9a194c56e479f32a5625dd9fec" Oct 30 17:28:01.419584 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:01.419555 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08baad08-cd72-4305-b92d-c11a84ef276f-home" pod "08baad08-cd72-4305-b92d-c11a84ef276f" (UID: "08baad08-cd72-4305-b92d-c11a84ef276f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:28:01.420983 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:01.419757 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18b72648-f523-405e-b5cd-1266ff13f8f2-tmp-dir" pod "18b72648-f523-405e-b5cd-1266ff13f8f2" (UID: "18b72648-f523-405e-b5cd-1266ff13f8f2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:28:01.420983 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:01.419973 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08baad08-cd72-4305-b92d-c11a84ef276f-tmp-dir" pod "08baad08-cd72-4305-b92d-c11a84ef276f" (UID: "08baad08-cd72-4305-b92d-c11a84ef276f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:28:01.420983 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:28:01.420146 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29896e9cf036eebbca0f710d59c4b8f028c45b9a194c56e479f32a5625dd9fec\": container with ID starting with 29896e9cf036eebbca0f710d59c4b8f028c45b9a194c56e479f32a5625dd9fec not found: ID does not exist" containerID="29896e9cf036eebbca0f710d59c4b8f028c45b9a194c56e479f32a5625dd9fec" Oct 30 17:28:01.420983 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:01.420180 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29896e9cf036eebbca0f710d59c4b8f028c45b9a194c56e479f32a5625dd9fec"} err="failed to get container status \"29896e9cf036eebbca0f710d59c4b8f028c45b9a194c56e479f32a5625dd9fec\": rpc error: code = NotFound desc = could not find container \"29896e9cf036eebbca0f710d59c4b8f028c45b9a194c56e479f32a5625dd9fec\": container with ID starting with 29896e9cf036eebbca0f710d59c4b8f028c45b9a194c56e479f32a5625dd9fec not found: ID does not exist" Oct 30 17:28:01.420983 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:01.420203 4493 scope.go:122] "RemoveContainer" containerID="a0f7895786747598a42513c8e971f1f7741fa9b45b156dbab73bb4052578a447" Oct 30 17:28:01.427354 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:01.426171 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08baad08-cd72-4305-b92d-c11a84ef276f-model-cache" pod "08baad08-cd72-4305-b92d-c11a84ef276f" (UID: "08baad08-cd72-4305-b92d-c11a84ef276f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:28:01.428059 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:01.427871 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18b72648-f523-405e-b5cd-1266ff13f8f2-dshm" pod "18b72648-f523-405e-b5cd-1266ff13f8f2" (UID: "18b72648-f523-405e-b5cd-1266ff13f8f2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:28:01.428477 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:01.428223 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18b72648-f523-405e-b5cd-1266ff13f8f2-model-cache" pod "18b72648-f523-405e-b5cd-1266ff13f8f2" (UID: "18b72648-f523-405e-b5cd-1266ff13f8f2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:28:01.429212 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:01.428975 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08baad08-cd72-4305-b92d-c11a84ef276f-dshm" pod "08baad08-cd72-4305-b92d-c11a84ef276f" (UID: "08baad08-cd72-4305-b92d-c11a84ef276f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:28:01.430252 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:01.430229 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18b72648-f523-405e-b5cd-1266ff13f8f2-kube-api-access-vbzwm" pod "18b72648-f523-405e-b5cd-1266ff13f8f2" (UID: "18b72648-f523-405e-b5cd-1266ff13f8f2"). InnerVolumeSpecName "kube-api-access-vbzwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:28:01.430673 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:01.430639 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08baad08-cd72-4305-b92d-c11a84ef276f-kube-api-access-vj7hs" pod "08baad08-cd72-4305-b92d-c11a84ef276f" (UID: "08baad08-cd72-4305-b92d-c11a84ef276f"). InnerVolumeSpecName "kube-api-access-vj7hs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:28:01.431198 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:01.431172 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08baad08-cd72-4305-b92d-c11a84ef276f-tls-certs" pod "08baad08-cd72-4305-b92d-c11a84ef276f" (UID: "08baad08-cd72-4305-b92d-c11a84ef276f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:28:01.441039 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:01.441001 4493 scope.go:122] "RemoveContainer" containerID="a0f7895786747598a42513c8e971f1f7741fa9b45b156dbab73bb4052578a447" Oct 30 17:28:01.441503 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:28:01.441472 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0f7895786747598a42513c8e971f1f7741fa9b45b156dbab73bb4052578a447\": container with ID starting with a0f7895786747598a42513c8e971f1f7741fa9b45b156dbab73bb4052578a447 not found: ID does not exist" containerID="a0f7895786747598a42513c8e971f1f7741fa9b45b156dbab73bb4052578a447" Oct 30 17:28:01.441619 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:01.441505 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0f7895786747598a42513c8e971f1f7741fa9b45b156dbab73bb4052578a447"} err="failed to get container status \"a0f7895786747598a42513c8e971f1f7741fa9b45b156dbab73bb4052578a447\": rpc error: code = NotFound desc = could not find container \"a0f7895786747598a42513c8e971f1f7741fa9b45b156dbab73bb4052578a447\": container with ID starting with a0f7895786747598a42513c8e971f1f7741fa9b45b156dbab73bb4052578a447 not found: ID does not exist" Oct 30 17:28:01.441619 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:01.441530 4493 scope.go:122] "RemoveContainer" containerID="e2dbe35a076a50a71f243b174960bfd91b291ca96001f7ddf20c71fd56fa4ccf" Oct 30 17:28:01.465391 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:01.465360 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8"] Oct 30 17:28:01.476618 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:01.476594 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-g7dz8"] Oct 30 17:28:01.516634 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:01.516603 4493 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18b72648-f523-405e-b5cd-1266ff13f8f2-tmp-dir\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:28:01.516634 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:01.516632 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vj7hs\" (UniqueName: \"kubernetes.io/projected/08baad08-cd72-4305-b92d-c11a84ef276f-kube-api-access-vj7hs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:28:01.516851 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:01.516641 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbzwm\" (UniqueName: \"kubernetes.io/projected/18b72648-f523-405e-b5cd-1266ff13f8f2-kube-api-access-vbzwm\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:28:01.516851 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:01.516651 4493 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08baad08-cd72-4305-b92d-c11a84ef276f-tmp-dir\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:28:01.516851 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:01.516660 4493 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18b72648-f523-405e-b5cd-1266ff13f8f2-home\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:28:01.516851 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:01.516667 4493 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18b72648-f523-405e-b5cd-1266ff13f8f2-tls-certs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:28:01.516851 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:01.516676 4493 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08baad08-cd72-4305-b92d-c11a84ef276f-home\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:28:01.516851 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:01.516683 4493 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08baad08-cd72-4305-b92d-c11a84ef276f-dshm\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:28:01.516851 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:01.516690 4493 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08baad08-cd72-4305-b92d-c11a84ef276f-model-cache\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:28:01.516851 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:01.516698 4493 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18b72648-f523-405e-b5cd-1266ff13f8f2-model-cache\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:28:01.516851 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:01.516705 4493 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08baad08-cd72-4305-b92d-c11a84ef276f-tls-certs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:28:01.516851 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:01.516714 4493 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18b72648-f523-405e-b5cd-1266ff13f8f2-dshm\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:28:01.749000 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:01.748968 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642"] Oct 30 17:28:01.752852 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:01.752815 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-f3503b/premium-w0-f3503b-kserve-7c85fbb6bc-rv642"] Oct 30 17:28:01.785323 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:01.785295 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq"] Oct 30 17:28:01.789326 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:01.789293 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq"] Oct 30 17:28:02.275591 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:02.275541 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/simulator-w0-f3503b-kserve-76869b4554-tj8hq" podUID="08baad08-cd72-4305-b92d-c11a84ef276f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:28:02.606857 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:02.606720 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 17:28:02.933667 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:02.933633 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08baad08-cd72-4305-b92d-c11a84ef276f" path="/var/lib/kubelet/pods/08baad08-cd72-4305-b92d-c11a84ef276f/volumes" Oct 30 17:28:02.934186 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:02.934169 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18b72648-f523-405e-b5cd-1266ff13f8f2" path="/var/lib/kubelet/pods/18b72648-f523-405e-b5cd-1266ff13f8f2/volumes" Oct 30 17:28:02.934621 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:02.934606 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2c02109-c263-42aa-bdad-a995f3577d99" path="/var/lib/kubelet/pods/e2c02109-c263-42aa-bdad-a995f3577d99/volumes" Oct 30 17:28:07.499988 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:07.499905 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-13c495-s6zsj"] Oct 30 17:28:07.500464 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:07.500449 4493 cpu_manager.go:414] "containerMap: removing container" podUID="08baad08-cd72-4305-b92d-c11a84ef276f" containerName="main" Oct 30 17:28:07.500524 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:07.500466 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08baad08-cd72-4305-b92d-c11a84ef276f" containerName="main" Oct 30 17:28:07.500524 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:07.500478 4493 cpu_manager.go:414] "containerMap: removing container" podUID="18b72648-f523-405e-b5cd-1266ff13f8f2" containerName="main" Oct 30 17:28:07.500524 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:07.500484 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18b72648-f523-405e-b5cd-1266ff13f8f2" containerName="main" Oct 30 17:28:07.500524 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:07.500494 4493 cpu_manager.go:414] "containerMap: removing container" podUID="9cc934bb-7ead-421d-88c1-13e1bc5e93d5" containerName="revoke-keys" Oct 30 17:28:07.500524 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:07.500499 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cc934bb-7ead-421d-88c1-13e1bc5e93d5" containerName="revoke-keys" Oct 30 17:28:07.500524 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:07.500509 4493 cpu_manager.go:414] "containerMap: removing container" podUID="e2c02109-c263-42aa-bdad-a995f3577d99" containerName="istio-proxy" Oct 30 17:28:07.500524 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:07.500514 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2c02109-c263-42aa-bdad-a995f3577d99" containerName="istio-proxy" Oct 30 17:28:07.500766 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:07.500540 4493 cpu_manager.go:414] "containerMap: removing container" podUID="58c27031-67a6-4647-9be5-6f8a0e2adb33" containerName="revoke-keys" Oct 30 17:28:07.500766 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:07.500545 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58c27031-67a6-4647-9be5-6f8a0e2adb33" containerName="revoke-keys" Oct 30 17:28:07.500766 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:07.500602 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="58c27031-67a6-4647-9be5-6f8a0e2adb33" containerName="revoke-keys" Oct 30 17:28:07.500766 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:07.500612 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="08baad08-cd72-4305-b92d-c11a84ef276f" containerName="main" Oct 30 17:28:07.500766 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:07.500620 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="18b72648-f523-405e-b5cd-1266ff13f8f2" containerName="main" Oct 30 17:28:07.500766 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:07.500627 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cc934bb-7ead-421d-88c1-13e1bc5e93d5" containerName="revoke-keys" Oct 30 17:28:07.500766 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:07.500634 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2c02109-c263-42aa-bdad-a995f3577d99" containerName="istio-proxy" Oct 30 17:28:07.505282 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:07.505262 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-13c495-s6zsj" Oct 30 17:28:07.521635 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:07.521609 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 17:28:07.534646 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:07.533469 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:28:07.534646 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:07.533469 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:28:07.537404 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:07.537377 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-13c495-s6zsj"] Oct 30 17:28:07.592376 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:07.592339 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69797206-e912-47bb-b38d-6351cba6851a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-13c495-s6zsj\" (UID: \"69797206-e912-47bb-b38d-6351cba6851a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-13c495-s6zsj" Oct 30 17:28:07.592574 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:07.592381 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69797206-e912-47bb-b38d-6351cba6851a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-13c495-s6zsj\" (UID: \"69797206-e912-47bb-b38d-6351cba6851a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-13c495-s6zsj" Oct 30 17:28:07.592574 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:07.592402 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgpqb\" (UniqueName: \"kubernetes.io/projected/69797206-e912-47bb-b38d-6351cba6851a-kube-api-access-hgpqb\") pod \"maas-api-revoke-keys-e2e-extmod-b-13c495-s6zsj\" (UID: \"69797206-e912-47bb-b38d-6351cba6851a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-13c495-s6zsj" Oct 30 17:28:07.613278 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:07.613244 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:28:07.693564 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:07.693523 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69797206-e912-47bb-b38d-6351cba6851a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-13c495-s6zsj\" (UID: \"69797206-e912-47bb-b38d-6351cba6851a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-13c495-s6zsj" Oct 30 17:28:07.693740 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:07.693589 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69797206-e912-47bb-b38d-6351cba6851a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-13c495-s6zsj\" (UID: \"69797206-e912-47bb-b38d-6351cba6851a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-13c495-s6zsj" Oct 30 17:28:07.693740 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:07.693616 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgpqb\" (UniqueName: \"kubernetes.io/projected/69797206-e912-47bb-b38d-6351cba6851a-kube-api-access-hgpqb\") pod \"maas-api-revoke-keys-e2e-extmod-b-13c495-s6zsj\" (UID: \"69797206-e912-47bb-b38d-6351cba6851a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-13c495-s6zsj" Oct 30 17:28:07.694386 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:07.694359 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69797206-e912-47bb-b38d-6351cba6851a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-13c495-s6zsj\" (UID: \"69797206-e912-47bb-b38d-6351cba6851a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-13c495-s6zsj" Oct 30 17:28:07.696309 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:07.696280 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69797206-e912-47bb-b38d-6351cba6851a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-13c495-s6zsj\" (UID: \"69797206-e912-47bb-b38d-6351cba6851a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-13c495-s6zsj" Oct 30 17:28:07.717270 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:07.717246 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgpqb\" (UniqueName: \"kubernetes.io/projected/69797206-e912-47bb-b38d-6351cba6851a-kube-api-access-hgpqb\") pod \"maas-api-revoke-keys-e2e-extmod-b-13c495-s6zsj\" (UID: \"69797206-e912-47bb-b38d-6351cba6851a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-13c495-s6zsj" Oct 30 17:28:07.819552 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:07.819455 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-13c495-s6zsj" Oct 30 17:28:08.034668 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:08.034630 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f775cacd-j5hhq"] Oct 30 17:28:08.037454 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:08.037434 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f775cacd-j5hhq"] Oct 30 17:28:08.124246 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:08.124222 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-13c495-s6zsj"] Oct 30 17:28:08.125978 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:28:08.125952 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69797206_e912_47bb_b38d_6351cba6851a.slice/crio-1501266127e307a64247bafcbd18118875f93fb26f77f76e67ad2ecac19b573e WatchSource:0}: Error finding container 1501266127e307a64247bafcbd18118875f93fb26f77f76e67ad2ecac19b573e: Status 404 returned error can't find the container with id 1501266127e307a64247bafcbd18118875f93fb26f77f76e67ad2ecac19b573e Oct 30 17:28:08.470227 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:08.470194 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-13c495-s6zsj" event={"ID":"69797206-e912-47bb-b38d-6351cba6851a","Type":"ContainerStarted","Data":"27653feda30ae1ae9c3b8af65d0c5ca4b1ff1ca72ecdcfb5a520c6d7ed1b9240"} Oct 30 17:28:08.470394 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:08.470251 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-13c495-s6zsj" event={"ID":"69797206-e912-47bb-b38d-6351cba6851a","Type":"ContainerStarted","Data":"1501266127e307a64247bafcbd18118875f93fb26f77f76e67ad2ecac19b573e"} Oct 30 17:28:08.509877 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:08.509818 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-13c495-s6zsj" podStartSLOduration=1.509775081 podStartE2EDuration="1.509775081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:28:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:28:08.509483337 +0000 UTC m=+3915.736658438" watchObservedRunningTime="2026-10-30 17:28:08.509775081 +0000 UTC m=+3915.736950180" Oct 30 17:28:08.933597 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:08.933564 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca45995c-3de8-46d3-9035-19ee8c3408a0" path="/var/lib/kubelet/pods/ca45995c-3de8-46d3-9035-19ee8c3408a0/volumes" Oct 30 17:28:09.031138 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:09.031108 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10ec30ee-5s8tt"] Oct 30 17:28:09.034041 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:09.034015 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10ec30ee-5s8tt"] Oct 30 17:28:09.480964 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:09.480921 4493 generic.go:356] "Generic (PLEG): container finished" podID="69797206-e912-47bb-b38d-6351cba6851a" containerID="27653feda30ae1ae9c3b8af65d0c5ca4b1ff1ca72ecdcfb5a520c6d7ed1b9240" exitCode=0 Oct 30 17:28:09.481149 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:09.481015 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-13c495-s6zsj" event={"ID":"69797206-e912-47bb-b38d-6351cba6851a","Type":"ContainerDied","Data":"27653feda30ae1ae9c3b8af65d0c5ca4b1ff1ca72ecdcfb5a520c6d7ed1b9240"} Oct 30 17:28:10.643354 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:10.643322 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-13c495-s6zsj" Oct 30 17:28:10.677139 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:10.677108 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f3503b-mqcjn"] Oct 30 17:28:10.680589 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:10.680568 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f3503b-mqcjn"] Oct 30 17:28:10.730752 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:10.730719 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69797206-e912-47bb-b38d-6351cba6851a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69797206-e912-47bb-b38d-6351cba6851a-maas-api-xks-ca\") pod \"69797206-e912-47bb-b38d-6351cba6851a\" (UID: \"69797206-e912-47bb-b38d-6351cba6851a\") " Oct 30 17:28:10.730971 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:10.730907 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69797206-e912-47bb-b38d-6351cba6851a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69797206-e912-47bb-b38d-6351cba6851a-maas-api-openshift-ca\") pod \"69797206-e912-47bb-b38d-6351cba6851a\" (UID: \"69797206-e912-47bb-b38d-6351cba6851a\") " Oct 30 17:28:10.731430 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:10.730970 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69797206-e912-47bb-b38d-6351cba6851a-kube-api-access-hgpqb\" (UniqueName: \"kubernetes.io/projected/69797206-e912-47bb-b38d-6351cba6851a-kube-api-access-hgpqb\") pod \"69797206-e912-47bb-b38d-6351cba6851a\" (UID: \"69797206-e912-47bb-b38d-6351cba6851a\") " Oct 30 17:28:10.731430 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:10.731364 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69797206-e912-47bb-b38d-6351cba6851a-maas-api-openshift-ca" pod "69797206-e912-47bb-b38d-6351cba6851a" (UID: "69797206-e912-47bb-b38d-6351cba6851a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:28:10.731664 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:10.731626 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69797206-e912-47bb-b38d-6351cba6851a-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:28:10.733544 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:10.733511 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69797206-e912-47bb-b38d-6351cba6851a-maas-api-xks-ca" pod "69797206-e912-47bb-b38d-6351cba6851a" (UID: "69797206-e912-47bb-b38d-6351cba6851a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:28:10.733544 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:10.733693 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69797206-e912-47bb-b38d-6351cba6851a-kube-api-access-hgpqb" pod "69797206-e912-47bb-b38d-6351cba6851a" (UID: "69797206-e912-47bb-b38d-6351cba6851a"). InnerVolumeSpecName "kube-api-access-hgpqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:28:10.832244 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:10.832204 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69797206-e912-47bb-b38d-6351cba6851a-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:28:10.832244 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:10.832236 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgpqb\" (UniqueName: \"kubernetes.io/projected/69797206-e912-47bb-b38d-6351cba6851a-kube-api-access-hgpqb\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:28:10.935499 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:10.935464 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a5999d8-6e17-496d-b28c-e3d6ac908b07" path="/var/lib/kubelet/pods/2a5999d8-6e17-496d-b28c-e3d6ac908b07/volumes" Oct 30 17:28:10.935949 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:10.935930 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89047d9c-0062-4d18-a8d1-3e119f1c01f8" path="/var/lib/kubelet/pods/89047d9c-0062-4d18-a8d1-3e119f1c01f8/volumes" Oct 30 17:28:11.498647 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:11.498617 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-13c495-s6zsj" Oct 30 17:28:11.498863 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:11.498657 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-13c495-s6zsj" event={"ID":"69797206-e912-47bb-b38d-6351cba6851a","Type":"ContainerDied","Data":"1501266127e307a64247bafcbd18118875f93fb26f77f76e67ad2ecac19b573e"} Oct 30 17:28:11.498863 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:11.498711 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1501266127e307a64247bafcbd18118875f93fb26f77f76e67ad2ecac19b573e" Oct 30 17:28:12.784835 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:12.784801 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 17:28:20.263875 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:20.263826 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-13c495-s6zsj"] Oct 30 17:28:20.274922 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:20.274896 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-13c495-s6zsj"] Oct 30 17:28:20.932578 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:20.932544 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69797206-e912-47bb-b38d-6351cba6851a" path="/var/lib/kubelet/pods/69797206-e912-47bb-b38d-6351cba6851a/volumes" Oct 30 17:28:23.199183 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:23.199142 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 17:28:27.735087 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:27.735048 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2"] Oct 30 17:28:27.736103 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:27.736064 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2" podUID="98af0974-af46-432d-a622-4279717ff916" containerName="istio-proxy" containerID="cri-o://09d5075eaf353d561bd7b8b3a7a0a550b6027cd31ea24c0d73cf4ca45c7d5e03" gracePeriod=30 Oct 30 17:28:28.031608 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:28.031477 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6df86982-g995k"] Oct 30 17:28:28.034489 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:28.034459 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6df86982-g995k"] Oct 30 17:28:28.375478 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:28.375404 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cf4621-65d9h"] Oct 30 17:28:28.375890 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:28.375865 4493 cpu_manager.go:414] "containerMap: removing container" podUID="69797206-e912-47bb-b38d-6351cba6851a" containerName="revoke-keys" Oct 30 17:28:28.375890 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:28.375891 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69797206-e912-47bb-b38d-6351cba6851a" containerName="revoke-keys" Oct 30 17:28:28.375993 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:28.375971 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="69797206-e912-47bb-b38d-6351cba6851a" containerName="revoke-keys" Oct 30 17:28:28.380622 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:28.380604 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cf4621-65d9h" Oct 30 17:28:28.407004 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:28.406958 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:28:28.407004 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:28.406981 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:28:28.409453 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:28.409427 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cf4621-65d9h"] Oct 30 17:28:28.422051 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:28.422021 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddd5cbdc-cefc-4ce8-8ab5-36ce7c3ca460-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-cf4621-65d9h\" (UID: \"ddd5cbdc-cefc-4ce8-8ab5-36ce7c3ca460\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cf4621-65d9h" Oct 30 17:28:28.422198 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:28.422093 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddd5cbdc-cefc-4ce8-8ab5-36ce7c3ca460-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-cf4621-65d9h\" (UID: \"ddd5cbdc-cefc-4ce8-8ab5-36ce7c3ca460\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cf4621-65d9h" Oct 30 17:28:28.422198 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:28.422176 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl4m5\" (UniqueName: \"kubernetes.io/projected/ddd5cbdc-cefc-4ce8-8ab5-36ce7c3ca460-kube-api-access-wl4m5\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-cf4621-65d9h\" (UID: \"ddd5cbdc-cefc-4ce8-8ab5-36ce7c3ca460\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cf4621-65d9h" Oct 30 17:28:28.441126 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:28.441077 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:28:28.441328 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:28.441236 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 17:28:28.523399 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:28.523354 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddd5cbdc-cefc-4ce8-8ab5-36ce7c3ca460-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-cf4621-65d9h\" (UID: \"ddd5cbdc-cefc-4ce8-8ab5-36ce7c3ca460\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cf4621-65d9h" Oct 30 17:28:28.523615 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:28.523432 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddd5cbdc-cefc-4ce8-8ab5-36ce7c3ca460-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-cf4621-65d9h\" (UID: \"ddd5cbdc-cefc-4ce8-8ab5-36ce7c3ca460\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cf4621-65d9h" Oct 30 17:28:28.523615 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:28.523459 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl4m5\" (UniqueName: \"kubernetes.io/projected/ddd5cbdc-cefc-4ce8-8ab5-36ce7c3ca460-kube-api-access-wl4m5\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-cf4621-65d9h\" (UID: \"ddd5cbdc-cefc-4ce8-8ab5-36ce7c3ca460\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cf4621-65d9h" Oct 30 17:28:28.524312 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:28.524281 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddd5cbdc-cefc-4ce8-8ab5-36ce7c3ca460-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-cf4621-65d9h\" (UID: \"ddd5cbdc-cefc-4ce8-8ab5-36ce7c3ca460\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cf4621-65d9h" Oct 30 17:28:28.526025 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:28.526002 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddd5cbdc-cefc-4ce8-8ab5-36ce7c3ca460-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-cf4621-65d9h\" (UID: \"ddd5cbdc-cefc-4ce8-8ab5-36ce7c3ca460\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cf4621-65d9h" Oct 30 17:28:28.546430 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:28.546399 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl4m5\" (UniqueName: \"kubernetes.io/projected/ddd5cbdc-cefc-4ce8-8ab5-36ce7c3ca460-kube-api-access-wl4m5\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-cf4621-65d9h\" (UID: \"ddd5cbdc-cefc-4ce8-8ab5-36ce7c3ca460\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cf4621-65d9h" Oct 30 17:28:28.694179 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:28.694145 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cf4621-65d9h" Oct 30 17:28:28.931719 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:28.931684 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3104ca8b-193a-4371-9dd4-119e7b69887a" path="/var/lib/kubelet/pods/3104ca8b-193a-4371-9dd4-119e7b69887a/volumes" Oct 30 17:28:29.004622 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:29.004592 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cf4621-65d9h"] Oct 30 17:28:29.006490 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:28:29.006457 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddd5cbdc_cefc_4ce8_8ab5_36ce7c3ca460.slice/crio-c1205c69627fd31f4d351eee3197f647f2cf3a9c4640885a837b0c5051b91a70 WatchSource:0}: Error finding container c1205c69627fd31f4d351eee3197f647f2cf3a9c4640885a837b0c5051b91a70: Status 404 returned error can't find the container with id c1205c69627fd31f4d351eee3197f647f2cf3a9c4640885a837b0c5051b91a70 Oct 30 17:28:29.662673 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:29.662634 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cf4621-65d9h" event={"ID":"ddd5cbdc-cefc-4ce8-8ab5-36ce7c3ca460","Type":"ContainerStarted","Data":"2e69b2c8a3d144049e75eccf1ba98aa60d2af006aa44dc0c347894341b812bac"} Oct 30 17:28:29.662887 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:29.662696 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cf4621-65d9h" event={"ID":"ddd5cbdc-cefc-4ce8-8ab5-36ce7c3ca460","Type":"ContainerStarted","Data":"c1205c69627fd31f4d351eee3197f647f2cf3a9c4640885a837b0c5051b91a70"} Oct 30 17:28:29.704600 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:29.704547 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cf4621-65d9h" podStartSLOduration=1.704532602 podStartE2EDuration="1.704532602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:28:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:28:29.702118264 +0000 UTC m=+3936.929293364" watchObservedRunningTime="2026-10-30 17:28:29.704532602 +0000 UTC m=+3936.931707702" Oct 30 17:28:30.342368 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:28:30.342325 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 17:28:30.677213 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:30.677176 4493 generic.go:356] "Generic (PLEG): container finished" podID="ddd5cbdc-cefc-4ce8-8ab5-36ce7c3ca460" containerID="2e69b2c8a3d144049e75eccf1ba98aa60d2af006aa44dc0c347894341b812bac" exitCode=0 Oct 30 17:28:30.677437 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:30.677259 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cf4621-65d9h" event={"ID":"ddd5cbdc-cefc-4ce8-8ab5-36ce7c3ca460","Type":"ContainerDied","Data":"2e69b2c8a3d144049e75eccf1ba98aa60d2af006aa44dc0c347894341b812bac"} Oct 30 17:28:31.828025 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:31.827997 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cf4621-65d9h" Oct 30 17:28:31.860665 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:31.860639 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddd5cbdc-cefc-4ce8-8ab5-36ce7c3ca460-kube-api-access-wl4m5\" (UniqueName: \"kubernetes.io/projected/ddd5cbdc-cefc-4ce8-8ab5-36ce7c3ca460-kube-api-access-wl4m5\") pod \"ddd5cbdc-cefc-4ce8-8ab5-36ce7c3ca460\" (UID: \"ddd5cbdc-cefc-4ce8-8ab5-36ce7c3ca460\") " Oct 30 17:28:31.860833 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:31.860718 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddd5cbdc-cefc-4ce8-8ab5-36ce7c3ca460-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddd5cbdc-cefc-4ce8-8ab5-36ce7c3ca460-maas-api-openshift-ca\") pod \"ddd5cbdc-cefc-4ce8-8ab5-36ce7c3ca460\" (UID: \"ddd5cbdc-cefc-4ce8-8ab5-36ce7c3ca460\") " Oct 30 17:28:31.860833 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:31.860755 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ddd5cbdc-cefc-4ce8-8ab5-36ce7c3ca460-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddd5cbdc-cefc-4ce8-8ab5-36ce7c3ca460-maas-api-xks-ca\") pod \"ddd5cbdc-cefc-4ce8-8ab5-36ce7c3ca460\" (UID: \"ddd5cbdc-cefc-4ce8-8ab5-36ce7c3ca460\") " Oct 30 17:28:31.861188 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:31.861161 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddd5cbdc-cefc-4ce8-8ab5-36ce7c3ca460-maas-api-openshift-ca" pod "ddd5cbdc-cefc-4ce8-8ab5-36ce7c3ca460" (UID: "ddd5cbdc-cefc-4ce8-8ab5-36ce7c3ca460"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:28:31.863034 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:31.863012 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddd5cbdc-cefc-4ce8-8ab5-36ce7c3ca460-maas-api-xks-ca" pod "ddd5cbdc-cefc-4ce8-8ab5-36ce7c3ca460" (UID: "ddd5cbdc-cefc-4ce8-8ab5-36ce7c3ca460"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:28:31.863126 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:31.863088 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddd5cbdc-cefc-4ce8-8ab5-36ce7c3ca460-kube-api-access-wl4m5" pod "ddd5cbdc-cefc-4ce8-8ab5-36ce7c3ca460" (UID: "ddd5cbdc-cefc-4ce8-8ab5-36ce7c3ca460"). InnerVolumeSpecName "kube-api-access-wl4m5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:28:31.961962 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:31.961869 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wl4m5\" (UniqueName: \"kubernetes.io/projected/ddd5cbdc-cefc-4ce8-8ab5-36ce7c3ca460-kube-api-access-wl4m5\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:28:31.961962 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:31.961908 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddd5cbdc-cefc-4ce8-8ab5-36ce7c3ca460-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:28:31.961962 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:31.961922 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddd5cbdc-cefc-4ce8-8ab5-36ce7c3ca460-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:28:32.694959 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:32.694928 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cf4621-65d9h" Oct 30 17:28:32.695150 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:32.694971 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cf4621-65d9h" event={"ID":"ddd5cbdc-cefc-4ce8-8ab5-36ce7c3ca460","Type":"ContainerDied","Data":"c1205c69627fd31f4d351eee3197f647f2cf3a9c4640885a837b0c5051b91a70"} Oct 30 17:28:32.695150 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:32.695029 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1205c69627fd31f4d351eee3197f647f2cf3a9c4640885a837b0c5051b91a70" Oct 30 17:28:32.898372 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:32.898340 4493 patch_prober.go:29] interesting pod/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Oct 30 17:28:32.898758 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:32.898395 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2" podUID="98af0974-af46-432d-a622-4279717ff916" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Oct 30 17:28:33.001982 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:33.001952 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2" Oct 30 17:28:33.072514 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:33.072483 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98af0974-af46-432d-a622-4279717ff916-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98af0974-af46-432d-a622-4279717ff916-istiod-ca-cert\") pod \"98af0974-af46-432d-a622-4279717ff916\" (UID: \"98af0974-af46-432d-a622-4279717ff916\") " Oct 30 17:28:33.072514 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:33.072516 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98af0974-af46-432d-a622-4279717ff916-kube-api-access-w4ds7\" (UniqueName: \"kubernetes.io/projected/98af0974-af46-432d-a622-4279717ff916-kube-api-access-w4ds7\") pod \"98af0974-af46-432d-a622-4279717ff916\" (UID: \"98af0974-af46-432d-a622-4279717ff916\") " Oct 30 17:28:33.072752 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:33.072580 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98af0974-af46-432d-a622-4279717ff916-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98af0974-af46-432d-a622-4279717ff916-istio-envoy\") pod \"98af0974-af46-432d-a622-4279717ff916\" (UID: \"98af0974-af46-432d-a622-4279717ff916\") " Oct 30 17:28:33.072752 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:33.072607 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98af0974-af46-432d-a622-4279717ff916-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98af0974-af46-432d-a622-4279717ff916-istio-data\") pod \"98af0974-af46-432d-a622-4279717ff916\" (UID: \"98af0974-af46-432d-a622-4279717ff916\") " Oct 30 17:28:33.072752 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:33.072649 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98af0974-af46-432d-a622-4279717ff916-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98af0974-af46-432d-a622-4279717ff916-workload-socket\") pod \"98af0974-af46-432d-a622-4279717ff916\" (UID: \"98af0974-af46-432d-a622-4279717ff916\") " Oct 30 17:28:33.072752 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:33.072693 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98af0974-af46-432d-a622-4279717ff916-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98af0974-af46-432d-a622-4279717ff916-credential-socket\") pod \"98af0974-af46-432d-a622-4279717ff916\" (UID: \"98af0974-af46-432d-a622-4279717ff916\") " Oct 30 17:28:33.072752 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:33.072718 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98af0974-af46-432d-a622-4279717ff916-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98af0974-af46-432d-a622-4279717ff916-workload-certs\") pod \"98af0974-af46-432d-a622-4279717ff916\" (UID: \"98af0974-af46-432d-a622-4279717ff916\") " Oct 30 17:28:33.072752 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:33.072750 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98af0974-af46-432d-a622-4279717ff916-istio-token\" (UniqueName: \"kubernetes.io/projected/98af0974-af46-432d-a622-4279717ff916-istio-token\") pod \"98af0974-af46-432d-a622-4279717ff916\" (UID: \"98af0974-af46-432d-a622-4279717ff916\") " Oct 30 17:28:33.073119 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:33.072796 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/98af0974-af46-432d-a622-4279717ff916-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98af0974-af46-432d-a622-4279717ff916-istio-podinfo\") pod \"98af0974-af46-432d-a622-4279717ff916\" (UID: \"98af0974-af46-432d-a622-4279717ff916\") " Oct 30 17:28:33.073119 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:33.072963 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98af0974-af46-432d-a622-4279717ff916-istio-data" pod "98af0974-af46-432d-a622-4279717ff916" (UID: "98af0974-af46-432d-a622-4279717ff916"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:28:33.073119 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:33.072991 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98af0974-af46-432d-a622-4279717ff916-istiod-ca-cert" pod "98af0974-af46-432d-a622-4279717ff916" (UID: "98af0974-af46-432d-a622-4279717ff916"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:28:33.073286 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:33.073162 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98af0974-af46-432d-a622-4279717ff916-workload-certs" pod "98af0974-af46-432d-a622-4279717ff916" (UID: "98af0974-af46-432d-a622-4279717ff916"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:28:33.073286 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:33.073217 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98af0974-af46-432d-a622-4279717ff916-istio-data\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:28:33.073286 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:33.073239 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98af0974-af46-432d-a622-4279717ff916-istiod-ca-cert\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:28:33.073619 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:33.073321 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98af0974-af46-432d-a622-4279717ff916-workload-socket" pod "98af0974-af46-432d-a622-4279717ff916" (UID: "98af0974-af46-432d-a622-4279717ff916"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:28:33.073619 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:33.073387 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98af0974-af46-432d-a622-4279717ff916-credential-socket" pod "98af0974-af46-432d-a622-4279717ff916" (UID: "98af0974-af46-432d-a622-4279717ff916"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:28:33.075109 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:33.075077 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98af0974-af46-432d-a622-4279717ff916-istio-envoy" pod "98af0974-af46-432d-a622-4279717ff916" (UID: "98af0974-af46-432d-a622-4279717ff916"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:28:33.075245 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:33.075136 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/98af0974-af46-432d-a622-4279717ff916-istio-podinfo" pod "98af0974-af46-432d-a622-4279717ff916" (UID: "98af0974-af46-432d-a622-4279717ff916"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:28:33.075306 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:33.075268 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98af0974-af46-432d-a622-4279717ff916-istio-token" pod "98af0974-af46-432d-a622-4279717ff916" (UID: "98af0974-af46-432d-a622-4279717ff916"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:28:33.075485 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:33.075468 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98af0974-af46-432d-a622-4279717ff916-kube-api-access-w4ds7" pod "98af0974-af46-432d-a622-4279717ff916" (UID: "98af0974-af46-432d-a622-4279717ff916"). InnerVolumeSpecName "kube-api-access-w4ds7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:28:33.174513 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:33.174486 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4ds7\" (UniqueName: \"kubernetes.io/projected/98af0974-af46-432d-a622-4279717ff916-kube-api-access-w4ds7\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:28:33.174513 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:33.174509 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98af0974-af46-432d-a622-4279717ff916-istio-envoy\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:28:33.174674 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:33.174520 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98af0974-af46-432d-a622-4279717ff916-workload-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:28:33.174674 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:33.174528 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98af0974-af46-432d-a622-4279717ff916-credential-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:28:33.174674 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:33.174536 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98af0974-af46-432d-a622-4279717ff916-workload-certs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:28:33.174674 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:33.174544 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98af0974-af46-432d-a622-4279717ff916-istio-token\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:28:33.174674 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:33.174551 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98af0974-af46-432d-a622-4279717ff916-istio-podinfo\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:28:33.704530 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:33.704495 4493 generic.go:356] "Generic (PLEG): container finished" podID="98af0974-af46-432d-a622-4279717ff916" containerID="09d5075eaf353d561bd7b8b3a7a0a550b6027cd31ea24c0d73cf4ca45c7d5e03" exitCode=0 Oct 30 17:28:33.704743 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:33.704542 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2" Oct 30 17:28:33.704743 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:33.704582 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2" event={"ID":"98af0974-af46-432d-a622-4279717ff916","Type":"ContainerDied","Data":"09d5075eaf353d561bd7b8b3a7a0a550b6027cd31ea24c0d73cf4ca45c7d5e03"} Oct 30 17:28:33.704743 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:33.704634 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2" event={"ID":"98af0974-af46-432d-a622-4279717ff916","Type":"ContainerDied","Data":"e4109deafca5d40be6679454f707462d4b290aae0d7604a89315b43b6a8a0c4c"} Oct 30 17:28:33.704743 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:33.704656 4493 scope.go:122] "RemoveContainer" containerID="09d5075eaf353d561bd7b8b3a7a0a550b6027cd31ea24c0d73cf4ca45c7d5e03" Oct 30 17:28:33.716753 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:33.716721 4493 scope.go:122] "RemoveContainer" containerID="8ef450d4c2c76eaad33b7c06f4f13933980135fe562eeff48190e8642a734f56" Oct 30 17:28:33.726892 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:33.726872 4493 scope.go:122] "RemoveContainer" containerID="09d5075eaf353d561bd7b8b3a7a0a550b6027cd31ea24c0d73cf4ca45c7d5e03" Oct 30 17:28:33.727153 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:28:33.727133 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09d5075eaf353d561bd7b8b3a7a0a550b6027cd31ea24c0d73cf4ca45c7d5e03\": container with ID starting with 09d5075eaf353d561bd7b8b3a7a0a550b6027cd31ea24c0d73cf4ca45c7d5e03 not found: ID does not exist" containerID="09d5075eaf353d561bd7b8b3a7a0a550b6027cd31ea24c0d73cf4ca45c7d5e03" Oct 30 17:28:33.727208 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:33.727163 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09d5075eaf353d561bd7b8b3a7a0a550b6027cd31ea24c0d73cf4ca45c7d5e03"} err="failed to get container status \"09d5075eaf353d561bd7b8b3a7a0a550b6027cd31ea24c0d73cf4ca45c7d5e03\": rpc error: code = NotFound desc = could not find container \"09d5075eaf353d561bd7b8b3a7a0a550b6027cd31ea24c0d73cf4ca45c7d5e03\": container with ID starting with 09d5075eaf353d561bd7b8b3a7a0a550b6027cd31ea24c0d73cf4ca45c7d5e03 not found: ID does not exist" Oct 30 17:28:33.727208 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:33.727179 4493 scope.go:122] "RemoveContainer" containerID="8ef450d4c2c76eaad33b7c06f4f13933980135fe562eeff48190e8642a734f56" Oct 30 17:28:33.727411 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:28:33.727394 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ef450d4c2c76eaad33b7c06f4f13933980135fe562eeff48190e8642a734f56\": container with ID starting with 8ef450d4c2c76eaad33b7c06f4f13933980135fe562eeff48190e8642a734f56 not found: ID does not exist" containerID="8ef450d4c2c76eaad33b7c06f4f13933980135fe562eeff48190e8642a734f56" Oct 30 17:28:33.727459 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:33.727417 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ef450d4c2c76eaad33b7c06f4f13933980135fe562eeff48190e8642a734f56"} err="failed to get container status \"8ef450d4c2c76eaad33b7c06f4f13933980135fe562eeff48190e8642a734f56\": rpc error: code = NotFound desc = could not find container \"8ef450d4c2c76eaad33b7c06f4f13933980135fe562eeff48190e8642a734f56\": container with ID starting with 8ef450d4c2c76eaad33b7c06f4f13933980135fe562eeff48190e8642a734f56 not found: ID does not exist" Oct 30 17:28:33.756133 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:33.756102 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2"] Oct 30 17:28:33.767549 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:33.767514 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-9mpt2"] Oct 30 17:28:34.932430 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:34.932397 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98af0974-af46-432d-a622-4279717ff916" path="/var/lib/kubelet/pods/98af0974-af46-432d-a622-4279717ff916/volumes" Oct 30 17:28:40.293635 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:40.293599 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cf4621-65d9h"] Oct 30 17:28:40.304872 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:40.304813 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cf4621-65d9h"] Oct 30 17:28:40.932320 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:40.932282 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddd5cbdc-cefc-4ce8-8ab5-36ce7c3ca460" path="/var/lib/kubelet/pods/ddd5cbdc-cefc-4ce8-8ab5-36ce7c3ca460/volumes" Oct 30 17:28:42.692476 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:42.690877 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 17:28:48.601447 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:48.601413 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3ca339-7mrnd"] Oct 30 17:28:48.601891 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:48.601869 4493 cpu_manager.go:414] "containerMap: removing container" podUID="ddd5cbdc-cefc-4ce8-8ab5-36ce7c3ca460" containerName="revoke-keys" Oct 30 17:28:48.601938 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:48.601891 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddd5cbdc-cefc-4ce8-8ab5-36ce7c3ca460" containerName="revoke-keys" Oct 30 17:28:48.601938 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:48.601902 4493 cpu_manager.go:414] "containerMap: removing container" podUID="98af0974-af46-432d-a622-4279717ff916" containerName="istio-proxy" Oct 30 17:28:48.601938 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:48.601907 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98af0974-af46-432d-a622-4279717ff916" containerName="istio-proxy" Oct 30 17:28:48.601938 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:48.601923 4493 cpu_manager.go:414] "containerMap: removing container" podUID="98af0974-af46-432d-a622-4279717ff916" containerName="istio-proxy" Oct 30 17:28:48.601938 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:48.601928 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98af0974-af46-432d-a622-4279717ff916" containerName="istio-proxy" Oct 30 17:28:48.602212 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:48.602018 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="98af0974-af46-432d-a622-4279717ff916" containerName="istio-proxy" Oct 30 17:28:48.602212 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:48.602035 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddd5cbdc-cefc-4ce8-8ab5-36ce7c3ca460" containerName="revoke-keys" Oct 30 17:28:48.607023 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:48.607004 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3ca339-7mrnd" Oct 30 17:28:48.623949 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:48.623917 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:28:48.635508 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:48.635489 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:28:48.639994 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:48.639970 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3ca339-7mrnd"] Oct 30 17:28:48.672556 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:48.672525 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:28:48.672556 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:48.672544 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 17:28:48.729224 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:48.729189 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wfkz\" (UniqueName: \"kubernetes.io/projected/40c28cc9-6a1b-486d-b95d-4501ad5f9383-kube-api-access-2wfkz\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-3ca339-7mrnd\" (UID: \"40c28cc9-6a1b-486d-b95d-4501ad5f9383\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3ca339-7mrnd" Oct 30 17:28:48.729371 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:48.729275 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40c28cc9-6a1b-486d-b95d-4501ad5f9383-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-3ca339-7mrnd\" (UID: \"40c28cc9-6a1b-486d-b95d-4501ad5f9383\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3ca339-7mrnd" Oct 30 17:28:48.729444 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:48.729421 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40c28cc9-6a1b-486d-b95d-4501ad5f9383-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-3ca339-7mrnd\" (UID: \"40c28cc9-6a1b-486d-b95d-4501ad5f9383\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3ca339-7mrnd" Oct 30 17:28:48.830481 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:48.830449 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40c28cc9-6a1b-486d-b95d-4501ad5f9383-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-3ca339-7mrnd\" (UID: \"40c28cc9-6a1b-486d-b95d-4501ad5f9383\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3ca339-7mrnd" Oct 30 17:28:48.830679 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:48.830552 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40c28cc9-6a1b-486d-b95d-4501ad5f9383-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-3ca339-7mrnd\" (UID: \"40c28cc9-6a1b-486d-b95d-4501ad5f9383\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3ca339-7mrnd" Oct 30 17:28:48.830679 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:48.830614 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wfkz\" (UniqueName: \"kubernetes.io/projected/40c28cc9-6a1b-486d-b95d-4501ad5f9383-kube-api-access-2wfkz\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-3ca339-7mrnd\" (UID: \"40c28cc9-6a1b-486d-b95d-4501ad5f9383\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3ca339-7mrnd" Oct 30 17:28:48.831303 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:48.831237 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40c28cc9-6a1b-486d-b95d-4501ad5f9383-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-3ca339-7mrnd\" (UID: \"40c28cc9-6a1b-486d-b95d-4501ad5f9383\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3ca339-7mrnd" Oct 30 17:28:48.833042 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:48.833019 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40c28cc9-6a1b-486d-b95d-4501ad5f9383-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-3ca339-7mrnd\" (UID: \"40c28cc9-6a1b-486d-b95d-4501ad5f9383\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3ca339-7mrnd" Oct 30 17:28:48.857185 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:48.857132 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wfkz\" (UniqueName: \"kubernetes.io/projected/40c28cc9-6a1b-486d-b95d-4501ad5f9383-kube-api-access-2wfkz\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-3ca339-7mrnd\" (UID: \"40c28cc9-6a1b-486d-b95d-4501ad5f9383\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3ca339-7mrnd" Oct 30 17:28:48.922387 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:48.922356 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3ca339-7mrnd" Oct 30 17:28:49.240882 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:49.236233 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3ca339-7mrnd"] Oct 30 17:28:49.829123 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:49.829030 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3ca339-7mrnd" event={"ID":"40c28cc9-6a1b-486d-b95d-4501ad5f9383","Type":"ContainerStarted","Data":"49a3c9556e053f9ae85dd68cd1346cf5d2a8ac4b4cb0e0720248154a26b087e6"} Oct 30 17:28:49.829123 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:49.829093 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3ca339-7mrnd" event={"ID":"40c28cc9-6a1b-486d-b95d-4501ad5f9383","Type":"ContainerStarted","Data":"06186feb27a46edf0463f3d2f50a5d10878eb60b082a5964aed8134cbdc2d37b"} Oct 30 17:28:49.869596 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:49.869548 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3ca339-7mrnd" podStartSLOduration=1.869532634 podStartE2EDuration="1.869532634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:28:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:28:49.868051776 +0000 UTC m=+3957.095226876" watchObservedRunningTime="2026-10-30 17:28:49.869532634 +0000 UTC m=+3957.096707733" Oct 30 17:28:50.837797 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:50.837745 4493 generic.go:356] "Generic (PLEG): container finished" podID="40c28cc9-6a1b-486d-b95d-4501ad5f9383" containerID="49a3c9556e053f9ae85dd68cd1346cf5d2a8ac4b4cb0e0720248154a26b087e6" exitCode=0 Oct 30 17:28:50.838210 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:50.837835 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3ca339-7mrnd" event={"ID":"40c28cc9-6a1b-486d-b95d-4501ad5f9383","Type":"ContainerDied","Data":"49a3c9556e053f9ae85dd68cd1346cf5d2a8ac4b4cb0e0720248154a26b087e6"} Oct 30 17:28:51.982691 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:51.982667 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3ca339-7mrnd" Oct 30 17:28:52.062558 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:52.062528 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40c28cc9-6a1b-486d-b95d-4501ad5f9383-kube-api-access-2wfkz\" (UniqueName: \"kubernetes.io/projected/40c28cc9-6a1b-486d-b95d-4501ad5f9383-kube-api-access-2wfkz\") pod \"40c28cc9-6a1b-486d-b95d-4501ad5f9383\" (UID: \"40c28cc9-6a1b-486d-b95d-4501ad5f9383\") " Oct 30 17:28:52.062745 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:52.062641 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40c28cc9-6a1b-486d-b95d-4501ad5f9383-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40c28cc9-6a1b-486d-b95d-4501ad5f9383-maas-api-xks-ca\") pod \"40c28cc9-6a1b-486d-b95d-4501ad5f9383\" (UID: \"40c28cc9-6a1b-486d-b95d-4501ad5f9383\") " Oct 30 17:28:52.062745 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:52.062675 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40c28cc9-6a1b-486d-b95d-4501ad5f9383-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40c28cc9-6a1b-486d-b95d-4501ad5f9383-maas-api-openshift-ca\") pod \"40c28cc9-6a1b-486d-b95d-4501ad5f9383\" (UID: \"40c28cc9-6a1b-486d-b95d-4501ad5f9383\") " Oct 30 17:28:52.063139 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:52.063114 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40c28cc9-6a1b-486d-b95d-4501ad5f9383-maas-api-openshift-ca" pod "40c28cc9-6a1b-486d-b95d-4501ad5f9383" (UID: "40c28cc9-6a1b-486d-b95d-4501ad5f9383"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:28:52.064897 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:52.064875 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40c28cc9-6a1b-486d-b95d-4501ad5f9383-kube-api-access-2wfkz" pod "40c28cc9-6a1b-486d-b95d-4501ad5f9383" (UID: "40c28cc9-6a1b-486d-b95d-4501ad5f9383"). InnerVolumeSpecName "kube-api-access-2wfkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:28:52.065077 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:52.065062 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40c28cc9-6a1b-486d-b95d-4501ad5f9383-maas-api-xks-ca" pod "40c28cc9-6a1b-486d-b95d-4501ad5f9383" (UID: "40c28cc9-6a1b-486d-b95d-4501ad5f9383"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:28:52.163975 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:52.163939 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wfkz\" (UniqueName: \"kubernetes.io/projected/40c28cc9-6a1b-486d-b95d-4501ad5f9383-kube-api-access-2wfkz\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:28:52.163975 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:52.163968 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40c28cc9-6a1b-486d-b95d-4501ad5f9383-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:28:52.163975 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:52.163978 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40c28cc9-6a1b-486d-b95d-4501ad5f9383-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:28:52.854003 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:52.853977 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3ca339-7mrnd" Oct 30 17:28:52.854164 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:52.853971 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3ca339-7mrnd" event={"ID":"40c28cc9-6a1b-486d-b95d-4501ad5f9383","Type":"ContainerDied","Data":"06186feb27a46edf0463f3d2f50a5d10878eb60b082a5964aed8134cbdc2d37b"} Oct 30 17:28:52.854164 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:52.854103 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06186feb27a46edf0463f3d2f50a5d10878eb60b082a5964aed8134cbdc2d37b" Oct 30 17:28:58.628552 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:58.628519 4493 scope.go:122] "RemoveContainer" containerID="98f3de2c58ee2a5126ceae0a0528df90b28d7685c5f99535be8f69f0b302d918" Oct 30 17:28:58.638950 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:58.638931 4493 scope.go:122] "RemoveContainer" containerID="cb92fa4084d6961370fb8be9f926ec76ecc988a2fe152db6a1aa6bfe10a75551" Oct 30 17:28:58.648184 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:28:58.648163 4493 scope.go:122] "RemoveContainer" containerID="5d805ec97f365170da9a04c98da5700edf300b462c207ab6d3a3b1b6403c3192" Oct 30 17:29:00.341263 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:29:00.341228 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 17:29:02.092456 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:02.092421 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3ca339-7mrnd"] Oct 30 17:29:02.103970 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:02.103945 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-3ca339-7mrnd"] Oct 30 17:29:02.933342 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:02.933307 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40c28cc9-6a1b-486d-b95d-4501ad5f9383" path="/var/lib/kubelet/pods/40c28cc9-6a1b-486d-b95d-4501ad5f9383/volumes" Oct 30 17:29:04.475074 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:04.475034 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 17:29:09.413288 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:09.413251 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq"] Oct 30 17:29:09.413934 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:09.413897 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq" podUID="352047e0-584f-4a19-b56a-2a05bd960f73" containerName="istio-proxy" containerID="cri-o://4ed8965da336e6dad873d92dd2bc6c3737edc05ae4e921f269eda44a0a3100d2" gracePeriod=30 Oct 30 17:29:14.082380 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:14.082344 4493 patch_prober.go:29] interesting pod/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:29:14.082775 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:14.082404 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq" podUID="352047e0-584f-4a19-b56a-2a05bd960f73" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:29:14.691029 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:14.690958 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq" Oct 30 17:29:14.783143 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:14.783110 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/352047e0-584f-4a19-b56a-2a05bd960f73-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/352047e0-584f-4a19-b56a-2a05bd960f73-credential-socket\") pod \"352047e0-584f-4a19-b56a-2a05bd960f73\" (UID: \"352047e0-584f-4a19-b56a-2a05bd960f73\") " Oct 30 17:29:14.783351 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:14.783150 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/352047e0-584f-4a19-b56a-2a05bd960f73-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/352047e0-584f-4a19-b56a-2a05bd960f73-istio-data\") pod \"352047e0-584f-4a19-b56a-2a05bd960f73\" (UID: \"352047e0-584f-4a19-b56a-2a05bd960f73\") " Oct 30 17:29:14.783351 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:14.783205 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/352047e0-584f-4a19-b56a-2a05bd960f73-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/352047e0-584f-4a19-b56a-2a05bd960f73-workload-certs\") pod \"352047e0-584f-4a19-b56a-2a05bd960f73\" (UID: \"352047e0-584f-4a19-b56a-2a05bd960f73\") " Oct 30 17:29:14.783351 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:14.783246 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/352047e0-584f-4a19-b56a-2a05bd960f73-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/352047e0-584f-4a19-b56a-2a05bd960f73-istiod-ca-cert\") pod \"352047e0-584f-4a19-b56a-2a05bd960f73\" (UID: \"352047e0-584f-4a19-b56a-2a05bd960f73\") " Oct 30 17:29:14.783351 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:14.783316 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/352047e0-584f-4a19-b56a-2a05bd960f73-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/352047e0-584f-4a19-b56a-2a05bd960f73-istio-envoy\") pod \"352047e0-584f-4a19-b56a-2a05bd960f73\" (UID: \"352047e0-584f-4a19-b56a-2a05bd960f73\") " Oct 30 17:29:14.783596 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:14.783357 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/352047e0-584f-4a19-b56a-2a05bd960f73-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/352047e0-584f-4a19-b56a-2a05bd960f73-istio-podinfo\") pod \"352047e0-584f-4a19-b56a-2a05bd960f73\" (UID: \"352047e0-584f-4a19-b56a-2a05bd960f73\") " Oct 30 17:29:14.783596 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:14.783460 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/352047e0-584f-4a19-b56a-2a05bd960f73-workload-certs" pod "352047e0-584f-4a19-b56a-2a05bd960f73" (UID: "352047e0-584f-4a19-b56a-2a05bd960f73"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:29:14.783596 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:14.783480 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/352047e0-584f-4a19-b56a-2a05bd960f73-kube-api-access-rmvzf\" (UniqueName: \"kubernetes.io/projected/352047e0-584f-4a19-b56a-2a05bd960f73-kube-api-access-rmvzf\") pod \"352047e0-584f-4a19-b56a-2a05bd960f73\" (UID: \"352047e0-584f-4a19-b56a-2a05bd960f73\") " Oct 30 17:29:14.783596 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:14.783538 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/352047e0-584f-4a19-b56a-2a05bd960f73-credential-socket" pod "352047e0-584f-4a19-b56a-2a05bd960f73" (UID: "352047e0-584f-4a19-b56a-2a05bd960f73"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:29:14.783596 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:14.783568 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/352047e0-584f-4a19-b56a-2a05bd960f73-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/352047e0-584f-4a19-b56a-2a05bd960f73-workload-socket\") pod \"352047e0-584f-4a19-b56a-2a05bd960f73\" (UID: \"352047e0-584f-4a19-b56a-2a05bd960f73\") " Oct 30 17:29:14.783908 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:14.783609 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/352047e0-584f-4a19-b56a-2a05bd960f73-istio-token\" (UniqueName: \"kubernetes.io/projected/352047e0-584f-4a19-b56a-2a05bd960f73-istio-token\") pod \"352047e0-584f-4a19-b56a-2a05bd960f73\" (UID: \"352047e0-584f-4a19-b56a-2a05bd960f73\") " Oct 30 17:29:14.783908 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:14.783475 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/352047e0-584f-4a19-b56a-2a05bd960f73-istio-data" pod "352047e0-584f-4a19-b56a-2a05bd960f73" (UID: "352047e0-584f-4a19-b56a-2a05bd960f73"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:29:14.783908 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:14.783746 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/352047e0-584f-4a19-b56a-2a05bd960f73-workload-socket" pod "352047e0-584f-4a19-b56a-2a05bd960f73" (UID: "352047e0-584f-4a19-b56a-2a05bd960f73"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:29:14.783908 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:14.783802 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/352047e0-584f-4a19-b56a-2a05bd960f73-istiod-ca-cert" pod "352047e0-584f-4a19-b56a-2a05bd960f73" (UID: "352047e0-584f-4a19-b56a-2a05bd960f73"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:29:14.784144 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:14.784037 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/352047e0-584f-4a19-b56a-2a05bd960f73-workload-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:29:14.784144 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:14.784061 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/352047e0-584f-4a19-b56a-2a05bd960f73-credential-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:29:14.784144 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:14.784075 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/352047e0-584f-4a19-b56a-2a05bd960f73-istio-data\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:29:14.784144 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:14.784088 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/352047e0-584f-4a19-b56a-2a05bd960f73-workload-certs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:29:14.784144 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:14.784100 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/352047e0-584f-4a19-b56a-2a05bd960f73-istiod-ca-cert\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:29:14.785922 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:14.785876 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/352047e0-584f-4a19-b56a-2a05bd960f73-istio-podinfo" pod "352047e0-584f-4a19-b56a-2a05bd960f73" (UID: "352047e0-584f-4a19-b56a-2a05bd960f73"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:29:14.786453 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:14.786434 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/352047e0-584f-4a19-b56a-2a05bd960f73-kube-api-access-rmvzf" pod "352047e0-584f-4a19-b56a-2a05bd960f73" (UID: "352047e0-584f-4a19-b56a-2a05bd960f73"). InnerVolumeSpecName "kube-api-access-rmvzf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:29:14.786453 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:14.786435 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/352047e0-584f-4a19-b56a-2a05bd960f73-istio-envoy" pod "352047e0-584f-4a19-b56a-2a05bd960f73" (UID: "352047e0-584f-4a19-b56a-2a05bd960f73"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:29:14.786609 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:14.786462 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/352047e0-584f-4a19-b56a-2a05bd960f73-istio-token" pod "352047e0-584f-4a19-b56a-2a05bd960f73" (UID: "352047e0-584f-4a19-b56a-2a05bd960f73"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:29:14.885543 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:14.885511 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmvzf\" (UniqueName: \"kubernetes.io/projected/352047e0-584f-4a19-b56a-2a05bd960f73-kube-api-access-rmvzf\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:29:14.885543 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:14.885542 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/352047e0-584f-4a19-b56a-2a05bd960f73-istio-token\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:29:14.885543 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:14.885552 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/352047e0-584f-4a19-b56a-2a05bd960f73-istio-envoy\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:29:14.885798 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:14.885560 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/352047e0-584f-4a19-b56a-2a05bd960f73-istio-podinfo\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:29:14.957631 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:14.957572 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj"] Oct 30 17:29:14.972113 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:14.972085 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd"] Oct 30 17:29:14.977694 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:14.977660 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s"] Oct 30 17:29:15.025313 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:15.025284 4493 generic.go:356] "Generic (PLEG): container finished" podID="352047e0-584f-4a19-b56a-2a05bd960f73" containerID="4ed8965da336e6dad873d92dd2bc6c3737edc05ae4e921f269eda44a0a3100d2" exitCode=0 Oct 30 17:29:15.025515 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:15.025328 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq" Oct 30 17:29:15.025515 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:15.025375 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq" event={"ID":"352047e0-584f-4a19-b56a-2a05bd960f73","Type":"ContainerDied","Data":"4ed8965da336e6dad873d92dd2bc6c3737edc05ae4e921f269eda44a0a3100d2"} Oct 30 17:29:15.025515 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:15.025436 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq" event={"ID":"352047e0-584f-4a19-b56a-2a05bd960f73","Type":"ContainerDied","Data":"842810a137f081f4b541a4feb377871e495efd9d943e32fcc964e31a9491fe1a"} Oct 30 17:29:15.025515 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:15.025468 4493 scope.go:122] "RemoveContainer" containerID="4ed8965da336e6dad873d92dd2bc6c3737edc05ae4e921f269eda44a0a3100d2" Oct 30 17:29:15.036383 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:15.036366 4493 scope.go:122] "RemoveContainer" containerID="22845bb6cf127000a5e57dfbdfd9a53b0362cf14638a9303ca3ca8cd8394e5a5" Oct 30 17:29:15.047305 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:15.047282 4493 scope.go:122] "RemoveContainer" containerID="4ed8965da336e6dad873d92dd2bc6c3737edc05ae4e921f269eda44a0a3100d2" Oct 30 17:29:15.047605 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:29:15.047584 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ed8965da336e6dad873d92dd2bc6c3737edc05ae4e921f269eda44a0a3100d2\": container with ID starting with 4ed8965da336e6dad873d92dd2bc6c3737edc05ae4e921f269eda44a0a3100d2 not found: ID does not exist" containerID="4ed8965da336e6dad873d92dd2bc6c3737edc05ae4e921f269eda44a0a3100d2" Oct 30 17:29:15.047703 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:15.047617 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ed8965da336e6dad873d92dd2bc6c3737edc05ae4e921f269eda44a0a3100d2"} err="failed to get container status \"4ed8965da336e6dad873d92dd2bc6c3737edc05ae4e921f269eda44a0a3100d2\": rpc error: code = NotFound desc = could not find container \"4ed8965da336e6dad873d92dd2bc6c3737edc05ae4e921f269eda44a0a3100d2\": container with ID starting with 4ed8965da336e6dad873d92dd2bc6c3737edc05ae4e921f269eda44a0a3100d2 not found: ID does not exist" Oct 30 17:29:15.047703 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:15.047639 4493 scope.go:122] "RemoveContainer" containerID="22845bb6cf127000a5e57dfbdfd9a53b0362cf14638a9303ca3ca8cd8394e5a5" Oct 30 17:29:15.048026 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:29:15.048000 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22845bb6cf127000a5e57dfbdfd9a53b0362cf14638a9303ca3ca8cd8394e5a5\": container with ID starting with 22845bb6cf127000a5e57dfbdfd9a53b0362cf14638a9303ca3ca8cd8394e5a5 not found: ID does not exist" containerID="22845bb6cf127000a5e57dfbdfd9a53b0362cf14638a9303ca3ca8cd8394e5a5" Oct 30 17:29:15.048111 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:15.048033 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22845bb6cf127000a5e57dfbdfd9a53b0362cf14638a9303ca3ca8cd8394e5a5"} err="failed to get container status \"22845bb6cf127000a5e57dfbdfd9a53b0362cf14638a9303ca3ca8cd8394e5a5\": rpc error: code = NotFound desc = could not find container \"22845bb6cf127000a5e57dfbdfd9a53b0362cf14638a9303ca3ca8cd8394e5a5\": container with ID starting with 22845bb6cf127000a5e57dfbdfd9a53b0362cf14638a9303ca3ca8cd8394e5a5 not found: ID does not exist" Oct 30 17:29:15.117129 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:15.117097 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq"] Oct 30 17:29:15.121621 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:15.121593 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-qw2fq"] Oct 30 17:29:16.931944 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:16.931912 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="352047e0-584f-4a19-b56a-2a05bd960f73" path="/var/lib/kubelet/pods/352047e0-584f-4a19-b56a-2a05bd960f73/volumes" Oct 30 17:29:20.687905 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:20.687852 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-a77b48-596b55946-4kzrx"] Oct 30 17:29:20.720479 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:20.720451 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-a77b48-699d989875-5w52s"] Oct 30 17:29:21.554348 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:21.554314 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8599a73b-bjkpz"] Oct 30 17:29:21.555026 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:21.555002 4493 cpu_manager.go:414] "containerMap: removing container" podUID="40c28cc9-6a1b-486d-b95d-4501ad5f9383" containerName="revoke-keys" Oct 30 17:29:21.555153 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:21.555028 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40c28cc9-6a1b-486d-b95d-4501ad5f9383" containerName="revoke-keys" Oct 30 17:29:21.555153 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:21.555068 4493 cpu_manager.go:414] "containerMap: removing container" podUID="352047e0-584f-4a19-b56a-2a05bd960f73" containerName="istio-proxy" Oct 30 17:29:21.555153 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:21.555076 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="352047e0-584f-4a19-b56a-2a05bd960f73" containerName="istio-proxy" Oct 30 17:29:21.555337 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:21.555193 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="352047e0-584f-4a19-b56a-2a05bd960f73" containerName="istio-proxy" Oct 30 17:29:21.555337 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:21.555213 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="40c28cc9-6a1b-486d-b95d-4501ad5f9383" containerName="revoke-keys" Oct 30 17:29:21.555337 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:21.555225 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="98af0974-af46-432d-a622-4279717ff916" containerName="istio-proxy" Oct 30 17:29:21.560344 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:21.560318 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8599a73b-bjkpz" Oct 30 17:29:21.580318 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:21.580293 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8599a73b-bjkpz"] Oct 30 17:29:21.641861 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:21.641834 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:29:21.641861 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:21.641859 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 17:29:21.642038 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:21.641876 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 17:29:21.642038 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:21.641877 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 17:29:21.645389 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:21.645371 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11356029-07cb-454d-afbb-edab8a6d250c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8599a73b-bjkpz\" (UID: \"11356029-07cb-454d-afbb-edab8a6d250c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8599a73b-bjkpz" Oct 30 17:29:21.645461 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:21.645416 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd9kg\" (UniqueName: \"kubernetes.io/projected/11356029-07cb-454d-afbb-edab8a6d250c-kube-api-access-vd9kg\") pod \"maas-api-revoke-sub-8599a73b-bjkpz\" (UID: \"11356029-07cb-454d-afbb-edab8a6d250c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8599a73b-bjkpz" Oct 30 17:29:21.645517 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:21.645498 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11356029-07cb-454d-afbb-edab8a6d250c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8599a73b-bjkpz\" (UID: \"11356029-07cb-454d-afbb-edab8a6d250c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8599a73b-bjkpz" Oct 30 17:29:21.746886 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:21.746860 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11356029-07cb-454d-afbb-edab8a6d250c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8599a73b-bjkpz\" (UID: \"11356029-07cb-454d-afbb-edab8a6d250c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8599a73b-bjkpz" Oct 30 17:29:21.747273 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:21.746909 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd9kg\" (UniqueName: \"kubernetes.io/projected/11356029-07cb-454d-afbb-edab8a6d250c-kube-api-access-vd9kg\") pod \"maas-api-revoke-sub-8599a73b-bjkpz\" (UID: \"11356029-07cb-454d-afbb-edab8a6d250c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8599a73b-bjkpz" Oct 30 17:29:21.747273 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:21.747081 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11356029-07cb-454d-afbb-edab8a6d250c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8599a73b-bjkpz\" (UID: \"11356029-07cb-454d-afbb-edab8a6d250c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8599a73b-bjkpz" Oct 30 17:29:21.747539 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:21.747519 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11356029-07cb-454d-afbb-edab8a6d250c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8599a73b-bjkpz\" (UID: \"11356029-07cb-454d-afbb-edab8a6d250c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8599a73b-bjkpz" Oct 30 17:29:21.749595 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:21.749574 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11356029-07cb-454d-afbb-edab8a6d250c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8599a73b-bjkpz\" (UID: \"11356029-07cb-454d-afbb-edab8a6d250c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8599a73b-bjkpz" Oct 30 17:29:21.769825 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:21.769808 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd9kg\" (UniqueName: \"kubernetes.io/projected/11356029-07cb-454d-afbb-edab8a6d250c-kube-api-access-vd9kg\") pod \"maas-api-revoke-sub-8599a73b-bjkpz\" (UID: \"11356029-07cb-454d-afbb-edab8a6d250c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8599a73b-bjkpz" Oct 30 17:29:21.874072 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:21.874017 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8599a73b-bjkpz" Oct 30 17:29:22.188025 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:22.187995 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8599a73b-bjkpz"] Oct 30 17:29:22.190364 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:29:22.190333 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11356029_07cb_454d_afbb_edab8a6d250c.slice/crio-b0c231c31a3b415d32fe157ccb60eed38a4b545e0358e0a9f0675b2883be8f8b WatchSource:0}: Error finding container b0c231c31a3b415d32fe157ccb60eed38a4b545e0358e0a9f0675b2883be8f8b: Status 404 returned error can't find the container with id b0c231c31a3b415d32fe157ccb60eed38a4b545e0358e0a9f0675b2883be8f8b Oct 30 17:29:23.083911 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:23.083871 4493 generic.go:356] "Generic (PLEG): container finished" podID="11356029-07cb-454d-afbb-edab8a6d250c" containerID="8e7abe76a999b9f0eceafb6a1f8807c844eff97723e5e310be912e79272ba20b" exitCode=0 Oct 30 17:29:23.084316 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:23.083917 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8599a73b-bjkpz" event={"ID":"11356029-07cb-454d-afbb-edab8a6d250c","Type":"ContainerDied","Data":"8e7abe76a999b9f0eceafb6a1f8807c844eff97723e5e310be912e79272ba20b"} Oct 30 17:29:23.084316 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:23.083971 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8599a73b-bjkpz" event={"ID":"11356029-07cb-454d-afbb-edab8a6d250c","Type":"ContainerStarted","Data":"b0c231c31a3b415d32fe157ccb60eed38a4b545e0358e0a9f0675b2883be8f8b"} Oct 30 17:29:24.250263 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:24.250237 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8599a73b-bjkpz" Oct 30 17:29:24.375405 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:24.375318 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11356029-07cb-454d-afbb-edab8a6d250c-kube-api-access-vd9kg\" (UniqueName: \"kubernetes.io/projected/11356029-07cb-454d-afbb-edab8a6d250c-kube-api-access-vd9kg\") pod \"11356029-07cb-454d-afbb-edab8a6d250c\" (UID: \"11356029-07cb-454d-afbb-edab8a6d250c\") " Oct 30 17:29:24.375405 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:24.375383 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11356029-07cb-454d-afbb-edab8a6d250c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11356029-07cb-454d-afbb-edab8a6d250c-maas-api-xks-ca\") pod \"11356029-07cb-454d-afbb-edab8a6d250c\" (UID: \"11356029-07cb-454d-afbb-edab8a6d250c\") " Oct 30 17:29:24.375646 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:24.375445 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11356029-07cb-454d-afbb-edab8a6d250c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11356029-07cb-454d-afbb-edab8a6d250c-maas-api-openshift-ca\") pod \"11356029-07cb-454d-afbb-edab8a6d250c\" (UID: \"11356029-07cb-454d-afbb-edab8a6d250c\") " Oct 30 17:29:24.375846 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:24.375823 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11356029-07cb-454d-afbb-edab8a6d250c-maas-api-openshift-ca" pod "11356029-07cb-454d-afbb-edab8a6d250c" (UID: "11356029-07cb-454d-afbb-edab8a6d250c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:29:24.377590 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:24.377561 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11356029-07cb-454d-afbb-edab8a6d250c-maas-api-xks-ca" pod "11356029-07cb-454d-afbb-edab8a6d250c" (UID: "11356029-07cb-454d-afbb-edab8a6d250c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:29:24.378092 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:24.378074 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11356029-07cb-454d-afbb-edab8a6d250c-kube-api-access-vd9kg" pod "11356029-07cb-454d-afbb-edab8a6d250c" (UID: "11356029-07cb-454d-afbb-edab8a6d250c"). InnerVolumeSpecName "kube-api-access-vd9kg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:29:24.477000 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:24.476964 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11356029-07cb-454d-afbb-edab8a6d250c-maas-api-openshift-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:29:24.477000 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:24.476997 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vd9kg\" (UniqueName: \"kubernetes.io/projected/11356029-07cb-454d-afbb-edab8a6d250c-kube-api-access-vd9kg\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:29:24.477000 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:24.477007 4493 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11356029-07cb-454d-afbb-edab8a6d250c-maas-api-xks-ca\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:29:25.134948 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:25.134029 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8599a73b-bjkpz" event={"ID":"11356029-07cb-454d-afbb-edab8a6d250c","Type":"ContainerDied","Data":"b0c231c31a3b415d32fe157ccb60eed38a4b545e0358e0a9f0675b2883be8f8b"} Oct 30 17:29:25.134948 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:25.134096 4493 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b0c231c31a3b415d32fe157ccb60eed38a4b545e0358e0a9f0675b2883be8f8b" Oct 30 17:29:25.134948 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:25.134171 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8599a73b-bjkpz" Oct 30 17:29:25.688916 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:25.688878 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-a77b48-596b55946-4kzrx" podUID="a6aa4382-931c-456e-a739-4580c07adf09" containerName="payload-processing" containerID="cri-o://6c70d0060d3d042006442faf627fd352bb3c0e9a566be62e36b273dcfd789bf0" gracePeriod=25 Oct 30 17:29:25.721663 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:25.721616 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a77b48-699d989875-5w52s" podUID="7a5145a9-7bd1-4f48-b116-6f08637fe08a" containerName="payload-pre-processing" containerID="cri-o://2ec3ec76b0d1bb6e9115b1b812b8b0be6ed98fa99f13f0154a345462696fe920" gracePeriod=25 Oct 30 17:29:29.422076 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:29.422038 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a77b48-699d989875-5w52s" podUID="7a5145a9-7bd1-4f48-b116-6f08637fe08a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:29:29.422494 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:29.422114 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-a77b48-596b55946-4kzrx" podUID="a6aa4382-931c-456e-a739-4580c07adf09" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:29:29.979532 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:29.979483 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" podUID="e90f43d3-f5eb-45ab-af1e-340d0ce18966" containerName="main" containerID="cri-o://2a0643d5e6dd8f438957ba320dd567863f76906a436a3d9bc75da172a37d9a88" gracePeriod=45 Oct 30 17:29:29.994293 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:29.994144 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" podUID="0eadc5d4-98c3-4bf2-b3c2-f73fe249182f" containerName="main" containerID="cri-o://e6f1fd9815791f9f13012d2fb5965ab06f886f5e8993baf69e5419f175b5512a" gracePeriod=45 Oct 30 17:29:30.002211 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:30.002179 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" podUID="e76172b1-659d-45bd-91c6-1a0ce4062335" containerName="main" containerID="cri-o://d2fa10a51b115430a6d73a19fa32de800ee2097a3b3d1812b86855f431596134" gracePeriod=45 Oct 30 17:29:30.175904 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:30.175379 4493 generic.go:356] "Generic (PLEG): container finished" podID="e76172b1-659d-45bd-91c6-1a0ce4062335" containerID="d2fa10a51b115430a6d73a19fa32de800ee2097a3b3d1812b86855f431596134" exitCode=0 Oct 30 17:29:30.175904 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:30.175531 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" event={"ID":"e76172b1-659d-45bd-91c6-1a0ce4062335","Type":"ContainerDied","Data":"d2fa10a51b115430a6d73a19fa32de800ee2097a3b3d1812b86855f431596134"} Oct 30 17:29:30.178801 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:30.177821 4493 generic.go:356] "Generic (PLEG): container finished" podID="e90f43d3-f5eb-45ab-af1e-340d0ce18966" containerID="2a0643d5e6dd8f438957ba320dd567863f76906a436a3d9bc75da172a37d9a88" exitCode=0 Oct 30 17:29:30.178801 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:30.178038 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" event={"ID":"e90f43d3-f5eb-45ab-af1e-340d0ce18966","Type":"ContainerDied","Data":"2a0643d5e6dd8f438957ba320dd567863f76906a436a3d9bc75da172a37d9a88"} Oct 30 17:29:30.182641 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:30.182466 4493 generic.go:356] "Generic (PLEG): container finished" podID="0eadc5d4-98c3-4bf2-b3c2-f73fe249182f" containerID="e6f1fd9815791f9f13012d2fb5965ab06f886f5e8993baf69e5419f175b5512a" exitCode=0 Oct 30 17:29:30.182641 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:30.182544 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" event={"ID":"0eadc5d4-98c3-4bf2-b3c2-f73fe249182f","Type":"ContainerDied","Data":"e6f1fd9815791f9f13012d2fb5965ab06f886f5e8993baf69e5419f175b5512a"} Oct 30 17:29:30.280763 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:30.280731 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" Oct 30 17:29:30.285255 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:30.285227 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" Oct 30 17:29:30.305296 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:30.305271 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" Oct 30 17:29:30.349075 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:29:30.349039 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 17:29:30.354531 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:30.354501 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0eadc5d4-98c3-4bf2-b3c2-f73fe249182f-home\" (UniqueName: \"kubernetes.io/empty-dir/0eadc5d4-98c3-4bf2-b3c2-f73fe249182f-home\") pod \"0eadc5d4-98c3-4bf2-b3c2-f73fe249182f\" (UID: \"0eadc5d4-98c3-4bf2-b3c2-f73fe249182f\") " Oct 30 17:29:30.354761 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:30.354549 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e90f43d3-f5eb-45ab-af1e-340d0ce18966-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e90f43d3-f5eb-45ab-af1e-340d0ce18966-dshm\") pod \"e90f43d3-f5eb-45ab-af1e-340d0ce18966\" (UID: \"e90f43d3-f5eb-45ab-af1e-340d0ce18966\") " Oct 30 17:29:30.354761 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:30.354603 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e90f43d3-f5eb-45ab-af1e-340d0ce18966-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e90f43d3-f5eb-45ab-af1e-340d0ce18966-tmp-dir\") pod \"e90f43d3-f5eb-45ab-af1e-340d0ce18966\" (UID: \"e90f43d3-f5eb-45ab-af1e-340d0ce18966\") " Oct 30 17:29:30.354761 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:30.354656 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0eadc5d4-98c3-4bf2-b3c2-f73fe249182f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0eadc5d4-98c3-4bf2-b3c2-f73fe249182f-model-cache\") pod \"0eadc5d4-98c3-4bf2-b3c2-f73fe249182f\" (UID: \"0eadc5d4-98c3-4bf2-b3c2-f73fe249182f\") " Oct 30 17:29:30.354761 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:30.354689 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e76172b1-659d-45bd-91c6-1a0ce4062335-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e76172b1-659d-45bd-91c6-1a0ce4062335-tmp-dir\") pod \"e76172b1-659d-45bd-91c6-1a0ce4062335\" (UID: \"e76172b1-659d-45bd-91c6-1a0ce4062335\") " Oct 30 17:29:30.354761 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:30.354718 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0eadc5d4-98c3-4bf2-b3c2-f73fe249182f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0eadc5d4-98c3-4bf2-b3c2-f73fe249182f-tmp-dir\") pod \"0eadc5d4-98c3-4bf2-b3c2-f73fe249182f\" (UID: \"0eadc5d4-98c3-4bf2-b3c2-f73fe249182f\") " Oct 30 17:29:30.354761 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:30.354749 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e76172b1-659d-45bd-91c6-1a0ce4062335-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e76172b1-659d-45bd-91c6-1a0ce4062335-dshm\") pod \"e76172b1-659d-45bd-91c6-1a0ce4062335\" (UID: \"e76172b1-659d-45bd-91c6-1a0ce4062335\") " Oct 30 17:29:30.356842 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:30.354820 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eadc5d4-98c3-4bf2-b3c2-f73fe249182f-home" pod "0eadc5d4-98c3-4bf2-b3c2-f73fe249182f" (UID: "0eadc5d4-98c3-4bf2-b3c2-f73fe249182f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:29:30.356842 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:30.354833 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0eadc5d4-98c3-4bf2-b3c2-f73fe249182f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0eadc5d4-98c3-4bf2-b3c2-f73fe249182f-dshm\") pod \"0eadc5d4-98c3-4bf2-b3c2-f73fe249182f\" (UID: \"0eadc5d4-98c3-4bf2-b3c2-f73fe249182f\") " Oct 30 17:29:30.356842 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:30.354869 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e76172b1-659d-45bd-91c6-1a0ce4062335-kube-api-access-7gcnl\" (UniqueName: \"kubernetes.io/projected/e76172b1-659d-45bd-91c6-1a0ce4062335-kube-api-access-7gcnl\") pod \"e76172b1-659d-45bd-91c6-1a0ce4062335\" (UID: \"e76172b1-659d-45bd-91c6-1a0ce4062335\") " Oct 30 17:29:30.356842 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:30.354921 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e90f43d3-f5eb-45ab-af1e-340d0ce18966-kube-api-access-pffkv\" (UniqueName: \"kubernetes.io/projected/e90f43d3-f5eb-45ab-af1e-340d0ce18966-kube-api-access-pffkv\") pod \"e90f43d3-f5eb-45ab-af1e-340d0ce18966\" (UID: \"e90f43d3-f5eb-45ab-af1e-340d0ce18966\") " Oct 30 17:29:30.356842 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:30.354956 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e90f43d3-f5eb-45ab-af1e-340d0ce18966-home\" (UniqueName: \"kubernetes.io/empty-dir/e90f43d3-f5eb-45ab-af1e-340d0ce18966-home\") pod \"e90f43d3-f5eb-45ab-af1e-340d0ce18966\" (UID: \"e90f43d3-f5eb-45ab-af1e-340d0ce18966\") " Oct 30 17:29:30.356842 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:30.354987 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0eadc5d4-98c3-4bf2-b3c2-f73fe249182f-kube-api-access-tt22l\" (UniqueName: \"kubernetes.io/projected/0eadc5d4-98c3-4bf2-b3c2-f73fe249182f-kube-api-access-tt22l\") pod \"0eadc5d4-98c3-4bf2-b3c2-f73fe249182f\" (UID: \"0eadc5d4-98c3-4bf2-b3c2-f73fe249182f\") " Oct 30 17:29:30.356842 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:30.354979 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e90f43d3-f5eb-45ab-af1e-340d0ce18966-tmp-dir" pod "e90f43d3-f5eb-45ab-af1e-340d0ce18966" (UID: "e90f43d3-f5eb-45ab-af1e-340d0ce18966"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:29:30.356842 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:30.355037 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eadc5d4-98c3-4bf2-b3c2-f73fe249182f-tmp-dir" pod "0eadc5d4-98c3-4bf2-b3c2-f73fe249182f" (UID: "0eadc5d4-98c3-4bf2-b3c2-f73fe249182f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:29:30.356842 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:30.355051 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e90f43d3-f5eb-45ab-af1e-340d0ce18966-tls-certs\" (UniqueName: \"kubernetes.io/secret/e90f43d3-f5eb-45ab-af1e-340d0ce18966-tls-certs\") pod \"e90f43d3-f5eb-45ab-af1e-340d0ce18966\" (UID: \"e90f43d3-f5eb-45ab-af1e-340d0ce18966\") " Oct 30 17:29:30.356842 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:30.355091 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0eadc5d4-98c3-4bf2-b3c2-f73fe249182f-tls-certs\" (UniqueName: \"kubernetes.io/secret/0eadc5d4-98c3-4bf2-b3c2-f73fe249182f-tls-certs\") pod \"0eadc5d4-98c3-4bf2-b3c2-f73fe249182f\" (UID: \"0eadc5d4-98c3-4bf2-b3c2-f73fe249182f\") " Oct 30 17:29:30.356842 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:30.355118 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e76172b1-659d-45bd-91c6-1a0ce4062335-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e76172b1-659d-45bd-91c6-1a0ce4062335-model-cache\") pod \"e76172b1-659d-45bd-91c6-1a0ce4062335\" (UID: \"e76172b1-659d-45bd-91c6-1a0ce4062335\") " Oct 30 17:29:30.356842 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:30.355123 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eadc5d4-98c3-4bf2-b3c2-f73fe249182f-model-cache" pod "0eadc5d4-98c3-4bf2-b3c2-f73fe249182f" (UID: "0eadc5d4-98c3-4bf2-b3c2-f73fe249182f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:29:30.356842 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:30.355161 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e76172b1-659d-45bd-91c6-1a0ce4062335-home\" (UniqueName: \"kubernetes.io/empty-dir/e76172b1-659d-45bd-91c6-1a0ce4062335-home\") pod \"e76172b1-659d-45bd-91c6-1a0ce4062335\" (UID: \"e76172b1-659d-45bd-91c6-1a0ce4062335\") " Oct 30 17:29:30.356842 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:30.355189 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e76172b1-659d-45bd-91c6-1a0ce4062335-tls-certs\" (UniqueName: \"kubernetes.io/secret/e76172b1-659d-45bd-91c6-1a0ce4062335-tls-certs\") pod \"e76172b1-659d-45bd-91c6-1a0ce4062335\" (UID: \"e76172b1-659d-45bd-91c6-1a0ce4062335\") " Oct 30 17:29:30.356842 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:30.355250 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e90f43d3-f5eb-45ab-af1e-340d0ce18966-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e90f43d3-f5eb-45ab-af1e-340d0ce18966-model-cache\") pod \"e90f43d3-f5eb-45ab-af1e-340d0ce18966\" (UID: \"e90f43d3-f5eb-45ab-af1e-340d0ce18966\") " Oct 30 17:29:30.356842 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:30.355584 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e76172b1-659d-45bd-91c6-1a0ce4062335-model-cache" pod "e76172b1-659d-45bd-91c6-1a0ce4062335" (UID: "e76172b1-659d-45bd-91c6-1a0ce4062335"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:29:30.356842 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:30.355661 4493 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e90f43d3-f5eb-45ab-af1e-340d0ce18966-tmp-dir\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:29:30.356842 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:30.355683 4493 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0eadc5d4-98c3-4bf2-b3c2-f73fe249182f-model-cache\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:29:30.356842 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:30.355696 4493 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0eadc5d4-98c3-4bf2-b3c2-f73fe249182f-tmp-dir\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:29:30.356842 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:30.355708 4493 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0eadc5d4-98c3-4bf2-b3c2-f73fe249182f-home\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:29:30.356842 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:30.355967 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e76172b1-659d-45bd-91c6-1a0ce4062335-home" pod "e76172b1-659d-45bd-91c6-1a0ce4062335" (UID: "e76172b1-659d-45bd-91c6-1a0ce4062335"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:29:30.356842 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:30.356823 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e90f43d3-f5eb-45ab-af1e-340d0ce18966-model-cache" pod "e90f43d3-f5eb-45ab-af1e-340d0ce18966" (UID: "e90f43d3-f5eb-45ab-af1e-340d0ce18966"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:29:30.358408 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:30.356880 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e76172b1-659d-45bd-91c6-1a0ce4062335-tmp-dir" pod "e76172b1-659d-45bd-91c6-1a0ce4062335" (UID: "e76172b1-659d-45bd-91c6-1a0ce4062335"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:29:30.358408 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:30.357116 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e90f43d3-f5eb-45ab-af1e-340d0ce18966-home" pod "e90f43d3-f5eb-45ab-af1e-340d0ce18966" (UID: "e90f43d3-f5eb-45ab-af1e-340d0ce18966"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:29:30.358761 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:30.358736 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e76172b1-659d-45bd-91c6-1a0ce4062335-tls-certs" pod "e76172b1-659d-45bd-91c6-1a0ce4062335" (UID: "e76172b1-659d-45bd-91c6-1a0ce4062335"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:29:30.359517 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:30.359464 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e76172b1-659d-45bd-91c6-1a0ce4062335-dshm" pod "e76172b1-659d-45bd-91c6-1a0ce4062335" (UID: "e76172b1-659d-45bd-91c6-1a0ce4062335"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:29:30.360067 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:30.360018 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eadc5d4-98c3-4bf2-b3c2-f73fe249182f-kube-api-access-tt22l" pod "0eadc5d4-98c3-4bf2-b3c2-f73fe249182f" (UID: "0eadc5d4-98c3-4bf2-b3c2-f73fe249182f"). InnerVolumeSpecName "kube-api-access-tt22l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:29:30.360175 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:30.360095 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eadc5d4-98c3-4bf2-b3c2-f73fe249182f-dshm" pod "0eadc5d4-98c3-4bf2-b3c2-f73fe249182f" (UID: "0eadc5d4-98c3-4bf2-b3c2-f73fe249182f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:29:30.361008 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:30.360978 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e90f43d3-f5eb-45ab-af1e-340d0ce18966-kube-api-access-pffkv" pod "e90f43d3-f5eb-45ab-af1e-340d0ce18966" (UID: "e90f43d3-f5eb-45ab-af1e-340d0ce18966"). InnerVolumeSpecName "kube-api-access-pffkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:29:30.361942 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:30.361922 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e90f43d3-f5eb-45ab-af1e-340d0ce18966-dshm" pod "e90f43d3-f5eb-45ab-af1e-340d0ce18966" (UID: "e90f43d3-f5eb-45ab-af1e-340d0ce18966"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:29:30.362989 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:30.362966 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0eadc5d4-98c3-4bf2-b3c2-f73fe249182f-tls-certs" pod "0eadc5d4-98c3-4bf2-b3c2-f73fe249182f" (UID: "0eadc5d4-98c3-4bf2-b3c2-f73fe249182f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:29:30.363075 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:30.363035 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e90f43d3-f5eb-45ab-af1e-340d0ce18966-tls-certs" pod "e90f43d3-f5eb-45ab-af1e-340d0ce18966" (UID: "e90f43d3-f5eb-45ab-af1e-340d0ce18966"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:29:30.363131 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:30.363060 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e76172b1-659d-45bd-91c6-1a0ce4062335-kube-api-access-7gcnl" pod "e76172b1-659d-45bd-91c6-1a0ce4062335" (UID: "e76172b1-659d-45bd-91c6-1a0ce4062335"). InnerVolumeSpecName "kube-api-access-7gcnl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:29:30.456274 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:30.456238 4493 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e90f43d3-f5eb-45ab-af1e-340d0ce18966-dshm\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:29:30.456709 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:30.456297 4493 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e76172b1-659d-45bd-91c6-1a0ce4062335-tmp-dir\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:29:30.456709 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:30.456313 4493 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e76172b1-659d-45bd-91c6-1a0ce4062335-dshm\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:29:30.456709 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:30.456323 4493 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0eadc5d4-98c3-4bf2-b3c2-f73fe249182f-dshm\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:29:30.456709 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:30.456337 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gcnl\" (UniqueName: \"kubernetes.io/projected/e76172b1-659d-45bd-91c6-1a0ce4062335-kube-api-access-7gcnl\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:29:30.456709 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:30.456352 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pffkv\" (UniqueName: \"kubernetes.io/projected/e90f43d3-f5eb-45ab-af1e-340d0ce18966-kube-api-access-pffkv\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:29:30.456709 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:30.456364 4493 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e90f43d3-f5eb-45ab-af1e-340d0ce18966-home\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:29:30.456709 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:30.456378 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tt22l\" (UniqueName: \"kubernetes.io/projected/0eadc5d4-98c3-4bf2-b3c2-f73fe249182f-kube-api-access-tt22l\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:29:30.456709 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:30.456390 4493 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e90f43d3-f5eb-45ab-af1e-340d0ce18966-tls-certs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:29:30.456709 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:30.456403 4493 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0eadc5d4-98c3-4bf2-b3c2-f73fe249182f-tls-certs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:29:30.456709 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:30.456417 4493 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e76172b1-659d-45bd-91c6-1a0ce4062335-model-cache\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:29:30.456709 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:30.456430 4493 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e76172b1-659d-45bd-91c6-1a0ce4062335-home\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:29:30.456709 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:30.456442 4493 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e76172b1-659d-45bd-91c6-1a0ce4062335-tls-certs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:29:30.456709 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:30.456455 4493 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e90f43d3-f5eb-45ab-af1e-340d0ce18966-model-cache\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:29:31.191480 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:31.191395 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" Oct 30 17:29:31.191480 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:31.191418 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" event={"ID":"e90f43d3-f5eb-45ab-af1e-340d0ce18966","Type":"ContainerDied","Data":"d0ed18014cc41a718118239104d1090c21977177c4293720197a164dd67edd09"} Oct 30 17:29:31.191480 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:31.191465 4493 scope.go:122] "RemoveContainer" containerID="2a0643d5e6dd8f438957ba320dd567863f76906a436a3d9bc75da172a37d9a88" Oct 30 17:29:31.193287 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:31.193267 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" Oct 30 17:29:31.193467 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:31.193310 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd" event={"ID":"0eadc5d4-98c3-4bf2-b3c2-f73fe249182f","Type":"ContainerDied","Data":"ed579b24529bb9a3949947f4de558316e24e67288fb3b5684bdee90762eea507"} Oct 30 17:29:31.195172 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:31.195148 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" event={"ID":"e76172b1-659d-45bd-91c6-1a0ce4062335","Type":"ContainerDied","Data":"9533242dfa45531743738f256243d2178534a2461f29f39801c2f3bb3e62e84d"} Oct 30 17:29:31.195291 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:31.195178 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" Oct 30 17:29:31.203521 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:31.203490 4493 scope.go:122] "RemoveContainer" containerID="e6f1fd9815791f9f13012d2fb5965ab06f886f5e8993baf69e5419f175b5512a" Oct 30 17:29:31.213134 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:31.213104 4493 scope.go:122] "RemoveContainer" containerID="d2fa10a51b115430a6d73a19fa32de800ee2097a3b3d1812b86855f431596134" Oct 30 17:29:31.242121 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:31.242092 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s" podUID="e76172b1-659d-45bd-91c6-1a0ce4062335" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:29:31.243113 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:31.243094 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj"] Oct 30 17:29:31.255003 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:31.254969 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj" podUID="e90f43d3-f5eb-45ab-af1e-340d0ce18966" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded" Oct 30 17:29:31.255105 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:31.254987 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-a77b48/distinct-w1-a77b48-kserve-946867657-kpkbj"] Oct 30 17:29:31.287305 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:31.287284 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd"] Oct 30 17:29:31.291720 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:31.291699 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-a77b48/premium-w1-a77b48-kserve-844856b7c4-rfxxd"] Oct 30 17:29:31.324843 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:31.324818 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s"] Oct 30 17:29:31.330411 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:31.330390 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-a77b48/simulator-w1-a77b48-kserve-569bdfdd67-rd82s"] Oct 30 17:29:32.932148 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:32.932111 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0eadc5d4-98c3-4bf2-b3c2-f73fe249182f" path="/var/lib/kubelet/pods/0eadc5d4-98c3-4bf2-b3c2-f73fe249182f/volumes" Oct 30 17:29:32.932620 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:32.932523 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e76172b1-659d-45bd-91c6-1a0ce4062335" path="/var/lib/kubelet/pods/e76172b1-659d-45bd-91c6-1a0ce4062335/volumes" Oct 30 17:29:32.932905 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:32.932890 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e90f43d3-f5eb-45ab-af1e-340d0ce18966" path="/var/lib/kubelet/pods/e90f43d3-f5eb-45ab-af1e-340d0ce18966/volumes" Oct 30 17:29:34.461676 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:34.461591 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 17:29:39.421734 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:39.421696 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a77b48-699d989875-5w52s" podUID="7a5145a9-7bd1-4f48-b116-6f08637fe08a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:29:39.422373 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:39.421698 4493 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-a77b48-596b55946-4kzrx" podUID="a6aa4382-931c-456e-a739-4580c07adf09" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:29:44.294666 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:44.294632 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-k9d99"] Oct 30 17:29:44.408636 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:44.408601 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4"] Oct 30 17:29:44.409140 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:44.409110 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4" podUID="da5ec9e6-8379-46ee-a2d9-2360237ccf2d" containerName="istio-proxy" containerID="cri-o://d42783125009f74712772bb9386d5740e54727c4ede5e78a24b84706230eecea" gracePeriod=30 Oct 30 17:29:45.588933 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:45.588906 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-78bcf8868f-w5c9k_ad767d7d-7fb3-480d-bf9b-6583a00936fb/authorino/0.log" Oct 30 17:29:45.834542 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:45.834514 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-gzvkk_39b545e1-1a61-4ddf-a00a-f396058e6a88/manager/0.log" Oct 30 17:29:45.979965 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:45.979944 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-a77b48-596b55946-4kzrx" Oct 30 17:29:46.004610 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:46.004590 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a77b48-699d989875-5w52s" Oct 30 17:29:46.038491 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:46.038466 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-8xm6t_cf7e09ae-b9ba-420d-b071-bc2f5ac8b009/manager/0.log" Oct 30 17:29:46.103704 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:46.103670 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a5145a9-7bd1-4f48-b116-6f08637fe08a-kube-api-access-fkf27\" (UniqueName: \"kubernetes.io/projected/7a5145a9-7bd1-4f48-b116-6f08637fe08a-kube-api-access-fkf27\") pod \"7a5145a9-7bd1-4f48-b116-6f08637fe08a\" (UID: \"7a5145a9-7bd1-4f48-b116-6f08637fe08a\") " Oct 30 17:29:46.103892 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:46.103772 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6aa4382-931c-456e-a739-4580c07adf09-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6aa4382-931c-456e-a739-4580c07adf09-plugins-config-volume\") pod \"a6aa4382-931c-456e-a739-4580c07adf09\" (UID: \"a6aa4382-931c-456e-a739-4580c07adf09\") " Oct 30 17:29:46.103892 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:46.103851 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a5145a9-7bd1-4f48-b116-6f08637fe08a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a5145a9-7bd1-4f48-b116-6f08637fe08a-plugins-config-volume\") pod \"7a5145a9-7bd1-4f48-b116-6f08637fe08a\" (UID: \"7a5145a9-7bd1-4f48-b116-6f08637fe08a\") " Oct 30 17:29:46.103995 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:46.103929 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6aa4382-931c-456e-a739-4580c07adf09-kube-api-access-65nz6\" (UniqueName: \"kubernetes.io/projected/a6aa4382-931c-456e-a739-4580c07adf09-kube-api-access-65nz6\") pod \"a6aa4382-931c-456e-a739-4580c07adf09\" (UID: \"a6aa4382-931c-456e-a739-4580c07adf09\") " Oct 30 17:29:46.104178 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:46.104157 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6aa4382-931c-456e-a739-4580c07adf09-plugins-config-volume" pod "a6aa4382-931c-456e-a739-4580c07adf09" (UID: "a6aa4382-931c-456e-a739-4580c07adf09"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:29:46.104241 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:46.104199 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a5145a9-7bd1-4f48-b116-6f08637fe08a-plugins-config-volume" pod "7a5145a9-7bd1-4f48-b116-6f08637fe08a" (UID: "7a5145a9-7bd1-4f48-b116-6f08637fe08a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:29:46.104311 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:46.104290 4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6aa4382-931c-456e-a739-4580c07adf09-plugins-config-volume\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:29:46.104311 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:46.104304 4493 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a5145a9-7bd1-4f48-b116-6f08637fe08a-plugins-config-volume\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:29:46.106210 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:46.106190 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a5145a9-7bd1-4f48-b116-6f08637fe08a-kube-api-access-fkf27" pod "7a5145a9-7bd1-4f48-b116-6f08637fe08a" (UID: "7a5145a9-7bd1-4f48-b116-6f08637fe08a"). InnerVolumeSpecName "kube-api-access-fkf27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:29:46.106281 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:46.106265 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6aa4382-931c-456e-a739-4580c07adf09-kube-api-access-65nz6" pod "a6aa4382-931c-456e-a739-4580c07adf09" (UID: "a6aa4382-931c-456e-a739-4580c07adf09"). InnerVolumeSpecName "kube-api-access-65nz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:29:46.205073 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:46.205032 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkf27\" (UniqueName: \"kubernetes.io/projected/7a5145a9-7bd1-4f48-b116-6f08637fe08a-kube-api-access-fkf27\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:29:46.205073 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:46.205064 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65nz6\" (UniqueName: \"kubernetes.io/projected/a6aa4382-931c-456e-a739-4580c07adf09-kube-api-access-65nz6\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:29:46.320693 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:46.320659 4493 generic.go:356] "Generic (PLEG): container finished" podID="7a5145a9-7bd1-4f48-b116-6f08637fe08a" containerID="2ec3ec76b0d1bb6e9115b1b812b8b0be6ed98fa99f13f0154a345462696fe920" exitCode=0 Oct 30 17:29:46.320911 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:46.320714 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a77b48-699d989875-5w52s" Oct 30 17:29:46.320911 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:46.320763 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a77b48-699d989875-5w52s" event={"ID":"7a5145a9-7bd1-4f48-b116-6f08637fe08a","Type":"ContainerDied","Data":"2ec3ec76b0d1bb6e9115b1b812b8b0be6ed98fa99f13f0154a345462696fe920"} Oct 30 17:29:46.320911 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:46.320833 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a77b48-699d989875-5w52s" event={"ID":"7a5145a9-7bd1-4f48-b116-6f08637fe08a","Type":"ContainerDied","Data":"8d2c92c33e615616991d0517ee5374deb2d91a22095c75442ff8dd43660a08a1"} Oct 30 17:29:46.320911 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:46.320863 4493 scope.go:122] "RemoveContainer" containerID="2ec3ec76b0d1bb6e9115b1b812b8b0be6ed98fa99f13f0154a345462696fe920" Oct 30 17:29:46.322800 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:46.322761 4493 generic.go:356] "Generic (PLEG): container finished" podID="a6aa4382-931c-456e-a739-4580c07adf09" containerID="6c70d0060d3d042006442faf627fd352bb3c0e9a566be62e36b273dcfd789bf0" exitCode=0 Oct 30 17:29:46.323014 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:46.322995 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-a77b48-596b55946-4kzrx" event={"ID":"a6aa4382-931c-456e-a739-4580c07adf09","Type":"ContainerDied","Data":"6c70d0060d3d042006442faf627fd352bb3c0e9a566be62e36b273dcfd789bf0"} Oct 30 17:29:46.323099 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:46.323043 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-a77b48-596b55946-4kzrx" event={"ID":"a6aa4382-931c-456e-a739-4580c07adf09","Type":"ContainerDied","Data":"15a00ba2bb7902055a3df4c548ce483fc51294aff36be4ca9e713f101ee3ca40"} Oct 30 17:29:46.323164 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:46.323106 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-a77b48-596b55946-4kzrx" Oct 30 17:29:46.333329 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:46.333308 4493 scope.go:122] "RemoveContainer" containerID="2ec3ec76b0d1bb6e9115b1b812b8b0be6ed98fa99f13f0154a345462696fe920" Oct 30 17:29:46.333654 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:29:46.333634 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ec3ec76b0d1bb6e9115b1b812b8b0be6ed98fa99f13f0154a345462696fe920\": container with ID starting with 2ec3ec76b0d1bb6e9115b1b812b8b0be6ed98fa99f13f0154a345462696fe920 not found: ID does not exist" containerID="2ec3ec76b0d1bb6e9115b1b812b8b0be6ed98fa99f13f0154a345462696fe920" Oct 30 17:29:46.333776 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:46.333667 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ec3ec76b0d1bb6e9115b1b812b8b0be6ed98fa99f13f0154a345462696fe920"} err="failed to get container status \"2ec3ec76b0d1bb6e9115b1b812b8b0be6ed98fa99f13f0154a345462696fe920\": rpc error: code = NotFound desc = could not find container \"2ec3ec76b0d1bb6e9115b1b812b8b0be6ed98fa99f13f0154a345462696fe920\": container with ID starting with 2ec3ec76b0d1bb6e9115b1b812b8b0be6ed98fa99f13f0154a345462696fe920 not found: ID does not exist" Oct 30 17:29:46.333776 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:46.333687 4493 scope.go:122] "RemoveContainer" containerID="6c70d0060d3d042006442faf627fd352bb3c0e9a566be62e36b273dcfd789bf0" Oct 30 17:29:46.346208 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:46.346183 4493 scope.go:122] "RemoveContainer" containerID="6c70d0060d3d042006442faf627fd352bb3c0e9a566be62e36b273dcfd789bf0" Oct 30 17:29:46.346526 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:29:46.346505 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c70d0060d3d042006442faf627fd352bb3c0e9a566be62e36b273dcfd789bf0\": container with ID starting with 6c70d0060d3d042006442faf627fd352bb3c0e9a566be62e36b273dcfd789bf0 not found: ID does not exist" containerID="6c70d0060d3d042006442faf627fd352bb3c0e9a566be62e36b273dcfd789bf0" Oct 30 17:29:46.346602 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:46.346534 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c70d0060d3d042006442faf627fd352bb3c0e9a566be62e36b273dcfd789bf0"} err="failed to get container status \"6c70d0060d3d042006442faf627fd352bb3c0e9a566be62e36b273dcfd789bf0\": rpc error: code = NotFound desc = could not find container \"6c70d0060d3d042006442faf627fd352bb3c0e9a566be62e36b273dcfd789bf0\": container with ID starting with 6c70d0060d3d042006442faf627fd352bb3c0e9a566be62e36b273dcfd789bf0 not found: ID does not exist" Oct 30 17:29:46.372359 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:46.372329 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-a77b48-596b55946-4kzrx"] Oct 30 17:29:46.376687 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:46.376662 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-a77b48-596b55946-4kzrx"] Oct 30 17:29:46.412334 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:46.412303 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-a77b48-699d989875-5w52s"] Oct 30 17:29:46.417998 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:46.417970 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-a77b48-699d989875-5w52s"] Oct 30 17:29:46.661422 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:46.661379 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-kksz8_6594d922-089d-41e9-9444-5a888d5fa675/manager/0.log" Oct 30 17:29:46.868878 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:46.868839 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-k9d99_19632c83-f396-437f-ae2d-93229596ce61/limitador/0.log" Oct 30 17:29:46.932615 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:46.932538 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a5145a9-7bd1-4f48-b116-6f08637fe08a" path="/var/lib/kubelet/pods/7a5145a9-7bd1-4f48-b116-6f08637fe08a/volumes" Oct 30 17:29:46.932942 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:46.932928 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6aa4382-931c-456e-a739-4580c07adf09" path="/var/lib/kubelet/pods/a6aa4382-931c-456e-a739-4580c07adf09/volumes" Oct 30 17:29:47.001480 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:47.001448 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5kzmj"] Oct 30 17:29:47.001683 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:47.001657 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5kzmj" podUID="439163d6-6cb9-4b15-bcae-2f6f6cdecde2" containerName="discovery" containerID="cri-o://521b68db462ca55a97f83f15e37756bec7728705237c57a8a856d1fde46d9187" gracePeriod=30 Oct 30 17:29:47.087442 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:47.087415 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-h6wgm_3319a971-2f82-469a-8fc3-20687e710719/manager/1.log" Oct 30 17:29:47.264230 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:47.264205 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5kzmj" Oct 30 17:29:47.332555 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:47.332525 4493 generic.go:356] "Generic (PLEG): container finished" podID="439163d6-6cb9-4b15-bcae-2f6f6cdecde2" containerID="521b68db462ca55a97f83f15e37756bec7728705237c57a8a856d1fde46d9187" exitCode=0 Oct 30 17:29:47.332737 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:47.332571 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5kzmj" Oct 30 17:29:47.332737 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:47.332620 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5kzmj" event={"ID":"439163d6-6cb9-4b15-bcae-2f6f6cdecde2","Type":"ContainerDied","Data":"521b68db462ca55a97f83f15e37756bec7728705237c57a8a856d1fde46d9187"} Oct 30 17:29:47.332737 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:47.332679 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5kzmj" event={"ID":"439163d6-6cb9-4b15-bcae-2f6f6cdecde2","Type":"ContainerDied","Data":"e32eb5e5358db9c173618433ddc5839863830e9c9a42420aae08e41203d8848e"} Oct 30 17:29:47.332737 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:47.332710 4493 scope.go:122] "RemoveContainer" containerID="521b68db462ca55a97f83f15e37756bec7728705237c57a8a856d1fde46d9187" Oct 30 17:29:47.346263 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:47.346241 4493 scope.go:122] "RemoveContainer" containerID="521b68db462ca55a97f83f15e37756bec7728705237c57a8a856d1fde46d9187" Oct 30 17:29:47.346552 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:29:47.346529 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"521b68db462ca55a97f83f15e37756bec7728705237c57a8a856d1fde46d9187\": container with ID starting with 521b68db462ca55a97f83f15e37756bec7728705237c57a8a856d1fde46d9187 not found: ID does not exist" containerID="521b68db462ca55a97f83f15e37756bec7728705237c57a8a856d1fde46d9187" Oct 30 17:29:47.346655 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:47.346564 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"521b68db462ca55a97f83f15e37756bec7728705237c57a8a856d1fde46d9187"} err="failed to get container status \"521b68db462ca55a97f83f15e37756bec7728705237c57a8a856d1fde46d9187\": rpc error: code = NotFound desc = could not find container \"521b68db462ca55a97f83f15e37756bec7728705237c57a8a856d1fde46d9187\": container with ID starting with 521b68db462ca55a97f83f15e37756bec7728705237c57a8a856d1fde46d9187 not found: ID does not exist" Oct 30 17:29:47.415533 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:47.415498 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/439163d6-6cb9-4b15-bcae-2f6f6cdecde2-cacerts\" (UniqueName: \"kubernetes.io/secret/439163d6-6cb9-4b15-bcae-2f6f6cdecde2-cacerts\") pod \"439163d6-6cb9-4b15-bcae-2f6f6cdecde2\" (UID: \"439163d6-6cb9-4b15-bcae-2f6f6cdecde2\") " Oct 30 17:29:47.415730 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:47.415591 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/439163d6-6cb9-4b15-bcae-2f6f6cdecde2-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/439163d6-6cb9-4b15-bcae-2f6f6cdecde2-istio-csr-dns-cert\") pod \"439163d6-6cb9-4b15-bcae-2f6f6cdecde2\" (UID: \"439163d6-6cb9-4b15-bcae-2f6f6cdecde2\") " Oct 30 17:29:47.415730 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:47.415630 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/439163d6-6cb9-4b15-bcae-2f6f6cdecde2-kube-api-access-hknxx\" (UniqueName: \"kubernetes.io/projected/439163d6-6cb9-4b15-bcae-2f6f6cdecde2-kube-api-access-hknxx\") pod \"439163d6-6cb9-4b15-bcae-2f6f6cdecde2\" (UID: \"439163d6-6cb9-4b15-bcae-2f6f6cdecde2\") " Oct 30 17:29:47.415900 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:47.415732 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/439163d6-6cb9-4b15-bcae-2f6f6cdecde2-istio-token\" (UniqueName: \"kubernetes.io/projected/439163d6-6cb9-4b15-bcae-2f6f6cdecde2-istio-token\") pod \"439163d6-6cb9-4b15-bcae-2f6f6cdecde2\" (UID: \"439163d6-6cb9-4b15-bcae-2f6f6cdecde2\") " Oct 30 17:29:47.415900 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:47.415778 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/439163d6-6cb9-4b15-bcae-2f6f6cdecde2-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/439163d6-6cb9-4b15-bcae-2f6f6cdecde2-local-certs\") pod \"439163d6-6cb9-4b15-bcae-2f6f6cdecde2\" (UID: \"439163d6-6cb9-4b15-bcae-2f6f6cdecde2\") " Oct 30 17:29:47.415900 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:47.415841 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/439163d6-6cb9-4b15-bcae-2f6f6cdecde2-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/439163d6-6cb9-4b15-bcae-2f6f6cdecde2-istio-kubeconfig\") pod \"439163d6-6cb9-4b15-bcae-2f6f6cdecde2\" (UID: \"439163d6-6cb9-4b15-bcae-2f6f6cdecde2\") " Oct 30 17:29:47.415900 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:47.415881 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/439163d6-6cb9-4b15-bcae-2f6f6cdecde2-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/439163d6-6cb9-4b15-bcae-2f6f6cdecde2-istio-csr-ca-configmap\") pod \"439163d6-6cb9-4b15-bcae-2f6f6cdecde2\" (UID: \"439163d6-6cb9-4b15-bcae-2f6f6cdecde2\") " Oct 30 17:29:47.416520 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:47.416493 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/439163d6-6cb9-4b15-bcae-2f6f6cdecde2-istio-csr-ca-configmap" pod "439163d6-6cb9-4b15-bcae-2f6f6cdecde2" (UID: "439163d6-6cb9-4b15-bcae-2f6f6cdecde2"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:29:47.418477 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:47.418449 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/439163d6-6cb9-4b15-bcae-2f6f6cdecde2-istio-csr-dns-cert" pod "439163d6-6cb9-4b15-bcae-2f6f6cdecde2" (UID: "439163d6-6cb9-4b15-bcae-2f6f6cdecde2"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:29:47.418588 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:47.418567 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/439163d6-6cb9-4b15-bcae-2f6f6cdecde2-kube-api-access-hknxx" pod "439163d6-6cb9-4b15-bcae-2f6f6cdecde2" (UID: "439163d6-6cb9-4b15-bcae-2f6f6cdecde2"). InnerVolumeSpecName "kube-api-access-hknxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:29:47.418744 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:47.418723 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/439163d6-6cb9-4b15-bcae-2f6f6cdecde2-cacerts" pod "439163d6-6cb9-4b15-bcae-2f6f6cdecde2" (UID: "439163d6-6cb9-4b15-bcae-2f6f6cdecde2"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:29:47.419095 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:47.419071 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/439163d6-6cb9-4b15-bcae-2f6f6cdecde2-istio-kubeconfig" pod "439163d6-6cb9-4b15-bcae-2f6f6cdecde2" (UID: "439163d6-6cb9-4b15-bcae-2f6f6cdecde2"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:29:47.419307 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:47.419288 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/439163d6-6cb9-4b15-bcae-2f6f6cdecde2-local-certs" pod "439163d6-6cb9-4b15-bcae-2f6f6cdecde2" (UID: "439163d6-6cb9-4b15-bcae-2f6f6cdecde2"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:29:47.419378 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:47.419327 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/439163d6-6cb9-4b15-bcae-2f6f6cdecde2-istio-token" pod "439163d6-6cb9-4b15-bcae-2f6f6cdecde2" (UID: "439163d6-6cb9-4b15-bcae-2f6f6cdecde2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:29:47.516738 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:47.516672 4493 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/439163d6-6cb9-4b15-bcae-2f6f6cdecde2-istio-csr-ca-configmap\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:29:47.516738 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:47.516698 4493 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/439163d6-6cb9-4b15-bcae-2f6f6cdecde2-cacerts\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:29:47.516738 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:47.516708 4493 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/439163d6-6cb9-4b15-bcae-2f6f6cdecde2-istio-csr-dns-cert\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:29:47.516738 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:47.516717 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hknxx\" (UniqueName: \"kubernetes.io/projected/439163d6-6cb9-4b15-bcae-2f6f6cdecde2-kube-api-access-hknxx\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:29:47.516738 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:47.516725 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/439163d6-6cb9-4b15-bcae-2f6f6cdecde2-istio-token\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:29:47.516738 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:47.516733 4493 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/439163d6-6cb9-4b15-bcae-2f6f6cdecde2-local-certs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:29:47.516738 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:47.516740 4493 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/439163d6-6cb9-4b15-bcae-2f6f6cdecde2-istio-kubeconfig\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:29:47.569254 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:47.569221 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-h6wgm_3319a971-2f82-469a-8fc3-20687e710719/manager/0.log" Oct 30 17:29:47.678892 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:47.678833 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5kzmj"] Oct 30 17:29:47.689371 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:47.689341 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-5kzmj"] Oct 30 17:29:48.467321 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:48.467283 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-6kvn2_58874bb8-6c51-4f03-ab7d-ebf99148b4d8/manager/0.log" Oct 30 17:29:48.890216 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:48.890138 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-7blqd_2003d468-00b8-48c3-9b13-f152737d18ed/manager/0.log" Oct 30 17:29:48.931755 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:48.931730 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="439163d6-6cb9-4b15-bcae-2f6f6cdecde2" path="/var/lib/kubelet/pods/439163d6-6cb9-4b15-bcae-2f6f6cdecde2/volumes" Oct 30 17:29:49.656260 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:49.656240 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4" Oct 30 17:29:49.825244 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:49.825162 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-jcntx_12f71e3d-d209-44f3-b5a8-9eab1f962214/manager/0.log" Oct 30 17:29:49.836254 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:49.836227 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da5ec9e6-8379-46ee-a2d9-2360237ccf2d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da5ec9e6-8379-46ee-a2d9-2360237ccf2d-istio-envoy\") pod \"da5ec9e6-8379-46ee-a2d9-2360237ccf2d\" (UID: \"da5ec9e6-8379-46ee-a2d9-2360237ccf2d\") " Oct 30 17:29:49.836430 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:49.836269 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da5ec9e6-8379-46ee-a2d9-2360237ccf2d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da5ec9e6-8379-46ee-a2d9-2360237ccf2d-istiod-ca-cert\") pod \"da5ec9e6-8379-46ee-a2d9-2360237ccf2d\" (UID: \"da5ec9e6-8379-46ee-a2d9-2360237ccf2d\") " Oct 30 17:29:49.836430 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:49.836299 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da5ec9e6-8379-46ee-a2d9-2360237ccf2d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da5ec9e6-8379-46ee-a2d9-2360237ccf2d-istio-data\") pod \"da5ec9e6-8379-46ee-a2d9-2360237ccf2d\" (UID: \"da5ec9e6-8379-46ee-a2d9-2360237ccf2d\") " Oct 30 17:29:49.836430 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:49.836336 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da5ec9e6-8379-46ee-a2d9-2360237ccf2d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da5ec9e6-8379-46ee-a2d9-2360237ccf2d-workload-socket\") pod \"da5ec9e6-8379-46ee-a2d9-2360237ccf2d\" (UID: \"da5ec9e6-8379-46ee-a2d9-2360237ccf2d\") " Oct 30 17:29:49.836430 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:49.836367 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da5ec9e6-8379-46ee-a2d9-2360237ccf2d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da5ec9e6-8379-46ee-a2d9-2360237ccf2d-credential-socket\") pod \"da5ec9e6-8379-46ee-a2d9-2360237ccf2d\" (UID: \"da5ec9e6-8379-46ee-a2d9-2360237ccf2d\") " Oct 30 17:29:49.836430 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:49.836388 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da5ec9e6-8379-46ee-a2d9-2360237ccf2d-istio-token\" (UniqueName: \"kubernetes.io/projected/da5ec9e6-8379-46ee-a2d9-2360237ccf2d-istio-token\") pod \"da5ec9e6-8379-46ee-a2d9-2360237ccf2d\" (UID: \"da5ec9e6-8379-46ee-a2d9-2360237ccf2d\") " Oct 30 17:29:49.836706 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:49.836446 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/da5ec9e6-8379-46ee-a2d9-2360237ccf2d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da5ec9e6-8379-46ee-a2d9-2360237ccf2d-istio-podinfo\") pod \"da5ec9e6-8379-46ee-a2d9-2360237ccf2d\" (UID: \"da5ec9e6-8379-46ee-a2d9-2360237ccf2d\") " Oct 30 17:29:49.836706 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:49.836482 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da5ec9e6-8379-46ee-a2d9-2360237ccf2d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da5ec9e6-8379-46ee-a2d9-2360237ccf2d-workload-certs\") pod \"da5ec9e6-8379-46ee-a2d9-2360237ccf2d\" (UID: \"da5ec9e6-8379-46ee-a2d9-2360237ccf2d\") " Oct 30 17:29:49.836706 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:49.836550 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da5ec9e6-8379-46ee-a2d9-2360237ccf2d-kube-api-access-snk48\" (UniqueName: \"kubernetes.io/projected/da5ec9e6-8379-46ee-a2d9-2360237ccf2d-kube-api-access-snk48\") pod \"da5ec9e6-8379-46ee-a2d9-2360237ccf2d\" (UID: \"da5ec9e6-8379-46ee-a2d9-2360237ccf2d\") " Oct 30 17:29:49.836706 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:49.836681 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da5ec9e6-8379-46ee-a2d9-2360237ccf2d-istio-data" pod "da5ec9e6-8379-46ee-a2d9-2360237ccf2d" (UID: "da5ec9e6-8379-46ee-a2d9-2360237ccf2d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:29:49.836979 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:49.836745 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da5ec9e6-8379-46ee-a2d9-2360237ccf2d-workload-socket" pod "da5ec9e6-8379-46ee-a2d9-2360237ccf2d" (UID: "da5ec9e6-8379-46ee-a2d9-2360237ccf2d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:29:49.836979 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:49.836773 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da5ec9e6-8379-46ee-a2d9-2360237ccf2d-credential-socket" pod "da5ec9e6-8379-46ee-a2d9-2360237ccf2d" (UID: "da5ec9e6-8379-46ee-a2d9-2360237ccf2d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:29:49.836979 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:49.836834 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da5ec9e6-8379-46ee-a2d9-2360237ccf2d-istiod-ca-cert" pod "da5ec9e6-8379-46ee-a2d9-2360237ccf2d" (UID: "da5ec9e6-8379-46ee-a2d9-2360237ccf2d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:29:49.837113 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:49.836983 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da5ec9e6-8379-46ee-a2d9-2360237ccf2d-workload-certs" pod "da5ec9e6-8379-46ee-a2d9-2360237ccf2d" (UID: "da5ec9e6-8379-46ee-a2d9-2360237ccf2d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:29:49.837113 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:49.837004 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da5ec9e6-8379-46ee-a2d9-2360237ccf2d-istiod-ca-cert\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:29:49.837113 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:49.837031 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da5ec9e6-8379-46ee-a2d9-2360237ccf2d-istio-data\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:29:49.837113 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:49.837048 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da5ec9e6-8379-46ee-a2d9-2360237ccf2d-workload-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:29:49.837113 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:49.837061 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da5ec9e6-8379-46ee-a2d9-2360237ccf2d-credential-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:29:49.838864 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:49.838846 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da5ec9e6-8379-46ee-a2d9-2360237ccf2d-kube-api-access-snk48" pod "da5ec9e6-8379-46ee-a2d9-2360237ccf2d" (UID: "da5ec9e6-8379-46ee-a2d9-2360237ccf2d"). InnerVolumeSpecName "kube-api-access-snk48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:29:49.839009 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:49.838986 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da5ec9e6-8379-46ee-a2d9-2360237ccf2d-istio-podinfo" pod "da5ec9e6-8379-46ee-a2d9-2360237ccf2d" (UID: "da5ec9e6-8379-46ee-a2d9-2360237ccf2d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:29:49.839315 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:49.839296 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da5ec9e6-8379-46ee-a2d9-2360237ccf2d-istio-envoy" pod "da5ec9e6-8379-46ee-a2d9-2360237ccf2d" (UID: "da5ec9e6-8379-46ee-a2d9-2360237ccf2d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:29:49.839362 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:49.839313 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da5ec9e6-8379-46ee-a2d9-2360237ccf2d-istio-token" pod "da5ec9e6-8379-46ee-a2d9-2360237ccf2d" (UID: "da5ec9e6-8379-46ee-a2d9-2360237ccf2d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:29:49.938068 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:49.938039 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da5ec9e6-8379-46ee-a2d9-2360237ccf2d-istio-token\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:29:49.938068 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:49.938064 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da5ec9e6-8379-46ee-a2d9-2360237ccf2d-istio-podinfo\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:29:49.938456 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:49.938073 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da5ec9e6-8379-46ee-a2d9-2360237ccf2d-workload-certs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:29:49.938456 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:49.938090 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snk48\" (UniqueName: \"kubernetes.io/projected/da5ec9e6-8379-46ee-a2d9-2360237ccf2d-kube-api-access-snk48\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:29:49.938456 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:49.938101 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da5ec9e6-8379-46ee-a2d9-2360237ccf2d-istio-envoy\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:29:50.357432 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:50.357393 4493 generic.go:356] "Generic (PLEG): container finished" podID="da5ec9e6-8379-46ee-a2d9-2360237ccf2d" containerID="d42783125009f74712772bb9386d5740e54727c4ede5e78a24b84706230eecea" exitCode=0 Oct 30 17:29:50.357613 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:50.357436 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4" Oct 30 17:29:50.357613 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:50.357482 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4" event={"ID":"da5ec9e6-8379-46ee-a2d9-2360237ccf2d","Type":"ContainerDied","Data":"d42783125009f74712772bb9386d5740e54727c4ede5e78a24b84706230eecea"} Oct 30 17:29:50.357613 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:50.357534 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4" event={"ID":"da5ec9e6-8379-46ee-a2d9-2360237ccf2d","Type":"ContainerDied","Data":"3189fba18e047d98e5d906d05678a4f1de1520a183a6b5ff7bb25e95c54fa5f8"} Oct 30 17:29:50.357613 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:50.357555 4493 scope.go:122] "RemoveContainer" containerID="d42783125009f74712772bb9386d5740e54727c4ede5e78a24b84706230eecea" Oct 30 17:29:50.368715 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:50.368696 4493 scope.go:122] "RemoveContainer" containerID="d42783125009f74712772bb9386d5740e54727c4ede5e78a24b84706230eecea" Oct 30 17:29:50.368998 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:29:50.368978 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d42783125009f74712772bb9386d5740e54727c4ede5e78a24b84706230eecea\": container with ID starting with d42783125009f74712772bb9386d5740e54727c4ede5e78a24b84706230eecea not found: ID does not exist" containerID="d42783125009f74712772bb9386d5740e54727c4ede5e78a24b84706230eecea" Oct 30 17:29:50.369067 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:50.369006 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d42783125009f74712772bb9386d5740e54727c4ede5e78a24b84706230eecea"} err="failed to get container status \"d42783125009f74712772bb9386d5740e54727c4ede5e78a24b84706230eecea\": rpc error: code = NotFound desc = could not find container \"d42783125009f74712772bb9386d5740e54727c4ede5e78a24b84706230eecea\": container with ID starting with d42783125009f74712772bb9386d5740e54727c4ede5e78a24b84706230eecea not found: ID does not exist" Oct 30 17:29:50.420852 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:50.420828 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4"] Oct 30 17:29:50.432296 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:50.432261 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nwtx4"] Oct 30 17:29:50.837210 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:50.837183 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-78bcf8868f-w5c9k_ad767d7d-7fb3-480d-bf9b-6583a00936fb/authorino/0.log" Oct 30 17:29:50.933436 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:50.933402 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da5ec9e6-8379-46ee-a2d9-2360237ccf2d" path="/var/lib/kubelet/pods/da5ec9e6-8379-46ee-a2d9-2360237ccf2d/volumes" Oct 30 17:29:58.730682 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:58.730645 4493 scope.go:122] "RemoveContainer" containerID="1ba3f30f3509e86b2bfb0fade6a8a317e3831547786d509c0c6eb9d599c12a23" Oct 30 17:29:58.740689 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:58.740668 4493 scope.go:122] "RemoveContainer" containerID="ed2bcc5c09009b68efad7aeb042081349233d760e8a8ddb9fbafd614807fbc77" Oct 30 17:29:58.749602 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:58.749582 4493 scope.go:122] "RemoveContainer" containerID="8f7a1a75b24ef405aa92fdb7c4bdff4fcff29b9e9fa73be7048933f99a6ccd3e" Oct 30 17:29:58.758349 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:58.758330 4493 scope.go:122] "RemoveContainer" containerID="77e1e14a1ea94032a5956f942e525ccf03c86f1b977595f5a38ad84c7c9d035c" Oct 30 17:29:58.766825 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:58.766811 4493 scope.go:122] "RemoveContainer" containerID="ce5ad454abc286bcc0fa9d88e4739e8790bee04f3d45897d3faba7a41740a377" Oct 30 17:29:58.779037 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:58.779019 4493 scope.go:122] "RemoveContainer" containerID="451967d264159ac7d1a60996b8f2d4a97013a806c3115c7e76e180ca0cfb57df" Oct 30 17:29:58.788433 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:58.788411 4493 scope.go:122] "RemoveContainer" containerID="9dcff2368ec2b4ef5da5c5ce7a6d4deab48379b07e8a270d8dbb8da5e3cb4fd8" Oct 30 17:29:58.796695 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:29:58.796672 4493 scope.go:122] "RemoveContainer" containerID="ff2288bb24c1f2814b517da918364af404918eaaba93cc6aff96d912c64e8940" Oct 30 17:30:00.344181 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:30:00.344144 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 17:30:01.643067 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:01.643038 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-6kvn2_58874bb8-6c51-4f03-ab7d-ebf99148b4d8/manager/0.log" Oct 30 17:30:02.034824 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:02.034709 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95f0efc4-qnx55"] Oct 30 17:30:02.046258 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:02.046220 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1beb74a-dcl87"] Oct 30 17:30:02.048462 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:02.048438 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1beb74a-dcl87"] Oct 30 17:30:02.050335 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:02.050310 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95f0efc4-qnx55"] Oct 30 17:30:02.063175 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:02.063144 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-7blqd_2003d468-00b8-48c3-9b13-f152737d18ed/manager/0.log" Oct 30 17:30:02.931250 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:02.931216 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99364c73-370a-443d-a082-454bc4fda069" path="/var/lib/kubelet/pods/99364c73-370a-443d-a082-454bc4fda069/volumes" Oct 30 17:30:02.931639 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:02.931625 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee384285-d605-493f-bbb1-eec6e15d1549" path="/var/lib/kubelet/pods/ee384285-d605-493f-bbb1-eec6e15d1549/volumes" Oct 30 17:30:02.979310 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:02.979273 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-jcntx_12f71e3d-d209-44f3-b5a8-9eab1f962214/manager/0.log" Oct 30 17:30:04.347880 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:04.347848 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-16140560-b8jfg_36ad9481-025e-488d-aa35-8fc61f4c3034/revoke-keys/0.log" Oct 30 17:30:04.533212 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:04.533180 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-21f27ac6-w8frp_136eca69-91b5-4b17-9d60-3cd393964f39/revoke-keys/0.log" Oct 30 17:30:04.719085 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:04.719055 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3fdce1e5-2rgds_7e4debf1-371b-4670-ac5e-a60249825753/revoke-keys/0.log" Oct 30 17:30:04.912128 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:04.912100 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-5b548074-mmcx6_9cc934bb-7ead-421d-88c1-13e1bc5e93d5/revoke-keys/0.log" Oct 30 17:30:05.116410 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:05.116326 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-8599a73b-bjkpz_11356029-07cb-454d-afbb-edab8a6d250c/revoke-keys/0.log" Oct 30 17:30:05.301879 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:05.301850 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-ba183451-tgmlj_58c27031-67a6-4647-9be5-6f8a0e2adb33/revoke-keys/0.log" Oct 30 17:30:05.484098 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:05.484069 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-bea7af65-j69p2_04895c70-8d15-4a8a-81fa-e40f18a2563c/revoke-keys/0.log" Oct 30 17:30:05.675487 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:05.675458 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d93a3edf-8s8ws_50e6c831-46ad-4a5f-adea-4271dcfd6eb2/revoke-keys/0.log" Oct 30 17:30:05.992987 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:05.992958 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-xjbtt"] Oct 30 17:30:05.993223 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:05.993199 4493 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-xjbtt" podUID="b9860f52-b69d-4de3-8999-eff8e90620dc" containerName="istio-proxy" containerID="cri-o://301d8812d0adec42dad331d678a4dec9af00b159c618daeff1f7973d18d24914" gracePeriod=30 Oct 30 17:30:06.065661 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:06.065634 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f188440c-qgjw7_6157f678-6cf7-4cc4-a4db-20c32afeb714/revoke-keys/0.log" Oct 30 17:30:07.923440 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:07.923405 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-78bcf8868f-w5c9k_ad767d7d-7fb3-480d-bf9b-6583a00936fb/authorino/0.log" Oct 30 17:30:08.169455 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:08.169416 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-gzvkk_39b545e1-1a61-4ddf-a00a-f396058e6a88/manager/0.log" Oct 30 17:30:08.375172 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:08.375092 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-8xm6t_cf7e09ae-b9ba-420d-b071-bc2f5ac8b009/manager/0.log" Oct 30 17:30:08.889529 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:08.889498 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-kksz8_6594d922-089d-41e9-9444-5a888d5fa675/manager/0.log" Oct 30 17:30:09.091346 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:09.091319 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-k9d99_19632c83-f396-437f-ae2d-93229596ce61/limitador/0.log" Oct 30 17:30:09.288398 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:09.288366 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-h6wgm_3319a971-2f82-469a-8fc3-20687e710719/manager/1.log" Oct 30 17:30:09.734736 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:09.734706 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-h6wgm_3319a971-2f82-469a-8fc3-20687e710719/manager/0.log" Oct 30 17:30:10.275523 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:10.275488 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7d45d5c789j7m84_1c9bc2df-9829-4346-a082-b8255c37bb3c/istio-proxy/0.log" Oct 30 17:30:11.250323 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:11.250300 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-xjbtt" Oct 30 17:30:11.304829 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:11.304740 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9860f52-b69d-4de3-8999-eff8e90620dc-kube-api-access-pg2nq\" (UniqueName: \"kubernetes.io/projected/b9860f52-b69d-4de3-8999-eff8e90620dc-kube-api-access-pg2nq\") pod \"b9860f52-b69d-4de3-8999-eff8e90620dc\" (UID: \"b9860f52-b69d-4de3-8999-eff8e90620dc\") " Oct 30 17:30:11.305223 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:11.304831 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9860f52-b69d-4de3-8999-eff8e90620dc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9860f52-b69d-4de3-8999-eff8e90620dc-workload-socket\") pod \"b9860f52-b69d-4de3-8999-eff8e90620dc\" (UID: \"b9860f52-b69d-4de3-8999-eff8e90620dc\") " Oct 30 17:30:11.305223 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:11.304862 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9860f52-b69d-4de3-8999-eff8e90620dc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9860f52-b69d-4de3-8999-eff8e90620dc-istio-data\") pod \"b9860f52-b69d-4de3-8999-eff8e90620dc\" (UID: \"b9860f52-b69d-4de3-8999-eff8e90620dc\") " Oct 30 17:30:11.305223 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:11.304900 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9860f52-b69d-4de3-8999-eff8e90620dc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9860f52-b69d-4de3-8999-eff8e90620dc-credential-socket\") pod \"b9860f52-b69d-4de3-8999-eff8e90620dc\" (UID: \"b9860f52-b69d-4de3-8999-eff8e90620dc\") " Oct 30 17:30:11.305223 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:11.304947 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9860f52-b69d-4de3-8999-eff8e90620dc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9860f52-b69d-4de3-8999-eff8e90620dc-workload-certs\") pod \"b9860f52-b69d-4de3-8999-eff8e90620dc\" (UID: \"b9860f52-b69d-4de3-8999-eff8e90620dc\") " Oct 30 17:30:11.305223 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:11.305010 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9860f52-b69d-4de3-8999-eff8e90620dc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9860f52-b69d-4de3-8999-eff8e90620dc-istio-envoy\") pod \"b9860f52-b69d-4de3-8999-eff8e90620dc\" (UID: \"b9860f52-b69d-4de3-8999-eff8e90620dc\") " Oct 30 17:30:11.305223 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:11.305069 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b9860f52-b69d-4de3-8999-eff8e90620dc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9860f52-b69d-4de3-8999-eff8e90620dc-istio-podinfo\") pod \"b9860f52-b69d-4de3-8999-eff8e90620dc\" (UID: \"b9860f52-b69d-4de3-8999-eff8e90620dc\") " Oct 30 17:30:11.305223 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:11.305102 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9860f52-b69d-4de3-8999-eff8e90620dc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9860f52-b69d-4de3-8999-eff8e90620dc-istiod-ca-cert\") pod \"b9860f52-b69d-4de3-8999-eff8e90620dc\" (UID: \"b9860f52-b69d-4de3-8999-eff8e90620dc\") " Oct 30 17:30:11.305223 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:11.305147 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9860f52-b69d-4de3-8999-eff8e90620dc-workload-socket" pod "b9860f52-b69d-4de3-8999-eff8e90620dc" (UID: "b9860f52-b69d-4de3-8999-eff8e90620dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:30:11.305223 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:11.305154 4493 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9860f52-b69d-4de3-8999-eff8e90620dc-istio-token\" (UniqueName: \"kubernetes.io/projected/b9860f52-b69d-4de3-8999-eff8e90620dc-istio-token\") pod \"b9860f52-b69d-4de3-8999-eff8e90620dc\" (UID: \"b9860f52-b69d-4de3-8999-eff8e90620dc\") " Oct 30 17:30:11.305677 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:11.305166 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9860f52-b69d-4de3-8999-eff8e90620dc-istio-data" pod "b9860f52-b69d-4de3-8999-eff8e90620dc" (UID: "b9860f52-b69d-4de3-8999-eff8e90620dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:30:11.305677 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:11.305243 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9860f52-b69d-4de3-8999-eff8e90620dc-credential-socket" pod "b9860f52-b69d-4de3-8999-eff8e90620dc" (UID: "b9860f52-b69d-4de3-8999-eff8e90620dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:30:11.305677 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:11.305265 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9860f52-b69d-4de3-8999-eff8e90620dc-workload-certs" pod "b9860f52-b69d-4de3-8999-eff8e90620dc" (UID: "b9860f52-b69d-4de3-8999-eff8e90620dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:30:11.305677 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:11.305478 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9860f52-b69d-4de3-8999-eff8e90620dc-istiod-ca-cert" pod "b9860f52-b69d-4de3-8999-eff8e90620dc" (UID: "b9860f52-b69d-4de3-8999-eff8e90620dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:30:11.305919 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:11.305728 4493 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9860f52-b69d-4de3-8999-eff8e90620dc-workload-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:30:11.305919 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:11.305809 4493 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9860f52-b69d-4de3-8999-eff8e90620dc-istio-data\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:30:11.306003 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:11.305914 4493 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9860f52-b69d-4de3-8999-eff8e90620dc-credential-socket\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:30:11.306003 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:11.305963 4493 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9860f52-b69d-4de3-8999-eff8e90620dc-workload-certs\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:30:11.306003 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:11.305977 4493 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9860f52-b69d-4de3-8999-eff8e90620dc-istiod-ca-cert\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:30:11.307851 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:11.307826 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9860f52-b69d-4de3-8999-eff8e90620dc-istio-token" pod "b9860f52-b69d-4de3-8999-eff8e90620dc" (UID: "b9860f52-b69d-4de3-8999-eff8e90620dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:30:11.308443 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:11.308413 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b9860f52-b69d-4de3-8999-eff8e90620dc-istio-podinfo" pod "b9860f52-b69d-4de3-8999-eff8e90620dc" (UID: "b9860f52-b69d-4de3-8999-eff8e90620dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:30:11.308772 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:11.308730 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9860f52-b69d-4de3-8999-eff8e90620dc-istio-envoy" pod "b9860f52-b69d-4de3-8999-eff8e90620dc" (UID: "b9860f52-b69d-4de3-8999-eff8e90620dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:30:11.308772 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:11.308740 4493 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9860f52-b69d-4de3-8999-eff8e90620dc-kube-api-access-pg2nq" pod "b9860f52-b69d-4de3-8999-eff8e90620dc" (UID: "b9860f52-b69d-4de3-8999-eff8e90620dc"). InnerVolumeSpecName "kube-api-access-pg2nq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:30:11.407397 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:11.407363 4493 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pg2nq\" (UniqueName: \"kubernetes.io/projected/b9860f52-b69d-4de3-8999-eff8e90620dc-kube-api-access-pg2nq\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:30:11.407397 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:11.407391 4493 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9860f52-b69d-4de3-8999-eff8e90620dc-istio-envoy\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:30:11.407397 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:11.407401 4493 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9860f52-b69d-4de3-8999-eff8e90620dc-istio-podinfo\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:30:11.407624 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:11.407409 4493 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9860f52-b69d-4de3-8999-eff8e90620dc-istio-token\") on node \"ip-10-0-132-180.ec2.internal\" DevicePath \"\"" Oct 30 17:30:11.465997 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:11.465970 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7854949c9-xjbtt_b9860f52-b69d-4de3-8999-eff8e90620dc/istio-proxy/0.log" Oct 30 17:30:11.508069 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:11.508038 4493 generic.go:356] "Generic (PLEG): container finished" podID="b9860f52-b69d-4de3-8999-eff8e90620dc" containerID="301d8812d0adec42dad331d678a4dec9af00b159c618daeff1f7973d18d24914" exitCode=0 Oct 30 17:30:11.508213 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:11.508085 4493 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-xjbtt" Oct 30 17:30:11.508213 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:11.508130 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-xjbtt" event={"ID":"b9860f52-b69d-4de3-8999-eff8e90620dc","Type":"ContainerDied","Data":"301d8812d0adec42dad331d678a4dec9af00b159c618daeff1f7973d18d24914"} Oct 30 17:30:11.508213 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:11.508187 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-xjbtt" event={"ID":"b9860f52-b69d-4de3-8999-eff8e90620dc","Type":"ContainerDied","Data":"5159e35db94133641e5f8db4482f36ceea6c6966a1d175784d4bbc9513b20833"} Oct 30 17:30:11.508350 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:11.508217 4493 scope.go:122] "RemoveContainer" containerID="301d8812d0adec42dad331d678a4dec9af00b159c618daeff1f7973d18d24914" Oct 30 17:30:11.519103 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:11.519089 4493 scope.go:122] "RemoveContainer" containerID="301d8812d0adec42dad331d678a4dec9af00b159c618daeff1f7973d18d24914" Oct 30 17:30:11.519310 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:30:11.519292 4493 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"301d8812d0adec42dad331d678a4dec9af00b159c618daeff1f7973d18d24914\": container with ID starting with 301d8812d0adec42dad331d678a4dec9af00b159c618daeff1f7973d18d24914 not found: ID does not exist" containerID="301d8812d0adec42dad331d678a4dec9af00b159c618daeff1f7973d18d24914" Oct 30 17:30:11.519361 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:11.519318 4493 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"301d8812d0adec42dad331d678a4dec9af00b159c618daeff1f7973d18d24914"} err="failed to get container status \"301d8812d0adec42dad331d678a4dec9af00b159c618daeff1f7973d18d24914\": rpc error: code = NotFound desc = could not find container \"301d8812d0adec42dad331d678a4dec9af00b159c618daeff1f7973d18d24914\": container with ID starting with 301d8812d0adec42dad331d678a4dec9af00b159c618daeff1f7973d18d24914 not found: ID does not exist" Oct 30 17:30:11.560491 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:11.560442 4493 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-xjbtt"] Oct 30 17:30:11.564515 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:11.564495 4493 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-xjbtt"] Oct 30 17:30:11.664687 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:11.664662 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-6f8685f4cf-fx429_5d9779f9-732c-4971-b52b-d26ce92abf2f/istio-proxy/0.log" Oct 30 17:30:12.938623 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:12.938586 4493 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9860f52-b69d-4de3-8999-eff8e90620dc" path="/var/lib/kubelet/pods/b9860f52-b69d-4de3-8999-eff8e90620dc/volumes" Oct 30 17:30:24.604388 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:24.604356 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cjwc7/must-gather-jzn84"] Oct 30 17:30:24.604798 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:24.604726 4493 cpu_manager.go:414] "containerMap: removing container" podUID="352047e0-584f-4a19-b56a-2a05bd960f73" containerName="istio-proxy" Oct 30 17:30:24.604798 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:24.604737 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="352047e0-584f-4a19-b56a-2a05bd960f73" containerName="istio-proxy" Oct 30 17:30:24.604798 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:24.604751 4493 cpu_manager.go:414] "containerMap: removing container" podUID="7a5145a9-7bd1-4f48-b116-6f08637fe08a" containerName="payload-pre-processing" Oct 30 17:30:24.604798 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:24.604757 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a5145a9-7bd1-4f48-b116-6f08637fe08a" containerName="payload-pre-processing" Oct 30 17:30:24.604798 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:24.604765 4493 cpu_manager.go:414] "containerMap: removing container" podUID="e76172b1-659d-45bd-91c6-1a0ce4062335" containerName="main" Oct 30 17:30:24.604798 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:24.604769 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e76172b1-659d-45bd-91c6-1a0ce4062335" containerName="main" Oct 30 17:30:24.604798 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:24.604798 4493 cpu_manager.go:414] "containerMap: removing container" podUID="0eadc5d4-98c3-4bf2-b3c2-f73fe249182f" containerName="main" Oct 30 17:30:24.604798 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:24.604803 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0eadc5d4-98c3-4bf2-b3c2-f73fe249182f" containerName="main" Oct 30 17:30:24.605072 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:24.604811 4493 cpu_manager.go:414] "containerMap: removing container" podUID="da5ec9e6-8379-46ee-a2d9-2360237ccf2d" containerName="istio-proxy" Oct 30 17:30:24.605072 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:24.604816 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da5ec9e6-8379-46ee-a2d9-2360237ccf2d" containerName="istio-proxy" Oct 30 17:30:24.605072 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:24.604823 4493 cpu_manager.go:414] "containerMap: removing container" podUID="e90f43d3-f5eb-45ab-af1e-340d0ce18966" containerName="main" Oct 30 17:30:24.605072 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:24.604827 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e90f43d3-f5eb-45ab-af1e-340d0ce18966" containerName="main" Oct 30 17:30:24.605072 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:24.604833 4493 cpu_manager.go:414] "containerMap: removing container" podUID="b9860f52-b69d-4de3-8999-eff8e90620dc" containerName="istio-proxy" Oct 30 17:30:24.605072 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:24.604838 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9860f52-b69d-4de3-8999-eff8e90620dc" containerName="istio-proxy" Oct 30 17:30:24.605072 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:24.604850 4493 cpu_manager.go:414] "containerMap: removing container" podUID="11356029-07cb-454d-afbb-edab8a6d250c" containerName="revoke-keys" Oct 30 17:30:24.605072 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:24.604855 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11356029-07cb-454d-afbb-edab8a6d250c" containerName="revoke-keys" Oct 30 17:30:24.605072 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:24.604866 4493 cpu_manager.go:414] "containerMap: removing container" podUID="a6aa4382-931c-456e-a739-4580c07adf09" containerName="payload-processing" Oct 30 17:30:24.605072 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:24.604871 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6aa4382-931c-456e-a739-4580c07adf09" containerName="payload-processing" Oct 30 17:30:24.605072 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:24.604877 4493 cpu_manager.go:414] "containerMap: removing container" podUID="439163d6-6cb9-4b15-bcae-2f6f6cdecde2" containerName="discovery" Oct 30 17:30:24.605072 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:24.604881 4493 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="439163d6-6cb9-4b15-bcae-2f6f6cdecde2" containerName="discovery" Oct 30 17:30:24.605072 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:24.604941 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a5145a9-7bd1-4f48-b116-6f08637fe08a" containerName="payload-pre-processing" Oct 30 17:30:24.605072 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:24.604947 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="e76172b1-659d-45bd-91c6-1a0ce4062335" containerName="main" Oct 30 17:30:24.605072 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:24.604954 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="352047e0-584f-4a19-b56a-2a05bd960f73" containerName="istio-proxy" Oct 30 17:30:24.605072 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:24.604961 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="0eadc5d4-98c3-4bf2-b3c2-f73fe249182f" containerName="main" Oct 30 17:30:24.605072 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:24.604967 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="da5ec9e6-8379-46ee-a2d9-2360237ccf2d" containerName="istio-proxy" Oct 30 17:30:24.605072 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:24.604973 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="e90f43d3-f5eb-45ab-af1e-340d0ce18966" containerName="main" Oct 30 17:30:24.605072 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:24.604978 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9860f52-b69d-4de3-8999-eff8e90620dc" containerName="istio-proxy" Oct 30 17:30:24.605072 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:24.604986 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="11356029-07cb-454d-afbb-edab8a6d250c" containerName="revoke-keys" Oct 30 17:30:24.605072 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:24.604993 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6aa4382-931c-456e-a739-4580c07adf09" containerName="payload-processing" Oct 30 17:30:24.605072 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:24.604997 4493 memory_manager.go:365] "RemoveStaleState removing state" podUID="439163d6-6cb9-4b15-bcae-2f6f6cdecde2" containerName="discovery" Oct 30 17:30:24.609326 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:24.609301 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cjwc7/must-gather-jzn84" Oct 30 17:30:24.627921 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:24.627900 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cjwc7\"/\"kube-root-ca.crt\"" Oct 30 17:30:24.639292 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:24.639257 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cjwc7\"/\"openshift-service-ca.crt\"" Oct 30 17:30:24.639292 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:24.639278 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-cjwc7\"/\"default-dockercfg-974ld\"" Oct 30 17:30:24.641637 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:24.641617 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cjwc7/must-gather-jzn84"] Oct 30 17:30:24.719099 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:24.719063 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lxdq\" (UniqueName: \"kubernetes.io/projected/8572b2dc-2297-4948-a18a-126345cc9b03-kube-api-access-5lxdq\") pod \"must-gather-jzn84\" (UID: \"8572b2dc-2297-4948-a18a-126345cc9b03\") " pod="openshift-must-gather-cjwc7/must-gather-jzn84" Oct 30 17:30:24.719296 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:24.719138 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8572b2dc-2297-4948-a18a-126345cc9b03-must-gather-output\") pod \"must-gather-jzn84\" (UID: \"8572b2dc-2297-4948-a18a-126345cc9b03\") " pod="openshift-must-gather-cjwc7/must-gather-jzn84" Oct 30 17:30:24.820166 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:24.820128 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8572b2dc-2297-4948-a18a-126345cc9b03-must-gather-output\") pod \"must-gather-jzn84\" (UID: \"8572b2dc-2297-4948-a18a-126345cc9b03\") " pod="openshift-must-gather-cjwc7/must-gather-jzn84" Oct 30 17:30:24.820343 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:24.820213 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lxdq\" (UniqueName: \"kubernetes.io/projected/8572b2dc-2297-4948-a18a-126345cc9b03-kube-api-access-5lxdq\") pod \"must-gather-jzn84\" (UID: \"8572b2dc-2297-4948-a18a-126345cc9b03\") " pod="openshift-must-gather-cjwc7/must-gather-jzn84" Oct 30 17:30:24.820499 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:24.820481 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8572b2dc-2297-4948-a18a-126345cc9b03-must-gather-output\") pod \"must-gather-jzn84\" (UID: \"8572b2dc-2297-4948-a18a-126345cc9b03\") " pod="openshift-must-gather-cjwc7/must-gather-jzn84" Oct 30 17:30:24.843078 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:24.843052 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lxdq\" (UniqueName: \"kubernetes.io/projected/8572b2dc-2297-4948-a18a-126345cc9b03-kube-api-access-5lxdq\") pod \"must-gather-jzn84\" (UID: \"8572b2dc-2297-4948-a18a-126345cc9b03\") " pod="openshift-must-gather-cjwc7/must-gather-jzn84" Oct 30 17:30:24.923257 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:24.923228 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cjwc7/must-gather-jzn84" Oct 30 17:30:25.237026 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:25.236993 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cjwc7/must-gather-jzn84"] Oct 30 17:30:25.238999 ip-10-0-132-180 kubenswrapper[4493]: W1030 17:30:25.238969 4493 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8572b2dc_2297_4948_a18a_126345cc9b03.slice/crio-9dce9f3f6a0993fcd858e1c8aad7593277b5817ea56e265ac2ff33a50aa5a736 WatchSource:0}: Error finding container 9dce9f3f6a0993fcd858e1c8aad7593277b5817ea56e265ac2ff33a50aa5a736: Status 404 returned error can't find the container with id 9dce9f3f6a0993fcd858e1c8aad7593277b5817ea56e265ac2ff33a50aa5a736 Oct 30 17:30:25.241766 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:25.241747 4493 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 17:30:25.597275 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:25.597179 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cjwc7/must-gather-jzn84" event={"ID":"8572b2dc-2297-4948-a18a-126345cc9b03","Type":"ContainerStarted","Data":"9dce9f3f6a0993fcd858e1c8aad7593277b5817ea56e265ac2ff33a50aa5a736"} Oct 30 17:30:29.090563 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:29.090527 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-2lrlr_dc77c50f-abe0-4d9c-a88b-7a186c17dc3e/global-pull-secret-syncer/0.log" Oct 30 17:30:30.341003 ip-10-0-132-180 kubenswrapper[4493]: E1030 17:30:30.340956 4493 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-c24kh" containerName="extractor" Oct 30 17:30:30.434546 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:30.434520 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-pg4kq_7c09760d-2160-41d4-b991-f6324ad5aff7/konnectivity-agent/0.log" Oct 30 17:30:30.652495 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:30.652460 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cjwc7/must-gather-jzn84" event={"ID":"8572b2dc-2297-4948-a18a-126345cc9b03","Type":"ContainerStarted","Data":"8da661e7ab0b7a06ca4a9f18a5006d7d5784048693d312fc2cff7bbc8aca499f"} Oct 30 17:30:30.652655 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:30.652513 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cjwc7/must-gather-jzn84" event={"ID":"8572b2dc-2297-4948-a18a-126345cc9b03","Type":"ContainerStarted","Data":"6555f0c3a8287b6b5e8d8e2f7b605288aa0c9e0967a8997a137bda8a5ba5316c"} Oct 30 17:30:30.693730 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:30.693670 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-cjwc7/must-gather-jzn84" podStartSLOduration=1.572522782 podStartE2EDuration="6.693652878s" totalImagesPullingTime="5.121130096s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:30.692301977 +0000 UTC m=+4057.919477077" watchObservedRunningTime="2026-10-30 17:30:30.693652878 +0000 UTC m=+4057.920827978" Oct 30 17:30:30.714457 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:30.714427 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-180.ec2.internal_9a63059ab80503558edad3ed2b226c76/haproxy/0.log" Oct 30 17:30:32.455393 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:32.455363 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-6kvn2_58874bb8-6c51-4f03-ab7d-ebf99148b4d8/manager/0.log" Oct 30 17:30:32.482904 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:32.482878 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-6kvn2_58874bb8-6c51-4f03-ab7d-ebf99148b4d8/manager/0.log" Oct 30 17:30:34.103820 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:34.103763 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-7blqd_2003d468-00b8-48c3-9b13-f152737d18ed/manager/0.log" Oct 30 17:30:34.226778 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:34.226744 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-7blqd_2003d468-00b8-48c3-9b13-f152737d18ed/manager/0.log" Oct 30 17:30:39.198725 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:39.198668 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-jcntx_12f71e3d-d209-44f3-b5a8-9eab1f962214/manager/0.log" Oct 30 17:30:39.454583 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:39.454440 4493 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-skgpd/perf-node-gather-daemonset-fd22z"] Oct 30 17:30:39.461702 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:39.461669 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-skgpd/perf-node-gather-daemonset-fd22z" Oct 30 17:30:39.466344 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:39.466311 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-jcntx_12f71e3d-d209-44f3-b5a8-9eab1f962214/manager/0.log" Oct 30 17:30:39.467593 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:39.467566 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-78bcf8868f-w5c9k_ad767d7d-7fb3-480d-bf9b-6583a00936fb/authorino/0.log" Oct 30 17:30:39.489692 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:39.489666 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-skgpd/perf-node-gather-daemonset-fd22z"] Oct 30 17:30:39.537112 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:39.537083 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f4a3145f-d425-40ed-b8b4-e4e3e424d283-proc\") pod \"perf-node-gather-daemonset-fd22z\" (UID: \"f4a3145f-d425-40ed-b8b4-e4e3e424d283\") " pod="openshift-must-gather-skgpd/perf-node-gather-daemonset-fd22z" Oct 30 17:30:39.537112 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:39.537114 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f4a3145f-d425-40ed-b8b4-e4e3e424d283-lib-modules\") pod \"perf-node-gather-daemonset-fd22z\" (UID: \"f4a3145f-d425-40ed-b8b4-e4e3e424d283\") " pod="openshift-must-gather-skgpd/perf-node-gather-daemonset-fd22z" Oct 30 17:30:39.537296 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:39.537140 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f4a3145f-d425-40ed-b8b4-e4e3e424d283-podres\") pod \"perf-node-gather-daemonset-fd22z\" (UID: \"f4a3145f-d425-40ed-b8b4-e4e3e424d283\") " pod="openshift-must-gather-skgpd/perf-node-gather-daemonset-fd22z" Oct 30 17:30:39.537296 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:39.537235 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f4a3145f-d425-40ed-b8b4-e4e3e424d283-sys\") pod \"perf-node-gather-daemonset-fd22z\" (UID: \"f4a3145f-d425-40ed-b8b4-e4e3e424d283\") " pod="openshift-must-gather-skgpd/perf-node-gather-daemonset-fd22z" Oct 30 17:30:39.537368 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:39.537333 4493 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98rvl\" (UniqueName: \"kubernetes.io/projected/f4a3145f-d425-40ed-b8b4-e4e3e424d283-kube-api-access-98rvl\") pod \"perf-node-gather-daemonset-fd22z\" (UID: \"f4a3145f-d425-40ed-b8b4-e4e3e424d283\") " pod="openshift-must-gather-skgpd/perf-node-gather-daemonset-fd22z" Oct 30 17:30:39.571761 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:39.571726 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-skgpd\"/\"kube-root-ca.crt\"" Oct 30 17:30:39.571928 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:39.571805 4493 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-skgpd\"/\"openshift-service-ca.crt\"" Oct 30 17:30:39.571928 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:39.571837 4493 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-skgpd\"/\"default-dockercfg-fv925\"" Oct 30 17:30:39.579468 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:39.579447 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-gzvkk_39b545e1-1a61-4ddf-a00a-f396058e6a88/manager/0.log" Oct 30 17:30:39.638066 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:39.638043 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f4a3145f-d425-40ed-b8b4-e4e3e424d283-proc\") pod \"perf-node-gather-daemonset-fd22z\" (UID: \"f4a3145f-d425-40ed-b8b4-e4e3e424d283\") " pod="openshift-must-gather-skgpd/perf-node-gather-daemonset-fd22z" Oct 30 17:30:39.638196 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:39.638074 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f4a3145f-d425-40ed-b8b4-e4e3e424d283-lib-modules\") pod \"perf-node-gather-daemonset-fd22z\" (UID: \"f4a3145f-d425-40ed-b8b4-e4e3e424d283\") " pod="openshift-must-gather-skgpd/perf-node-gather-daemonset-fd22z" Oct 30 17:30:39.638196 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:39.638110 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f4a3145f-d425-40ed-b8b4-e4e3e424d283-podres\") pod \"perf-node-gather-daemonset-fd22z\" (UID: \"f4a3145f-d425-40ed-b8b4-e4e3e424d283\") " pod="openshift-must-gather-skgpd/perf-node-gather-daemonset-fd22z" Oct 30 17:30:39.638196 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:39.638142 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f4a3145f-d425-40ed-b8b4-e4e3e424d283-sys\") pod \"perf-node-gather-daemonset-fd22z\" (UID: \"f4a3145f-d425-40ed-b8b4-e4e3e424d283\") " pod="openshift-must-gather-skgpd/perf-node-gather-daemonset-fd22z" Oct 30 17:30:39.638196 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:39.638181 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f4a3145f-d425-40ed-b8b4-e4e3e424d283-lib-modules\") pod \"perf-node-gather-daemonset-fd22z\" (UID: \"f4a3145f-d425-40ed-b8b4-e4e3e424d283\") " pod="openshift-must-gather-skgpd/perf-node-gather-daemonset-fd22z" Oct 30 17:30:39.638196 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:39.638181 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f4a3145f-d425-40ed-b8b4-e4e3e424d283-proc\") pod \"perf-node-gather-daemonset-fd22z\" (UID: \"f4a3145f-d425-40ed-b8b4-e4e3e424d283\") " pod="openshift-must-gather-skgpd/perf-node-gather-daemonset-fd22z" Oct 30 17:30:39.638418 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:39.638253 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f4a3145f-d425-40ed-b8b4-e4e3e424d283-sys\") pod \"perf-node-gather-daemonset-fd22z\" (UID: \"f4a3145f-d425-40ed-b8b4-e4e3e424d283\") " pod="openshift-must-gather-skgpd/perf-node-gather-daemonset-fd22z" Oct 30 17:30:39.638418 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:39.638257 4493 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98rvl\" (UniqueName: \"kubernetes.io/projected/f4a3145f-d425-40ed-b8b4-e4e3e424d283-kube-api-access-98rvl\") pod \"perf-node-gather-daemonset-fd22z\" (UID: \"f4a3145f-d425-40ed-b8b4-e4e3e424d283\") " pod="openshift-must-gather-skgpd/perf-node-gather-daemonset-fd22z" Oct 30 17:30:39.638418 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:39.638252 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f4a3145f-d425-40ed-b8b4-e4e3e424d283-podres\") pod \"perf-node-gather-daemonset-fd22z\" (UID: \"f4a3145f-d425-40ed-b8b4-e4e3e424d283\") " pod="openshift-must-gather-skgpd/perf-node-gather-daemonset-fd22z" Oct 30 17:30:39.660759 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:39.660738 4493 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98rvl\" (UniqueName: \"kubernetes.io/projected/f4a3145f-d425-40ed-b8b4-e4e3e424d283-kube-api-access-98rvl\") pod \"perf-node-gather-daemonset-fd22z\" (UID: \"f4a3145f-d425-40ed-b8b4-e4e3e424d283\") " pod="openshift-must-gather-skgpd/perf-node-gather-daemonset-fd22z" Oct 30 17:30:39.703667 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:39.703629 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-8xm6t_cf7e09ae-b9ba-420d-b071-bc2f5ac8b009/manager/0.log" Oct 30 17:30:39.777829 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:39.777725 4493 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-skgpd/perf-node-gather-daemonset-fd22z" Oct 30 17:30:40.090712 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:40.090628 4493 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-skgpd/perf-node-gather-daemonset-fd22z"] Oct 30 17:30:40.098131 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:40.098063 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-kksz8_6594d922-089d-41e9-9444-5a888d5fa675/manager/0.log" Oct 30 17:30:40.704772 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:40.704742 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-k9d99_19632c83-f396-437f-ae2d-93229596ce61/limitador/0.log" Oct 30 17:30:40.722317 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:40.722297 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-6kvn2_58874bb8-6c51-4f03-ab7d-ebf99148b4d8/manager/0.log" Oct 30 17:30:40.733528 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:40.733504 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-skgpd/perf-node-gather-daemonset-fd22z" event={"ID":"f4a3145f-d425-40ed-b8b4-e4e3e424d283","Type":"ContainerStarted","Data":"0376360c99a394c845c410a721ba570c436857fb95318cd74cb505ba8363ac28"} Oct 30 17:30:40.733668 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:40.733554 4493 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-skgpd/perf-node-gather-daemonset-fd22z" event={"ID":"f4a3145f-d425-40ed-b8b4-e4e3e424d283","Type":"ContainerStarted","Data":"c632b1086f5b91ee613589de188ac0ed72b04000dd41d3c86df594efb20996f7"} Oct 30 17:30:40.733668 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:40.733579 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-skgpd/perf-node-gather-daemonset-fd22z" Oct 30 17:30:40.773691 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:40.773645 4493 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-skgpd/perf-node-gather-daemonset-fd22z" podStartSLOduration=1.7736294209999999 podStartE2EDuration="1.773629421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:30:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:40.77217106 +0000 UTC m=+4067.999346159" watchObservedRunningTime="2026-10-30 17:30:40.773629421 +0000 UTC m=+4068.000804524" Oct 30 17:30:40.829998 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:40.829968 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-6kvn2_58874bb8-6c51-4f03-ab7d-ebf99148b4d8/manager/0.log" Oct 30 17:30:40.869320 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:40.869280 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-h6wgm_3319a971-2f82-469a-8fc3-20687e710719/manager/1.log" Oct 30 17:30:40.919535 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:40.919500 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-h6wgm_3319a971-2f82-469a-8fc3-20687e710719/manager/0.log" Oct 30 17:30:42.115699 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:42.115651 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4779dd76-1dee-475b-8276-5823961467dc/alertmanager/0.log" Oct 30 17:30:42.248011 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:42.247977 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4779dd76-1dee-475b-8276-5823961467dc/config-reloader/0.log" Oct 30 17:30:42.377540 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:42.377453 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4779dd76-1dee-475b-8276-5823961467dc/kube-rbac-proxy-web/0.log" Oct 30 17:30:42.419824 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:42.419772 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-7blqd_2003d468-00b8-48c3-9b13-f152737d18ed/manager/0.log" Oct 30 17:30:42.498277 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:42.498238 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-7blqd_2003d468-00b8-48c3-9b13-f152737d18ed/manager/0.log" Oct 30 17:30:42.507419 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:42.507393 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4779dd76-1dee-475b-8276-5823961467dc/kube-rbac-proxy/0.log" Oct 30 17:30:42.635198 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:42.635079 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4779dd76-1dee-475b-8276-5823961467dc/kube-rbac-proxy-metric/0.log" Oct 30 17:30:42.764187 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:42.764160 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4779dd76-1dee-475b-8276-5823961467dc/prom-label-proxy/0.log" Oct 30 17:30:42.898074 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:42.898042 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4779dd76-1dee-475b-8276-5823961467dc/init-config-reloader/0.log" Oct 30 17:30:43.175395 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:43.175317 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-d5phv_4a801701-a1db-4feb-a49e-018c3fd03038/kube-state-metrics/0.log" Oct 30 17:30:43.301751 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:43.301714 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-d5phv_4a801701-a1db-4feb-a49e-018c3fd03038/kube-rbac-proxy-main/0.log" Oct 30 17:30:43.431272 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:43.431184 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-d5phv_4a801701-a1db-4feb-a49e-018c3fd03038/kube-rbac-proxy-self/0.log" Oct 30 17:30:44.585109 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:44.585053 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-v27z6_4471fff4-f324-4b99-96bd-8b4feeb518a2/node-exporter/0.log" Oct 30 17:30:44.832178 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:44.832121 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-v27z6_4471fff4-f324-4b99-96bd-8b4feeb518a2/kube-rbac-proxy/0.log" Oct 30 17:30:44.963445 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:44.963414 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-v27z6_4471fff4-f324-4b99-96bd-8b4feeb518a2/init-textfile/0.log" Oct 30 17:30:45.097334 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:45.097293 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-2hmkq_7cdc9574-e634-408a-8084-b7263520724d/kube-rbac-proxy-main/0.log" Oct 30 17:30:45.226555 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:45.226472 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-2hmkq_7cdc9574-e634-408a-8084-b7263520724d/kube-rbac-proxy-self/0.log" Oct 30 17:30:45.356535 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:45.356492 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-2hmkq_7cdc9574-e634-408a-8084-b7263520724d/openshift-state-metrics/0.log" Oct 30 17:30:46.594183 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:46.594134 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-8bnmd_a3b1f16c-b2cc-4f04-92a9-b424da398fe7/prometheus-operator-admission-webhook/0.log" Oct 30 17:30:46.729183 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:46.729145 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-64fdfd6547-66n2g_4709930f-fc73-4b82-951f-a428ad89313a/telemeter-client/0.log" Oct 30 17:30:46.758214 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:46.758172 4493 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-skgpd/perf-node-gather-daemonset-fd22z" Oct 30 17:30:46.859184 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:46.859099 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-64fdfd6547-66n2g_4709930f-fc73-4b82-951f-a428ad89313a/reload/0.log" Oct 30 17:30:46.896441 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:46.896406 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-jcntx_12f71e3d-d209-44f3-b5a8-9eab1f962214/manager/0.log" Oct 30 17:30:46.975372 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:46.975334 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-jcntx_12f71e3d-d209-44f3-b5a8-9eab1f962214/manager/0.log" Oct 30 17:30:46.989400 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:46.989377 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-64fdfd6547-66n2g_4709930f-fc73-4b82-951f-a428ad89313a/kube-rbac-proxy/0.log" Oct 30 17:30:48.456322 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:48.456278 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-6kvn2_58874bb8-6c51-4f03-ab7d-ebf99148b4d8/manager/0.log" Oct 30 17:30:48.953020 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:48.952960 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-6kvn2_58874bb8-6c51-4f03-ab7d-ebf99148b4d8/manager/0.log" Oct 30 17:30:49.908821 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:49.908763 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-7blqd_2003d468-00b8-48c3-9b13-f152737d18ed/manager/0.log" Oct 30 17:30:50.392863 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:50.392829 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-7blqd_2003d468-00b8-48c3-9b13-f152737d18ed/manager/0.log" Oct 30 17:30:52.595549 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:52.595515 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7d45d5c789j7m84_1c9bc2df-9829-4346-a082-b8255c37bb3c/istio-proxy/0.log" Oct 30 17:30:53.128899 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:53.128850 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-6f8685f4cf-fx429_5d9779f9-732c-4971-b52b-d26ce92abf2f/istio-proxy/0.log" Oct 30 17:30:54.660876 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:54.660836 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-jcntx_12f71e3d-d209-44f3-b5a8-9eab1f962214/manager/0.log" Oct 30 17:30:54.785935 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:54.785896 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-jcntx_12f71e3d-d209-44f3-b5a8-9eab1f962214/manager/0.log" Oct 30 17:30:55.607396 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:55.607364 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jvqzq_7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc/dns/0.log" Oct 30 17:30:55.731021 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:55.730989 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jvqzq_7c4a0bb4-3e7f-4ea1-91e2-4ae2b79f0ccc/kube-rbac-proxy/0.log" Oct 30 17:30:56.290130 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:56.290099 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7d45d5c789j7m84_1c9bc2df-9829-4346-a082-b8255c37bb3c/istio-proxy/0.log" Oct 30 17:30:56.350729 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:56.350706 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-r5ppb_0e649944-a09d-44b1-91d2-fda00b30292d/dns-node-resolver/0.log" Oct 30 17:30:56.406911 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:56.406871 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7d45d5c789j7m84_1c9bc2df-9829-4346-a082-b8255c37bb3c/istio-proxy/0.log" Oct 30 17:30:56.778008 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:56.777979 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-6f8685f4cf-fx429_5d9779f9-732c-4971-b52b-d26ce92abf2f/istio-proxy/0.log" Oct 30 17:30:56.959767 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:56.959735 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-6f8685f4cf-fx429_5d9779f9-732c-4971-b52b-d26ce92abf2f/istio-proxy/0.log" Oct 30 17:30:57.481422 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:57.481391 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7d45d5c789j7m84_1c9bc2df-9829-4346-a082-b8255c37bb3c/istio-proxy/0.log" Oct 30 17:30:57.619162 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:57.619131 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-8b5bbf7-s2s69_2a272af5-e22c-4de5-8af9-28a810fe7035/registry/0.log" Oct 30 17:30:57.727057 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:57.727019 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jlr9c_17bb82ba-5262-47f7-abbc-99f1329500fa/node-ca/0.log" Oct 30 17:30:57.968207 ip-10-0-132-180 kubenswrapper[4493]: I1030 17:30:57.968176 4493 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-6f8685f4cf-fx429_5d9779f9-732c-4971-b52b-d26ce92abf2f/istio-proxy/0.log"